From its inception, Facebook’s culture has been built around a mantra called The Hacker Way. This may sound vague to non-techies, or trite to seasoned software developers, but Facebook is proving that living a hacker culture is actually possible, and the innovation produced can reap significant rewards in the face of more stoic methods.
A prime example of this type of culture is Cade Metz recent article in Wired Magazine that digs into the deep architecture that runs the massive Facebook site (used by over 1 billion people worldwide) and highlights the culture that allowed 3 engineers the freedom to completely rebuild it.
Everything is Replaceable
In 2010 Facebook finalized a tool they dubbed HipHop that would convert PHP code into C++ before it was executed on the server. This allowed the site to handle 500 to 600 percent more traffic than the pure PHP site on the same number of servers.
So you would think they were done. HipHop would be the operating back-end of the site for the next five, maybe ten years, right? Try less than three.
But shortly after [HipHop released], Paroski, Adams, and third Facebook engineer, Jason Evans, decided they could do better. Standing around the proverbial water cooler one afternoon, the trio agreed that Facebook could take PHP performance to an even higher level if they replaced HipHop with something called a virtual machine — software that could provide even greater synergy between the site’s PHP code and the server hardware that runs it.
Within weeks, they went to work on their virtual machine, and after a few months, they’d made enough progress to get the company behind the project. Eventually, the Facebook brass put another seven engineers to work on the new platform — and halted the development of HipHop.
Only months after releasing a new back-end architecture, Facebook engineers were already thinking of replacing it. But thoughts are one thing. These three engineers were given several months to test their new theories…before having to engage upper management to get buy-in.
It’s not clear how much political infighting ensued during talks of replacing the new expensive architecture they just released. I’m sure they had lots of healthy discussions, but the plan to continuously improve is the decision that won. This is The Hacker Way in action.
It’s Not Done Until It’s Better
So, with management approval, a larger team and financial backing, they were off to build a faster solution to the solution they had just released.
The only trouble is that after Adams, Evans, and Paroski spent two years building this virtual machine, it was slower than the live website.
Two years, and still not there yet. Let’s just quantify this real quick. We know these types of companies move incredibly fast. There is a lot of pressure to get functionality into production as soon as possible. Some might call it “the time it takes to get to market” or something close to that.
But the reason these technology companies move so fast is that they know many of their solutions will take a year or more to be fully realized. Delaying on a decision to start building the next generation system or application is what truly slows things down.
But once the effort is started, functionality is not released based on arbitrary deadlines or quarterly goals. It’s released only when it’s better than the previous solution, and sometimes that is a multi-year journey. But this doesn’t mean they don’t move fast.
Recruit the Best and Brightest
Keith Adams personifies Facebook’s recent evolution. The Brown University graduate began his career at VMware, alongside Eli Collins, where he built the most complex of systems software. VMware makes software that lets the world’s businesses run a different kind of virtual machine — a way of treating one computer server like many servers – and Adams worked on the guts of this “hypervisor” software.
It may seem odd that Adams would move from VMware to Facebook — even Collins was somewhat surprised when he made the leap — but that’s what Facebook has become. Like Google and Amazon and Yahoo and even Twitter, Facebook has grown so large, it needs engineers capable of rethinking the fundamental ways our computers operate.
Recruiting is critical to gaining the knowledge required to think differently about problems, to try new solutions, and simply to have folks smart enough to build systems, tools and processes from scratch.
Give Teams the Ability to Focus
But things took far longer than expected. Part of the problem, Adams says, is that they underestimated the complexity of the task, but the other issue is that HipHop continued to improve. For months, they were chasing a moving target. After two years, they got to a point where the virtual machine could run the entire world of Facebook, but it was still three times slower than the original HipHop system.
So what did they do? They locked themselves down and focused.
[The project manager] moved Facebook’s virtual machine team into a room on the ground floor of Building 18, where there were relatively few engineers, and according to Paroksi, they didn’t even tell anyone they were there. “It was like we had been lost in the desert waiting for someone to rescue us by helicopter,” Adams remembers. “But then we decided we just had to hike out. We didn’t know quite where to go, but we had to get there under our own power.”
“The idea was to find small things, where you could quickly do an experiment and find out if changing that code was going to help,” says Paroski. “If the experiment failed, you would put it down and quickly pick up something else.”
On the far wall of the room, they organized their efforts on a massive white board. It was littered with Post-It Notes, each identifying a possible way of improving the system and each positioned according to how long the improvement would take. If a Post-It Note proved a dead-end, it was summarily moved to the side.
The Battle Cave
Then, above this white board, the team installed two monitors that tracked the speed of the new system relative to HipHop. At first, the line barely moved. But with Adams taking the early shift, and others, such as Evans and Paroski, working well into the night for a good five weeks, the tiny improvements began to pay off.
Even for the best and brightest, perseverance is a quality that has to be nurtured, encouraged and rewarded.
It was finally in November 2012 when they achieved victory. Dubbed the HipHop Virtual Machine (HHVM), it was now regularly outpacing the original HipHop architecture and was quickly transitioned to the production environment. When you login to Facebook today, HHVM is what makes the site run.
“Apple is about polish. Google is about scale. Microsoft is about, well, 30 years old,” says ex-Googler and Box vice president of engineering Sam Schillace. “But Facebook is about innovation. They’re not necessarily optimized for elegance. They’re optimized for innovation. The idea is to crush everyone with pure experimentation and velocity.”
Since those first lines of code in Zuckerberg’s dorm room in 2003, Facebook has changed large portions of its architecture several times over. The Hacker Way is a culture that invites in great talent, gives them the freedom to do big things, and doesn’t sit on stale technology…or even brand new technology for that matter. This continuous improvement is the hard work that keeps consumers glued to their Facebook accounts.
Fox Tip: The system / process / app you built last month, is probably in need of a healthy overhaul. Start today!