The Facebook Special: How Intel Builds Custom Chips for Giants of the Web

Facebook's Frank Frankovsky. Photo: Brian Frank / Wired

We know that Facebook writes a lot of the software that underpins its massive social network. And we know it designs its very own computer servers to juggle your every Like, comment, and cat photo. But to get a sense of how deep the tinkering goes inside the company, consider this: Facebook now works with Intel to customize microprocessors for its particular needs.

According to Frank Frankovsky — who oversees hardware design at Facebook — this started as far back as 2009. That’s when the company first asked Intel for certain changes to the design of the silicon chips it builds for computer servers — the machines that drive the internet and the private computer networks inside the world’s businesses. Since then, Frankovsky says, Intel has worked hand-in-hand with the company to accommodate such changes.

For the most part, he explains, these changes are rolled into the same processors that Intel sells to the world at large, though there are cases where the changes are unlikely to benefit anyone but Facebook. “The more insight we can give our technology suppliers about what makes our software work the best, the more we’re able to influence their design roadmaps,” Frankovsky says. “We do influence their roadmaps — way upstream — but then they are able to bring these changes out to all of their customers.”

‘Sometimes OEMs and end customers will ask us to put a feature into the silicon and it sort of depends upon how big a deal it is and whether it has to be invisible [to other buyers] or proprietary to a customer. We’re always happy to, if we can find a way to get it into the silicon.’

— Intel’s Jason Waxman

This longtime collaboration between Facebook and Intel, the world’s dominant chip maker, is just one sign of a larger shift in the multi-billion-dollar market for server chips and other hardware that powers the web’s most popular services. For years, says Intel’s Jason Waxman, the chip giant has worked to tweak chip designs at the request of big-name server manufacturers along the lines of Dell and HP and IBM — original equipment manufacturers, or OEMs — but more recently, the company has been customizing processors for companies along the lines of Facebook — companies that run more computer servers than almost any other operation on the planet.

In some cases, he says, Intel will even make changes that are only available to a particular customer. “Sometimes OEMs and end customers will ask us to put a feature into the silicon and it sort of depends upon how big a deal it is and whether it has to be invisible [to other buyers] or proprietary to a customer,” Waxman says. “We’re always happy to, if we can find a way to get it into the silicon.”

Waxman says that Intel typically makes custom changes for about “five to eight” OEMs and about five “end customers,” meaning companies like Facebook that use servers rather than sell them. But he declines to name the particular companies involved.

Meanwhile, chief-Intel-rival AMD says it customizes chips in similar ways. Facebook works with all sorts of other hardware makers on custom parts, from memory outfits to server makers, as it seeks to build data center hardware that’s more efficient and less costly than the norm. And Google and Amazon have indicated they work in much the same way.

Both Facebook and Intel decline to identify specific changes the two companies have made to Intel servers chips with Facebook’s operation in mind, but Frankovsky indicates that some changes involve hardware that can accelerate certain software tasks and that others involve tools for managing the way chips behave. “We do some management engine work,” he says. “Some of that may make sense for broad communities to adopt — but other parts are pretty Facebook-specific.”

Waxman adds that at the request of certain customers, Intel will go so far as to etch new instructions or new logic into a chip. Typically, this logic is worked into the chips the company sells to the rest of the market, but he indicates that the company will offer specialized chips only to a particular customer — if that customer is willing to purchase an usually large number of processors. Sometimes, he says, Intel will turn this logic off for most customers, but then provide one buyer with a way to turn it on.

He also says that there are cases where Intel will crank the chip clock speeds to unusually high levels at the request of certain buyers — if the buyer is willing to deal with the extra heat this generates.

According to Frankovsky, Facebook has long worked with Intel in an effort to squeeze more performance per watt out of the company’s processors. “Early on, we collaborated with Intel on how many cores you could put into a package and what the maximum clock rate you could run those at,” he says. This sort of thing, Waxman says, is common among the large “cloud” companies. Some companies, he says, will even request chips with a certain number of “cores” per chip, where each core is basically its own processor, and then ask for a clock speed tuned to that particular number of cores.

But this is just one way the market is changing. Facebook is at the forefront of a movement to fashion all sorts of hardware that’s better suited to running the world’s largest web services. It now designs its own machines in tandem with manufacturers in Asia, and it “open sources” these designs, sharing them with the world at large. Other companies, including cloud-computing-outfit Rackspace, are already using these designs to fashion their own hardware.

This process has shrunk the the supply chain for computer servers, bringing the likes of Facebook much closer to the companies that actually manufacturer the hardware. According to someone familiar with the matter, Facebook negotiates directly with Intel on the price of chips, and the chips are then sent to manufacturers in Asia, who build machines on behalf of the social networking giant. Google and Amazon work in similar ways, and so many others, including Rackspace, are following suit.

Processor customization is the most complicated part of this new server world. But it will likely become more prevalent as more and more chip makers get into the server game behind designs from Intel-rival ARM. For years, ARM has licenses its cellphone chip designs to a wide range of manufacturers so that they can be tweaked and customized, and it’s now doing the same thing with server chips.

“Open source is having an impact on the hardware industry,” says Frankovsky. “The [hardware] companies that are becoming the most successful…are the ones that are setup to work with an open development model and do that level of customization for their customers. Customers know a lot more about what they need than the hardware world has typically given them credit for.”