Facebook Open Compute Project Shapes Big Data Hardware

Big data practitioners like Facebook, Goldman Sachs and Fidelity are setting the standards for cheaper, more efficient servers and systems from the likes of Applied Micro, AMD, Dell and Intel.

Big Data's Surprising Uses:
From Lady Gaga To CIA

(click image for larger view and for slideshow)

Facebook and the Open Compute Project (OCP) announced Wednesday that they've made huge strides toward the goal of setting standards for the most efficient server, storage and data center hardware available for scalable computing.

Facebook launched OCP 18 months ago hoping to crowdsource the problem of creating better hardware for high-scale computing. From its start with one member, Facebook, and 200 participants, the group now has more than 50 member companies and saw more than 2,000 participants attend this week's Open Compute Platform Summit in Santa Clara, Calif.

OCP says its standards promise to deliver hardware that is 24% more energy efficient and 38% more cost efficient, on average, than so-called commodity hardware. The group is working on specs for storage, motherboard and server design, racks, interoperability, hardware management, and data center design.

"We know what we need and we see the current and future challenges that the deluge of data, analytics, compute power and these massively scaled data centers are bringing forward," said Frank Frankovsky, VP of hardware design and supply chain at Facebook and chairman of OCP's Open Compute Foundation, which is modeled after the Apache Software Foundation.

Hardware vendors have also joined OCP and are designing hardware to its specs. AMD and Intel announced in May collaborative work with OCP members Fidelity and Goldman Sachs to develop new boards to suit their financial processing workloads, and both chip makers presented finished products at this week's event.

Intel has contributed specs to OCP for Silicon Photonic interconnect technologies that already surpass 100 gigabits per second -- nearly twice the speed of the fastest interconnect technologies currently available. Presenting on the technology, Intel CTO Justin Rattner said the products will soon be commercially available from Intel and would solve data I/O constraints both inside servers and in connecting servers within racks and racks within data centers.

"It has been a decade-long effort to create the technology, and it has enormous potential due to its combination of speed, low power consumption and low cost," Rattner said.

Dell also presented at the event, showing off Intel- and ARM-based OCP-compliant servers that will soon be commercially available. Other manufacturers working on or delivering OCP-compliant hardware and components include Applied Micro, Calxeda, Delta, Emerson and Hyve Solutions.

Many of the cost savings in OCP-standard products come down to two design themes: commonality and disaggregation. In an example of commonality, OCP unveiled a "common slot architecture" that will enable ARM and X86 chips to coexist on the same motherboard. The design will give end-user organizations flexibility to test and configure servers for best possible performance and then reconfigure as workloads change without facing proprietary design constraints. The spec is being supported by AMD, Applied Micro, Calxeda and Intel.

Disaggregation is about separating and modularizing storage, compute, interconnects, power, cooling and other components so companies can custom configure to their workload requirements.

This approach also supports smarter technology refreshes, so companies can swap out and replace quickly evolving components, such as CPUs, while keeping in service slowly evolving components, such as memory and network interface cards.

"You might be five generations behind on the processor while 80% of the other components of the server are still good," Frankovsky said, noting that monolithic designs don't let you replace dated components such as CPUs. "Smarter technology refreshes not only make sense from a financial perspective; think about how much less IT equipment is going to go into the waste stream."

While many OCP members are hyper-scale Internet companies, the point was made Wednesday that the group's efforts will benefit companies with mainstream computing requirements within a few short years.

"Hadoop, Hive, Hbase and other new platforms are accelerating the growth of big data adoption, so our big data challenges of today are everyone's big data challenges of tomorrow," said Jay Parikh, Facebook's VP of infrastructure engineering.

Silicon photonics sounds like it has huge advantages over copper-wire-based interconnects. Components that used to have to be on the motherboard (close to the processing power) can now be "disaggregated" (meaning separated from the board). The photonic components themselves can be stamped mass-production-style, just like chips, for low cost. The cables can be much longer than copper without speed penalties, thus they're suitable for connecting servers within racks and racks across data centers as well as components inside the server.

Silicon photonics sounds like it has huge advantages over copper wire. Components that used to have to be on the mother board can now be "disaggregated." The components can be stamped, just like chips for low cost. The cables can run longer than copper without speed penalties, thus they're suitable for connecting racks across data centers.

To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.

Transformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.