The Altoona facility is the first in Facebook’s fleet to feature a building-wide network fabric – an entirely new way to do intra-data center networking the company’s infrastructure engineers have devised.

The social network is moving away from the approach of arranging servers into multiple massive compute clusters within a building and interconnecting them with each other. Altoona has a single network fabric whose scalability is limited only by the building’s physical size and power capacity.

Alexey Andreyev, network engineer at Facebook, said the new architecture addresses bandwidth limitations in connecting the massive several-hundred-rack clusters the company has been deploying thus far. A huge amount of traffic takes place within each cluster, but the ability of one cluster to communicate with another is limited by the already high-bandwidth, high-density switches. This means the size of the clusters was limited by capacity of these inter-cluster switches.

By deploying smaller clusters (or “pods,” as Facebook engineers call them) and using a flat network architecture, where every pod can talk to every other pod, the need for high-density switch chassis goes away. “We don’t have to use huge port density on these switches,” Andreyev said.

Each pod includes four devices Facebook calls “fabric switches,” and 48 top-of-rack switches, every one of them connected to every fabric switch via 40G uplinks. Servers in a rack are connected to the TOR switch via 10G links, and every rack has 160G total bandwidth to the fabric.

Here’s a graphic representation of the architecture, courtesy of Facebook:

The system is fully automated, and engineers never have to manually configure an individual device. If a device fails, it gets replaced and automatically configured by software. The same goes for capacity expansion. The system configures any device that gets added automatically.

The new fabric relies on gear available from the regular hardware suppliers, Najam Ahmad, vice president of network engineering at Facebook, said. The architecture is designed, however, to use the most basic functionality in switches available on the market, which means the company has many more supplier options than it has had in the older facilities that rely on those high-octane chassis for inter-cluster connectivity. “Individual platforms are relatively simple and available in multiple forms or multiple sources,” Ahmad said.

The company has built data centers in Prineville, Oregon, Forest City, North Carolina, and Luleå, Sweden. It also leases data centers space from wholesale providers in California and Northern Virginia, but has been moving out of those facilities and subleasing the space until its long-term lease agreements expire.

Recently at the Open Compute Summit, Facebook VP of Engineering, Jay Parikhdescribed how Facebook has rigged together a system based on 10,000 Blu-ray discs that fit into a single cabinet, capable of storing 1 petabyte of data. Facebook plans to improve on this first attempt and be able to reach 5 petabytes per cabinet. Compared to existing hard disk solutions, Parikh says that the Blu-ray solution will be 50 percent cheaper and use 80 percent less energy. It will also provide 50 years worth of durability for data stored on the discs.

Sony, original creator of Blu-Ray technology, and Panasonic have also recently teamed up to address the need for finding a robust long-term low-cost solution for archiving content. They’ve recently announced that they would jointly produce a 300 GB optical disc for long-term archival. The product family is to be named “Archive Disc” and should become available in 2015. The target is to create Blu-Ray optical discs with a 1 TB capacity and to then package them into a disc cartridge containing 12 discs.

Not everyone is buying into the Facebook strategy. Mark Peters, a senior analyst at Enterprise Strategy Group, commented that “some of what Facebook was doing was laying out a challenge to storage vendors. I think what they were really communicating was that there is a need for low cost reliable storage.”

Bruce Kornfeld, Interim CMO at Spectra Logic, questions the logic of using Blu-Ray for commercial purposes. Kornfeld said that “over the last 20 years the commercial data storage industry has attempted to commercialize multiple consumer technologies including CD ROM, CD Writeable, DVD writeable, 4mm Digital Audio Tape, 8mm video tape, and VHS video tape – with mixed results. Consumer technologies offer high-volume and typically low cost storage media. The ‘low cost’ strengths of these technologies have also led to their downfall as they lack the device robustness and data reliability required by commercial data storage applications. In addition, consumer grade drives and media just don’t last very long. Blu-ray disc drive load mechanisms are probably good to a few thousand load/unload cycles. This compares with an LTO tape drive that is rated to 250,000 load/unload cycles. As you can see – the difference in durability is substantial!”

SAN JOSE, Calif. – Over the last three years, Facebook has saved more than $1.2 billion by using Open Compute designs to streamline its data centers and servers, the company said today. Those massive gains savings are the result of hundreds of small improvements in design, architecture and process, write large across hundreds of thousands of servers.

Facebook CEO Mark Zuckerberg, at left, discusses the company’s infrastructure with Tim O’Reilly of O’Reilly Media yesterday at the Open Compute Summit in San Jose, Calif. (Photo: Colleen Miller)

The savings go beyond the cost of hardware and data centers, Facebook CEO Mark Zuckerberg told 3,300 attendees at the Open Compute Summit in San Jose. The innovations from Open Compute Project have saved Facebook enough energy to power 40,000 homes, and brought carbon savings equivalent to taking 50,000 cars off the road, Zuckerberg said.

The innovations created by Open Compute provide a major opportunity for other companies to achieve similar gains, he said “People are spending so much on infrastructure,” said Zuckerberg, who discussed Facebook’s initiatives with technology thought leader Tim O’Reilly of O’Reilly Media. “It creates a pretty broad incentive for people to participate.”

The focus on open hardware was a logical outgrowth of the way the company views the world, according to Zuckerberg. “Facebook has always been a systems company,” he said. “A lot of the early work we did was with open systems and open source.”

The $1.2 billion in savings serves as a validation of Facebook’s decision to customize its own data center infrastructure and build its own servers and storage. These refinements were first implemented in the company’s data center in Prineville, Oregon and have since been used in subsequent server farms Facebook has built in North Carolina, Sweden and Iowa.

This happened, Parikh said, because Facebook was willing to set aside the conventional ways that servers and data centers were built, and try something new. The Open Compute Project can extend these savings to other companies, he said.

Summary: During the Open Compute Summit in San Jose, Facebook VP of Engineering Jay Parikh shared some big statistics for the company’s cold storage efforts, including those for a protoytpe Blu-ray system capable of storing a petabyte of data today.

At last year’s Open Compute Summit, Facebook VP of Engineering Jay Parikh suggested the company might consider Blu-ray discs as a medium for improving durability and cutting costs for its long-term cold storage efforts. Now, Parikh said Tuesday during a keynote at this year’s event, the company is actually doing it.

Facebook has built a prototype Blu-ray system capable of storing 10,000 discs and 1 petabyte of data in a single cabinet, with plans to scale it to 5 petabytes per cabinet. Blu-ray storage would save the company 50 percent in costs and 80 percent in energy usage over its existing hard-disk-based cold storage methods, Parikh said, and will provide 50 years worth of durability for data stored on the discs.

That’s not the only tidbits Parikh shared about the company’s cold storage efforts, though. He said that the first site is now in production and storing 30 petabytes of data, but a second will be coming online soon. Within a couple months, Facebook expects to have 150 petabytes of data in cold storage, which is just a fraction of the 3 exabytes each facility is designed to house.

One other nugget worth noting from the morning as it relates to Facebook’s innovation on the storage front: In order to save money over data center-grade SSDs but still speed up the process, Facebook is now booting web servers using netbook-grade SSDs, hardware executive Matt Corddry said.