Intel wants to kill the traditional server rack with 100Gbps links

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

Power and cooling would be shared across CPUs, rather than having separate power supplies for each server. Server, memory, network, and storage resources would all be disaggregated and shared across the rack. Incredibly fast interconnects will be needed to prevent slowdowns because disaggregating components pushes them further apart, and Intel is thus building an interconnect that's capable of 100Gbps.

"We are developing a rack-scale architecture," Lisa Graff, VP and general manager of Intel's data center marketing group, said in a briefing with reporters last week. "We're working with end users, OEMs, and ISVs to drive common standards in a reference architecture."

The first version of this reference architecture is expected to be published sometime in 2014. Graff said the idea is to let data center managers "mix and match components instead of forklifting a rack" when pieces need to be replaced. Sharing things like memory and storage across CPUs will allow higher utilization of computing resources, and a design that eliminates unnecessary parts will let data centers cram more computing power into each rack.

The effort is complementary to Facebook's Open Compute Project. Facebook is already designing its own servers, stripping out extraneous bits of hardware, and it has worked with Intel on possible designs for racks that disaggregate and share resources.

The networking technology used by typical data centers isn't quite fast enough to power disaggregated racks just yet. That's why Intel is developing silicon photonics technology that uses light to move data at up to 100Gbps. Silicon photonics has the added benefit of reducing the amount of cabling needed in a rack.

"Silicon photonics made with inexpensive silicon rather than expensive and exotic optical materials provides a distinct cost advantage over older optical technologies in addition to providing greater speed, reliability, and scalability benefits," Intel said in January, when it announced that it has produced engineering samples of the technology. "Businesses with server farms or massive data centers could eliminate performance bottlenecks and ensure long-term upgradability while saving significant operational costs in space and energy."

Intel isn't the only company trying to speed up networking with silicon photonics. As we reported in December, IBM has "developed a technology that integrates optical communications and electronics in silicon, allowing optical interconnects to be integrated directly with integrated circuits in a chip." IBM said its silicon nanophotonics technology is ready for mass production, although specific implementations haven't been announced.

Corporate data centers won't be receiving an immediate overhaul, but Intel's vision (and IBM's) is one that may be appealing to companies with thousands or tens of thousands of servers.

Intel describes its goals as follows:

Physical Aggregation. All non-critical sheet metal removed and key components such as power supplies and fans taken out of individual servers and consolidated at the rack level. Savings are expected due to higher levels of efficiency and lower costs by reducing the number of fans and power supplies.

Fabric Integration and Storage Virtualization. Disaggregate and separate out the storage from compute systems with direct attached storage, and achieve higher utilization through storage virtualization. The compute and network fabric is the key technology that is enabling disaggregation of storage without impact to performance. Intel Silicon Photonics interconnects will enable higher speed connections between various computing resources within the rack, thus enabling the eventual disaggregation of server, memory, network and storage within the rack.

Future. Ultimately, the industry will move to subsystem disaggregation where processing, memory and I/O will be completely separated into modular subsystems, making it possible to easily upgrade these subsystems rather than doing a complete system upgrade.

Early versions of the rack-scale architecture are planned for deployment by several companies in China, namely Alibaba, Baidu, Tencent, and China Telecom.

New “smartphone-class” chips for the data center

Intel's rack-scale architecture was part of a larger announcement to be made Wednesday at the Intel Developer Forum in Beijing. Intel is introducing new Atom SoCs (system-on-a-chip) for the data center. As we reported last December, Intel is pitching "smartphone-class CPUs" to data centers by combining "extremely low power usage with server-class features, including virtualization technology and Error-Correcting Code (ECC) for higher reliability."

The December 2012 announcement unveiled Intel Atom S1200 processors, the "world's first 64-bit SoC for servers." What's new today is the S12x9 Atom processor for storage systems.

Key features include 40 lanes of integrated PCIe 2.0, the physical paths between I/O and processor; RAID storage acceleration built into the hardware; failover support; native dual-casting that "can allow data to be read from a source and delivered to two memory locations simultaneously"; and Asynchronous DRAM Self-Refresh, which protects DRAM data during power failures.

MacroSAN, Accusys, Qsan, and Qnap are releasing systems based on the new storage chip, Intel said.

Intel also offered brief previews of chips that don't have firm release dates yet. Avoton, the second generation of Intel's 64-bit Atom chips for microservers, is based on Intel's 22nm process.

"Avoton will feature an integrated Ethernet controller and is expected to deliver significant improvements in performance-per-watt," Intel said. "Avoton is now being sampled to customers and the first systems are expected to be available in the second half of 2013."

Another Atom-based SoC built on 22nm process technology, codenamed Rangely, is targeted at network and communications infrastructure such as "entry-level to mid-range routers, switches and security appliances." Like Avoton, Rangely is expected to come out in the second half of this year.

Intel will also be refreshing its Xeon E3, E5, and E7 processor lines later this year. The Haswell-based E3 will improve performance for video analytics with integrated graphics, and lower power consumption to as little as 13 watts. Xeon E5 will improve security with several new features including "Intel OS Guard, the next generation of Intel Execute Disable Bit, [which] protects against privilege attacks by preventing malicious code from executing out of application memory space, in addition to data memory."

Xeon E7, targeted at analyzing massive data sets, triples memory capacity to 12TB in an eight-socket node. The new Xeon E5 will debut in the third quarter of 2013, while Xeon E7 is scheduled for the fourth quarter. The refreshed E3 is coming this year, but Intel did not get more specific than that.

Promoted Comments

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

That looks sensible, although the number of use cases for an entire rack of servers is getting smaller.

A single blade with a couple of 8 core xeon's, 384GB of RAM, an SSD and a bunch of spinning disks... only one in a million companies need more power than that.

They might have a second blade as a backup, but being a backup it's probably also in an entirely different datacenter. Our backups are literally on the opposite side of the planet from the primary servers.

Wow, we really are just reinventing mainframes at this point, aren't we?

Yes! And that's not necessarily a bad thing. The cost balance between CPU/Memory/Storage/Power/Network/Cooling has evolved over the last few decades. We may continue to cycle between mainframes and blades being most cost efficient for the next few decades as technology advances in each area.

There is an extra drawback to the rack-is-a-server model, though: your fault zone is now shared between what used to be somewhat isolated servers.

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

That looks sensible, although the number of use cases for an entire rack of servers is getting smaller.

A single blade with a couple of 8 core xeon's, 384GB of RAM, an SSD and a bunch of spinning disks... only one in a million companies need more power than that.

They might have a second blade as a backup, but being a backup it's probably also in an entirely different datacenter. Our backups are literally on the opposite side of the planet from the primary servers.

It seems like a really, really bad idea for anything that isn't on a gigantic scale. This amplifies the damage done if a component fails, catastrophically.

Plus, I can't imagine (as a self taught geek) GPGPU stuff being feasible separate from the CPUs. With Nvidia and AMD talking about cloud gaming/multimedia applications, this seems a bit backwards.

I wonder what VMware pricing model reaction will be to this? How do do you separate out CPUs and memory if you could technically turn a rack into a server. This could revolutionize the virtual offerings model, and if 1) This becomes popular and 2)VMware tries to pull a gouging price increase or set small limits for vram, this could push more people to other hypervisors.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

I wonder what VMware pricing model reaction will be to this? How do do you separate out CPUs and memory if you could technically turn a rack into a server. This could revolutionize the virtual offerings model, and if 1) This becomes popular and 2)VMware tries to pull a gouging price increase or set small limits for vram, this could push more people to other hypervisors.

It probably won't affect it much honestly. This is actually REALLY REALLY similar to how mainframes operate and there most products are either licensed by the processor or by the "timeslice" or some similar variant.

I'm more surprised that Intel hasn't gone this direction earlier. I believe actually in the Pentium Pro days a few companies DID do this using PPro chips, but they were radically different architecuture machines rather than this. This is very much the built-in low level hypervisors like many of the larger machines could do.

I wonder what VMware pricing model reaction will be to this? How do do you separate out CPUs and memory if you could technically turn a rack into a server. This could revolutionize the virtual offerings model, and if 1) This becomes popular and 2)VMware tries to pull a gouging price increase or set small limits for vram, this could push more people to other hypervisors.

They will do what they do in software licensing for mainframes, they will view the entire rack as a CEC (Central Electronic Complex) and charge you based on how many MIPS(Million instructions per second) the CEC has in CPU power.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

This, plus if you want to separate out main memory 100Gbps doesn't even get you started for a single processor never-mind a large group of them trying to share a memory pool.

Not saying Intel can't do it better(cheaper...?) but much, not all, of what is described here is what HP, Dell, Cisco, et al, already do in blade chassis.

Well, it can certainly be more resource efficient and more robust for day to day failures... BUT similarly to when you do this with, say, our food system then any troublesome thing that affects more than one instance of shared components will make the entire thing fail hard. With food that could be a pathogen, with servers it could be a bit rot bug, a virus, a problem with the interconnects...

Also tangentially, I wonder if this sort of tradeoff between latency and pooling would favor certain kinds of web app architectures over others, subtly influencing the entire web.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

Hah! I came here to post that too. Each foot of separation is at least two nanoseconds of latency (light travels about 1 foot/ns in a vacuum, less because it's not a vacuum, and 2x for roundtrip), plus electric->photonic and photonic->electric conversion overhead (2x also), if that is significant. As machines get faster, that will become more of a problem, not less. This seems like it is moving in the wrong direction.

On the other hand, servers are much more about throughput than latency, and with a hyperthreading type architecture or any other Niagara-type architecture where latency can be hidden by rotating between threads, maybe it doesn't matter as much. But I doubt current architectures could hide it adequately.

There are also some explorations in the industry about using serial channels between CPU and memory. These leverage the idea that most of the latency is in the memory chips and going serial only slightly increases latency, but the advantage of serial might be to allow a finite number of pins (which have maxed out around 1,000 per socket some years back) to run many more channels independently, a better match to the multi-core multi-threaded chips. But, this has not resulted in anything on the mainstream memory plans. Indeed, if anything the trend is going the opposite way with talk of ultrawide memory bus for stacked, adjacent memory modules (with just a few mm of distance from CPU to DRAM). These wide busses can run at relatively low clocks and low power.

Separating functionality is interesting though whether cloud services want to have replaceable units at fine granularity is an open question. When you have 100k or more servers in a site and many sites the trend is to go to minimal human intervention and maximum stability. For some the ideal is to install the servers in a container on a concrete pad, plug in power and networking, fire it up, then come back in a few years and switch it off, drive it away for recycling. Replacing stuff in mid-cycle is costly in terms of inventory, procedures, risk of collateral damage, and shorter use for the replacements. If we could get failures down to 5 or even 10% over 3 years, just let things fail in place. This ultrareplaceable modular idea may work better in smaller data centers.

I do, however, like the disaggregated approach just because of its flexibility. With maybe 3 or 4 SKUs, and a good fat network, you can adjust the balance of functionality to track shifting needs. If business turns out to be more storage intensive, or more compute intensive, simply place an order for more of the kind of SKU you want. If this allows capacity planning on shorter time cycles it is valuable. It may also allow different kinds of SKU to be rev'd independently, allowing for more competitive sourcing.

In a datacentre filled with cabinets, the vertical integration has been the power system located below and blades above. Each blade is rather independent except for the backplane that ties the other components storage plane, control-plane, network-plane. Horizontally across cabinets are tied by 10G-ethernet, potentially SAN cable connect.

Now to vertically tie in all connections via multiplex-switch-chip would be nice. Here the PCIe, LAN, Ram channels links to the 100G connector and that's it. Vertically integrated multiple blades into a horizontal plane with switch-fabric routing this elsewhere and rise up vertically as well. Yes, latency can be an issue so it would potentially limited by horizontal cable distance.Now we can see processor-array blades section, storage-array blades, Ram array blades. Network is integrated already in the vertical horizontal fabric!. Cooling can then be tuned to each section-blades to minimise power usage.This is simplification by grouping within a cabinet. To horizontalise the grouping, the switch fabric will need to form groupings and bind them. So it is possible to horizontalise until the cable limit is reached.

Its really nice how all this tech companies come up with more ideas to have more power while the world cannot solve so important needed to protect the Internet which are DDOS attacks. Its amazing that neither Intel, neither Cisco, neither Google or anyone can come up with a cost effective solution or technology to stop this. Today, it costs hundreds if not thousands of dollars to mitigate even a small attack and yet everyone is talking about web apps, cloud computing, putting your business online. There will be NO internet and no cloud, and no web apps, and not small business owners putting their business online or any evolution if any kid can take down a company server just because he can.

If Cisco, the ISP united, or even Intel, or anyone for that matter do not first take priority on this, they are not going to sell any servers anymore, and Cisco is not going to sell as many router or home switches anymore either.

My point is that everyone is so focused on technology and nobody is taking into account that the Internet is broken.I have meet dozen of persons that shut down their websites or Internet project because they were constantly targeted by attacks and did not even knew why, they were not willing to pay for this either and as opposed to Spamhaus nobody helped them.

ISP don´t care one bit about this, unless technology itself solves this, like one router telling another one to cut off traffic or server chips intelligently blocking illegal traffic I don´t know where all this HTML5, Cloud and Web apps is going.

There is no point on having stuff online if everyone else can take it offline and leave you in the dark.

Its really nice how all this tech companies come up with more ideas to have more power while the world cannot solve so important needed to protect the Internet which are DDOS attacks. Its amazing that neither Intel, neither Cisco, neither Google or anyone can come up with a cost effective solution or technology to stop this. Today, it costs hundreds if not thousands of dollars to mitigate even a small attack and yet everyone is talking about web apps, cloud computing, putting your business online. There will be NO internet and no cloud, and no web apps, and not small business owners putting their business online or any evolution if any kid can take down a company server just because he can.

If Cisco, the ISP united, or even Intel, or anyone for that matter do not first take priority on this, they are not going to sell any servers anymore, and Cisco is not going to sell as many router or home switches anymore either.

My point is that everyone is so focused on technology and nobody is taking into account that the Internet is broken.I have meet dozen of persons that shut down their websites or Internet project because they were constantly targeted by attacks and did not even knew why, they were not willing to pay for this either and as opposed to Spamhaus nobody helped them.

ISP don´t care one bit about this, unless technology itself solves this, like one router telling another one to cut off traffic or server chips intelligently blocking illegal traffic I don´t know where all this HTML5, Cloud and Web apps is going.

There is no point on having stuff online if everyone else can take it offline and leave you in the dark.

Not that you don't have a point, but how is this Intel's problem? Intel makes hardware. A lot of the tools to fix what's broken already exist, but companies are too lazy/stingy/risk averse to implement them. You think Intel should shut down their R&D division and shovel cash at those companies to get them working on fixing things?

For that matter, why don't we just shut down the internet right now, since skiddies are going to hack us all anyway? Oh, right, business absolutely relies on it now. And now that I think of it, I've got some servers that are positively elderly...

you'll rent time on the mainframe instead, being given a VM environment in which to work. I guess this is the premise of the future - cloud computing meaning massive server farms (of 2 or 3 mainframes) that give you more power than we have ever had when we used those old 1U servers.

This might help address the DDoS issue- if you're being attacked on your VM slice of a mainframe, then the ISP will be receiving all the network traffic and that will affect all their operations. They'll suddenly have an incentive to deal with this that doesn't just involve shutting you down. Current ISPs are smaller and don't have the kind of ability to deal with it, if we all shift to seriously-managed datacentres, then maybe they'll start to fix the internet.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

I'd imagine that, in practice(even if not in Intel's marketing glossies), system will end up being somewhat similar to the single-system-image cluster computing arrangements you can get today(albeit for downright exciting amounts of money).

A given CPU will still have fastest access to its own memory, and the further out you go the further into 'Well, it's easier than explicitly using MPI in your application, and faster than doing it over ethernet, right?' territory you are.

This isn't false, NUMA-aware systems aren't terribly new and being able to at least address memory anywhere in the entire cabinet would be a convenience feature; but I suspect that the actual module layouts of properly-designed systems will end up looking a lot more like a traditional stack-of-servers than Intel's powerpoint diagrams would suggest.

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

Once again, Intel is late to the party:

Any idea what all those are doing? At first I thought it was a bitcoin mining operation, but those GPUs aren't fat enough.

Intel is working to replace the traditional server rack with a more efficient architecture that separates CPU, storage, power, and networking resources into individual components that can be swapped out as needed.

Once again, Intel is late to the party:

Any idea what all those are doing? At first I thought it was a bitcoin mining operation, but those GPUs aren't fat enough.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.

And... that's OK for certain applications. But, you're right: the buyer should know exactly what kind of problem they are trying to solve and whether this architecture will work or not.

I see lots of talk about bandwidth in the article, but virtually none about latency. Intel seems to be talking about separating components by distances where even the speed of light delay between memory and processor can become significant.