Cray to plug Kepler GPUs into future Cascade supers

Cray's next-generation "Cascade" supercomputer is based on a future Intel "Ivy Bridge" Xeon processors, and the company has just taken $140m from Chipzilla in exchange for the intellectual property and people associated with its "Gemini" and "Aries" supercomputer interconnects. So it was no surprise that Cray talked about using Intel's Xeon Phi x86 coprocessors in the future Cascade machines, which lash tens of thousands of those future Xeons together using the Aries interconnect.

Now Nvidia is finally getting a little love for its "Kepler" GPU coprocessors inside the future Cascade machines. So if you have written code that runs in hybrid mode on AMD's Opteron processors and Nvidia's Tesla GPU coprocessors, as many supercomputing shops have begun doing, you can breathe a sigh of relief. You won't be forced to port your code to Xeon Phi coprocessors, formerly known as "Knights Corner."

The current Cray XK6 hybrid CPU-GPU supercomputers are what you get when you rip out half of the eight Opteron 6200 processors on a system blade, replace them with Tesla X2090 GPU coprocessors, and then link them to the remaining four Opterons.

Given that two of the largest supercomputers in the world – the 20 petaflops "Titan" super going into Oak Ridge National Laboratory and the similarly-sized "Blue Waters" super going into the University of Illinois – are using Nvidia's future "Kepler" GPUs in their coprocessors (and specifically, the Tesla K20 coprocessors aimed at jobs that need lots of double-precision floating point math), it was a pretty good bet that Cray was going to support Nvidia GPU coprocessors in Cascades. The US government labs paying for Cascade systems would not have it any other way.

Cray has been working under contract with US Defense Advanced Research Projects Agency for the past several years to build the Cascade system, which uses a new generation of high radix router interconnect called Aries that offers a substantial performance and scalability boost over the current Gemini XE interconnect used in Cray XE6 and XK6 supercomputers.

The Aries interconnect hooks supercomputer nodes through PCI-Express 3.0 buses instead of point-to-point interconnects for SMP and NUMA clustering such as Intel's QuickPath Interconnect or AMD's HyperTransport, and is therefore – in theory, at least – processor agnostic.

Cray has not, however, said that Cascade machines will support future Opterons. With x86 processors being more or less compatible, it may not have to, and that would be very bad news indeed for AMD. Perhaps bad enough for AMD to sue Cray and Intel if they are not careful, but you can't blame Cray for being miffed after several Opteron delays caused it much financial woe over the years.

Cray has sold a number of XE6 and XK6 systems with upgrades to Cascade machines, and sources at Cray tell El Reg that none of these Cascade deals thus far have had any Nvidia GPU coprocessors associated with them.

Cray is not saying which Nvidia GPUs it will support in the Kepler family, but it could support the Tesla K10, which has lots of single-precision oomph thanks to its GK104 GPU but very little double precision, as well as the K20, which it almost certainly has to support to keep the big supercomputer customers who are now investing in XK6 systems.

It will be interesting to see if Cray is going to be as flexible with processors as it is being with coprocessors. ®