Technology Lab —

The ten fastest supercomputers on the planet, in pictures

A Chinese supercomputer known as Tianhe-2 was today named the world's fastest machine, nearly doubling the previous speed record with its performance of 33.86 petaflops. Tianhe-2's ascendance was revealed in advance and was made official today with the release of the new Top 500 supercomputer list.

Tianhe-2 was developed at China's National University of Defense Technology and will be deployed in the country's National Supercomputing Center before the end of this year. "The surprise appearance of Tianhe-2, two years ahead of the expected deployment, marks China’s first return to the No. 1 position since November 2010, when Tianhe-1A was the top system," the Top 500 announcement states. "Tianhe-2 has 16,000 nodes, each with two Intel Xeon Ivy Bridge processors and three Xeon Phi processors for a combined total of 3,120,000 computing cores."

The combined performance of the 500 systems on the list is 223 petaflops, up from 162 petaflops in the previous list released six months ago. A petaflop represents one quadrillion floating point operations per second, or a million billion.

26 systems hit at least a petaflop. IBM's Blue Gene/Q accounted for four of the top 10, while Intel provided the processors for 80.4 percent of all Top 500 systems. 39 systems use Nvidia GPUs to speed up calculations, and another 15 use other accelerator or co-processor technology such as AMD's ATI Radeon and Intel's Xeon Phi.

252 of the 500 are installed in the US, 112 are in Europe, 66 are in China, and 30 are in Japan. The slowest computer on the list hit 96.6 teraflops, compared to 76.5 teraflops for the slowest computer on last November's list.

Besides Tianhe-2, the only new entrant in the top ten is a Blue Gene/Q system named Vulcan at Lawrence Livermore National Laboratory. Here is a look at the top ten:

Ranked #1 in November 2010, Tianhe-1A uses Intel Xeon CPUs and Nvidia GPUs across its 183,368 processing cores for a rating of 2.6 petaflops.

Using IBM iDataPlex servers, 300TB of RAM, and an InfiniBand interconnect, SuperMUC's 147,456 cores achieved a speed of 2.9 petaflops. Energy costs are cut by directly cooling chips and memory with water at unusually high temperatures of 104 degrees fahrenheit.

This 4.3 petaflop system is based on IBM's Blue Gene/Q supercomputing technology and has 393,216 cores. This supercomputer isn't devoted solely to government use; Vulcan was recently opened to industry and research universities for collaborative projects.

With Dell PowerEdge servers powered by Xeon processors and an InfiniBand interconnect, Stampede scored 5.2 petaflops. It is one of the largest systems in the world devoted to open science research—any researcher at a US institution can submit a request to use some of its computing power.

This Blue Gene/Q system uses 786,432 cores to hit 8.6 petaflops. When it hits full production in 2014 Mira will offer more than 5 billion computing hours per year to scientists (counting time on each core separately).

The world's #1 supercomputer in June 2012, Sequoia is used by the National Nuclear Security Administration to conduct simulations aimed at extending the lifespan of nuclear weapons. The Blue Gene/Q system has nearly 1.6 million cores and hits speeds of 17.2 petaflops.

Titan was #1 last November with a speed of 17.6 petaflops. The system uses AMD-based Cray CPUs and Nvidia GPUs in its 560,640 cores. Rated the third most energy efficient supercomputer last November, Titan uses 8,209 kilowatts of power.

Promoted Comments

It's interesting just how "art-like" those supercomputers look. The "K-computer" also looked like it uses heat-pipes.

Edit: forgot to ask, who exactly does the engineering and building of these supercomputers?

1: they all use heat pipes of some sort or another. K-computer is all liquid-cooled, so what you're seeing is one board's worth of CPU+memory+etc cooling. It plugs into the racks cooling lines, then up farther.

Engineering varies, because different companies/groups provide different parts of each each system. For example, the Cray systems all have multiple cabinets for the compute, which come with their own propriatary interconnects between all the nodes/cores/ranks/etc, they then connect to an IB network which connects to a storage system through interface nodes (their only job is to talk to the "outside" world).

The IB (or other external high speed network) is made by one of a few companies. It's used to talk to the storage (which again, is a different set of companies for the hardware+software)

Then there are the utilities to manage what is running where, monitoring, fault-detection, importing and exporting data from the cluster, etc.

... but it's a *really* small world. Working in the industry, I recognize several of the people pictured from conferences.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

For quite some time, supercomputers have been giant clusters, operating in unison. For a while now, one of the big limitations of supercomputers has been interconnect technology, the idea being that when you get high enough speed between nodes, you can start treating the whole thing as one big system. You'll see Infiniband mentioned in the article, that's the most mainstream (and I believe most common) of the 'networks' tying everything together, letting you do things like *remote* DMA, which helps you make everything look like one single many-cored computer. The discussion about n-dimensional torus topology elsewhere in the comments is just more about how to connect thousands of nodes together in a high-speed, low-latency network.

The second benefit of high-speed clustering is you can (sort of) lash together commodity hardware into a supercomputer. The K Computer looks pretty 'custom', a purpose-built supercomputer, while things like Stampede and the Chinese computers look pretty 'commodity', with the IBM systems leaning a bit more the 'custom' way. ...anyway I went on too long already. Rambling.

More impressive, to me at least, is The GREEN 500, a list ranked based on FLOPS per watt. It's not out yet, but usually published within a month or so of the TOP500. I highly doubt the Tianhe-2 will top that list. It's easy enough to crank up the core count, but it takes a special bit of engineering to design something that's efficient, too. Number 1 from last November's GREEN500 ranked a mere 253rd on the TOP500-- Beacon @ the National Institute for Computational Sciences, University of Tennessee. Tianhe-1A only came in at 106rd on the GREEN500, though Titan came in 3rd.

If processing power keeps expanding at the same rate it has been, 40-50 years for a handheld device, I'd guess. There has to be a few major breakthroughs in semi-conductors before that can happen though, we're supposedly starting to hit the limits of what we can do with silicon and you can only shrink it down so much before there is too much electrical spillage.

If processing power keeps expanding at the same rate it has been, 40-50 years for a handheld device, I'd guess. There has to be a few major breakthroughs in semi-conductors before that can happen though, we're supposedly starting to hit the limits of what we can do with silicon and you can only shrink it down so much before there is too much electrical spillage.

That's not the only thing. Note the power usage of one of those units. Never mind the heat.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

It's interesting just how "art-like" those supercomputers look. The "K-computer" also looked like it uses heat-pipes.

Edit: forgot to ask, who exactly does the engineering and building of these supercomputers?

1: they all use heat pipes of some sort or another. K-computer is all liquid-cooled, so what you're seeing is one board's worth of CPU+memory+etc cooling. It plugs into the racks cooling lines, then up farther.

Engineering varies, because different companies/groups provide different parts of each each system. For example, the Cray systems all have multiple cabinets for the compute, which come with their own propriatary interconnects between all the nodes/cores/ranks/etc, they then connect to an IB network which connects to a storage system through interface nodes (their only job is to talk to the "outside" world).

The IB (or other external high speed network) is made by one of a few companies. It's used to talk to the storage (which again, is a different set of companies for the hardware+software)

Then there are the utilities to manage what is running where, monitoring, fault-detection, importing and exporting data from the cluster, etc.

... but it's a *really* small world. Working in the industry, I recognize several of the people pictured from conferences.

Can someone explain to me how they use hot water to cool a super computer? I've seen it mentioned in quite a few articles but I've never understood why they use such hot water instead of water at ambient temp or even lower (though I assume if it was lower, you'd run into condensation issues and you wouldn't want that)? Or hell, even a nice article that offers quick summation would suffice.

Power consumption or alternatively energy efficiency is becoming a real driver for continued operation of these supercomputers. For example the 2009 Roadrunner system (first petaflop machine) was dismantled partially because of the power bill. See: http://arstechnica.com/information-tech ... ismantled/

Can someone explain to me how they use hot water to cool a super computer? I've seen it mentioned in quite a few articles but I've never understood why they use such hot water instead of water at ambient temp or even lower (though I assume if it was lower, you'd run into condensation issues and you wouldn't want that)? Or hell, even a nice article that offers quick summation would suffice.

The new LRZ SuperMUC system was built with IBM System x iDataPlex Direct Water Cooled dx360 M4 servers. IBM’s hot-water cooling technology directly cools active components in the system such as processors and memory modules with coolant temperatures that can reach as high as 113 degrees Fahrenheit.

By bringing the cooling directly to components, SuperMUC allows an increased inlet temperature. “It is easily possible to provide water having up to 40 degrees Celsius using simple ‘free-cooling’ equipment, as outside temperatures in Germany hardly ever exceed 35 degrees Celsius,” LRZ says. “At the same time the outlet water can be made quite hot (up to 70 degrees Celsius) and re-used in other technical processes – for example to heat buildings or in other technical processes.”

SuperMUC is based on the liquid cooling system developed for the Aquasar supercomputer at the Swiss Federal Institute of Technology Zurich (ETH) in 20120. The cooling system system features a system of capillary-like pipes that bring coolant to the components, remove the heat, and than are returned to a passive cooling system that uses fresh air to cool the water. IBM has a video providing additioal infromation on the SuperMUC cooling system.

In a two dimensional torus interconnect, the nodes are imagined laid out in a two dimensional rectangular lattice of rows and columns, with each node connected to its 4 nearest neighbors, and corresponding nodes on opposite edges connected. The connection of opposite edges can be visualized by rolling the rectangular array into a "tube" to connect two opposite edges and then bending the "tube" into a torus to connect the other two.

In a three dimensional torus interconnect the nodes are imagined in a three dimensional lattice in the shape of a rectangular prism, with each node connected with its 6 neighbors, with corresponding nodes on opposing faces of the array connected.

Higher dimensional arrays can't be directly visualized, but each higher dimension adds another pair of nearest neighbor connections to each node.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

Look at Jon Brodkins links above, as an example. What you're saying is pretty much how it works (well... to my understanding anyway), but it's not exactly a bunch of desktops wired together with CAT6.

I would like to know where the NSA computers would fall on this list. Has to be near the top to crunch all the data they collect. They should at least tell us about their computing capacity.

Yes they should tell us about all their technology because you would like to know. Regardless of any of the merits and right/wrong of what is going on in the news right now, the simple fact is that these govt agencies need to operate under various levels of secrecy. If people think the US is the only civilized connected nation that operates like this, then they are living in a dream land.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

For quite some time, supercomputers have been giant clusters, operating in unison. For a while now, one of the big limitations of supercomputers has been interconnect technology, the idea being that when you get high enough speed between nodes, you can start treating the whole thing as one big system. You'll see Infiniband mentioned in the article, that's the most mainstream (and I believe most common) of the 'networks' tying everything together, letting you do things like *remote* DMA, which helps you make everything look like one single many-cored computer. The discussion about n-dimensional torus topology elsewhere in the comments is just more about how to connect thousands of nodes together in a high-speed, low-latency network.

The second benefit of high-speed clustering is you can (sort of) lash together commodity hardware into a supercomputer. The K Computer looks pretty 'custom', a purpose-built supercomputer, while things like Stampede and the Chinese computers look pretty 'commodity', with the IBM systems leaning a bit more the 'custom' way. ...anyway I went on too long already. Rambling.

Over three million cores... Looks to me more like a huge network, rather than a 'computer'. Particularly if you note that most of the cores have a space-like separation-- i.e., there is no (and cannot be any) actual causal connection between most of the cores.

I'd call it cheating, but at the end of the day the results are the results no matter how you get there. And lets be honest. China has a known history of using the bruit force approach. See three gorges dam as a perfect example.

or if you skip all the terminology and simplify it a little bit , a torus is basically a coordinate system that loops back on it self. so what they call six-dimensional torus interconnect , is actually a network where point A connect to B then C then … and then A again. they do this in the x - direction (up/down the rack) , y (left/right neighboring rack ), z (back/forward neighboring rack) in the computer cluster. thats "3 -dimension" , what the extra 3d comes from is that every "point" in this 3d mesh is a node(sub unit (3 processing board high)(deep/wide)) that also have a 3d mesh topology .

and instead of saying all that they get to call it a six-dimensional torus interconnect