Every six months, Earth’s biggest supercomputers have a giant race to see which can lay claim to being the world’s fastest high-performance computing cluster.

In the latest Top 500 Supercomputer Sites list unveiled Monday morning, a newly assembled cluster built with IBM hardware at the Department of Energy’s Lawrence Livermore National Laboratory (LLNL) takes the top prize. Its speed? A whopping 16.32 petaflops, or 16 thousand trillion calculations per second. With 96 racks, 98,304 compute nodes, 1.6 million cores, and 1.6 petabytes of memory across 4,500 square feet, the IBM Blue Gene/Q system installed at LLNL overtakes the 10-petaflop, 705,000-core “K computer” in Japan's RIKEN Advanced Institute for Computational Science.

The Japanese computer had been world’s fastest twice in a row. Before that, the top spot was held by a Chinese system. The DOE computer, named “Sequoia,” was delivered to LLNL between January and April. It's the first US system to be ranked #1 since November 2009.

To get to 16 petaflops, Sequoia ran the Linpack benchmark for 23 hours without a single core failing, LLNL division leader Kim Cupps told Ars Friday in advance of the list’s release. The system is capable of hitting more than 20 petaflops—during the tests it ran at 81 percent efficiency.

“For a machine with 1.6 million cores to run for over 23 hours six weeks after the last rack arrived on our floor is nothing short of amazing,” she said.

The cluster is extremely efficient for one so large, with 7,890 kilowatts of power, compared to 12,659 kilowatts for the second-best K Computer. It’s primarily cooled by water running through tiny copper pipes encircling the node cards. Each card holds 32 chips, with each chip having 16 cores.

The entire cluster is Linux-based. Compute Node Linux is run on nearly 98,000 nodes, and Red Hat Enterprise Linux runs on 768 I/O nodes which connect to the file system, Cupps said.

To start, the cluster is on a relatively open network, allowing many scientists to use it. But after IBM’s debugging process is over around February 2013, the cluster will be moved to a classified network that isn’t open to academics or outside organizations. At that point, it will be devoted almost exclusively to simulations aimed at extending the lifespan of nuclear weapons.

“The kind of science we need to do is lifetime extension programs for nuclear weapons,” Cupps said. “That requires suites of codes running. What we’re able to do on this machine is to run large numbers of calculations simultaneously on the machine. You can turn many knobs in a short amount of time.”

In November 2011's Top 500 list, three of the top five clusters used NVIDIA GPUs (graphics processing units) in combination with CPUs to achieve very high speeds. This time around, only one of the top five integrates GPUs, although the overall number in the Top 500 integrating GPUs or similar accelerators rose from 39 to 58.

The use of GPUs in supercomputing tends to be experimental so far, said Dave Turek, IBM vice president of high performance computing. “The objective of this is to do real science,” he said. GPUs are a bit more difficult to program for, he said.

While the majority of Top 500 computers use Ethernet or Infiniband as their primary interconnects, Sequoia uses IBM’s proprietary 5D Torus. It's an optical network that provides 40 Gbps throughput to IBM’s Blue Gene/Q clusters. I/O nodes are connected to the file system via Infiniband and the management network uses Ethernet, Cupps said.

IBM leads the Top 500 list with 213 systems, ahead of HP’s 138. Nearly 80 percent—372 of the 500 systems—use Intel processors, followed by 63 using AMD Operton and 58 using IBM Power.

Three DOE systems are in the top 10. The rest hail from Japan, Germany, China, Italy, and France. All 10 have performance of at least at least 1.27 petaflops.

Petascale computers have become relatively commonplace since the IBM Roadrunner system at Los Alamos National Laboratory was the first to hit a petaflop in 2008. In fact, each of the top 20 systems on the new list hit at least a petaflop. Exascale, which would be 1,000 times faster, is the next big breakthrough for the IBMs, HPs, and Crays of the world to aspire to.

But a big advance in price-performance is necessary. Today’s technology could scale up a lot higher—it just wouldn’t be practical. Supercomputers are naturally expensive (even more expensive than the new MacBook Pro). The K Computer in Japan, for example, cost more than $1 billion to build and $10 million to operate each year. Livermore told us it spent roughly $250 million on Sequoia.

“We could get another order of magnitude with this technology if someone would write a check,” Turek said. “But no one would want to write that check.”

95 Reader Comments

It's not really that impressive that the numbers keep getting bigger—someone writes a bigger check and wham, you've got a machine with more processors. What I'd like to see is how many FLOPS per watt these machines can do. Improving *that* would actually be impressive.

It's not really that impressive that the numbers keep getting bigger—someone writes a bigger check and wham, you've got a machine with more processors. What I'd like to see is how many FLOPS per watt these machines can do. Improving *that* would actually be impressive.

16.32 petaflops, 7,890 kilowatts of power. While the second fastest one does ~10 petaflops for 12,659 kilowatts.

It's not really that impressive that the numbers keep getting bigger—someone writes a bigger check and wham, you've got a machine with more processors. What I'd like to see is how many FLOPS per watt these machines can do. Improving *that* would actually be impressive.

Take a look at the Green 500 list. The most current one is from last Nov but should be soon updated.

Meanwhile, a few scientists will be the beta testers. After that phase, they'll be locked out. haha! Not even Microsoft is so sneaky. J/K

Why evil? Nuclears exist, and if they can extend the life of the pre-existing ones, that means less new ones, I'd suppose. Also, I don't think this will only apply to weapons... What about Power Plants? Unless you mindlessly consider 'evil' generally anything related to nuclear energy...

Geopolitical maneuvering is such a waste of this machine. Imagine what we could learn with 16-20 petaflops of additional power devoted to genome sequencing, climate or social modelling, or any other academic pursuit.

It's not really that impressive that the numbers keep getting bigger—someone writes a bigger check and wham, you've got a machine with more processors. What I'd like to see is how many FLOPS per watt these machines can do. Improving *that* would actually be impressive.

I agree.. Anyone can stuff a couple of rooms full of CPUs. It's not very elegant.

I miss the days when computers were still evolving and getting faster.. Now it's just Moar Coars!, both in the consumer space and in enterprise computing.

Your taxpayer money being wasted on something that in the long term gives us precisely nothing as an end result. All that energy, engineering talent, hardware resources being spent on making sure that something will still go boom when asked to do so, which in a sane world would be never.Hey, $100 million here, another $100 million there, it just money right? Never mind that it could be better spent strengthening our declining educational or health standards.

Sorry if I'm being really dumb but has any one got a better image of how the 32 "Compute Card" instances fit into the "Node Card"?

I just can't visualize it. Is it actually 2 compute cards (thus 32 *cores*) or is the scale of the photos just messing with my head?

it's the scale of the small photos messing with you.if you check out the header photo where the technician loads a compute card into a node card you'll see the true scale of the things. note that he's loading them into a node card sitting in a midplane, from which are two per rack.

“The kind of science we need to do is lifetime extension programs for nuclear weapons,” Cupps said

What does this mean? My layman's impression is fissile materials are sitting around in nuclear weapons. Over time, they degrade through natural radioactivity. Minimal quantities were used to start with (hey, it's expensive!), so at some point there will no longer be enough to cause a nuclear explosion.

The decay curve is extremely well known, and it's simple mathematics. There are only a few types of nuclear weapons / missiles etc. So why is a supercomputer required to calculate this stuff?

if you check out the header photo where the technician loads a compute card into a node card you'll see the true scale of the things. note that he's loading them into a node card sitting in a midplane, from which are two per rack.

Ah right, the 'sequence' photo just doesn't show the compute cards in the node card then?

The decay curve is extremely well known, and it's simple mathematics. There are only a few types of nuclear weapons / missiles etc. So why is a supercomputer required to calculate this stuff?

What is not known, is how will such a partially-decayed weapon perform, and when has it decayed so much that it no longer works. "Works" in this case means "explodes as and when it is designed to do so, and with a predictable yield". As the original plutonium/tritium/whatever decays, the products of that decay build up and alter the characteristics.

Having a lot of old nukes has not been done before. In the era of active nuke development and testing, old weapons were recycled/rebuilt into new ones, and those were tested to see how they performed. It's a long time since the test-ban treaty, and since then no-one has done any (significant) testing - thus we don't know.

Ideally we'd of course scrap the lot, but I don't think the nuclear-weapon genie can be stuffed back into its bottle. So the next-best thing is to simulate, if you don't want to restart actual testing. And while the math behind radioactive decay is simple in principle, the actual physics of a nuclear detonation in a partially decayed warhead are anything but - so the model has to be extremely detailed, which in turn raises the number of computations through the roof.

All in all, I'd say that $200m is a small amount, well spent, considering the subject. "Gee, I wonder if those nukes will actually work" is a potentially very expensive state of mind to be in :-)

“The kind of science we need to do is lifetime extension programs for nuclear weapons,” Cupps said

What does this mean? My layman's impression is fissile materials are sitting around in nuclear weapons. Over time, they degrade through natural radioactivity. Minimal quantities were used to start with (hey, it's expensive!), so at some point there will no longer be enough to cause a nuclear explosion.

The decay curve is extremely well known, and it's simple mathematics. There are only a few types of nuclear weapons / missiles etc. So why is a supercomputer required to calculate this stuff?

Its not the whole warhead that degrades just the triggers (deuterium or tritium - Lithium deuteride).Well the whole thing does decay but the triggers go a lot lot faster.

The decay curve is extremely well known, and it's simple mathematics. There are only a few types of nuclear weapons / missiles etc. So why is a supercomputer required to calculate this stuff?

What is not known, is how will such a partially-decayed weapon perform, and when has it decayed so much that it no longer works. "Works" in this case means "explodes as and when it is designed to do so, and with a predictable yield". As the original plutonium/tritium/whatever decays, the products of that decay build up and alter the characteristics.

...

All in all, I'd say that $200m is a small amount, well spent, considering the subject. "Gee, I wonder if those nukes will actually work" is a potentially very expensive state of mind to be in :-)

Or on the flip side, it reminds me of Dirty Harry, "I've got all these nuclear bombs and in all this excitement I've forgotten how old they are. Well do you feel lucky?".

What surprises me about this is the cost. Only $200 million. Yes that's a big 'only' but still it's easily achievable by the heavyweights of the corporate world. Intel's latest net profit was $2.74 billion - what do you think they've got to spend on simulations of future chips. What have Microsoft and Google got tucked away crunching the next big thing? What has big pharma got crunching away at proteins and compounds looking for the next Billion dollar patent?

I understand a supercomputer is completely different as it is tightly integrated into one cohesive system - but I'd like to see how it compares to the compute capacity of major corporations.

Nuclear decay simulations are a worthy goal for that drives supercomputer research but all the whiners can keep this in mind:

What does Sequoia do for you?

It gives IBM a place to dump 1.6 million 45nm PowerPC cores. The same core the Wii U is going to use. These big budget supercomputer orders help drive down processor costs by giving chip fabs early customers with deep pockets who can tolerate the initial high defect rate involved with making new chip designs. Sequoia's initial workload will be less about doing actual research and more about giving the system a good shake to see if any more processors go defective under thermal load. IBM gets to have close access to a huge pool of processors that they can get back to study when they go boom.

Tritium half-life equals 12.32 years. That's why they need the massive compute power. Combined with the half-life of the fissile material, yield of the weapon becomes a complex equation. Also have to consider the aging of the non-radioactive trigger explosives. Constant exposure to air and radioactivity probably has some sort of effect. Long-term exposure of metal to radioactivity usually makes most metal brittle. Most missile delivered weapons also use uranium metal as the ablative surface for the warhead on re-entry. Warheads are precisely shaped to be predictable on flight paths, especially the MIRVed ones. Ya gotta do the math on that as well. The Strangelove H-bombs don't really exist. Grim engineering, lots of variables.