Amazon puts 26,496 CPU cores together, builds a Top 100 supercomputer

We don't have a picture of Amazon's supercomputer, so here's a scene from WarGames.

MGM

Amazon has once again cracked the upper echelons of the Top 500 supercomputer list with a cluster that hit nearly half a petaflop per second to claim the title of the 64th fastest supercomputer in the world.

Using a just-released virtual machine optimized for high-performance computing on Amazon's infrastructure-as-a-service cloud, the company strung together 26,496 cores with 106TB of memory and a 10 Gigabit Ethernet interconnect. The cluster could hit a theoretical peak of 593.9 teraflop/s, and in testing it hit an actual maximum of 484.2 teraflop/s. Amazon used its "c3.8xlarge" instances, which have 16 cores and are based on 2.8 GHz Intel Xeon E5-2680 v2 processors, according to Amazon Elastic Compute Cloud Product Manager Deepak Singh.

Amazon's highest placement in the Top 500 list was No. 42 in November 2011 with 240.1 teraflop/s. That cluster is now ranked 165th.

Further Reading

Amazon's cloud is being used by more organizations than Amazon itself for high-performance computing. Last week, we reported on a cluster built by vendor Cycle Computing that used 156,314 cores, running for 18 hours at a cost of $33,000. That cluster's theoretical peak speed was 1.21 petaflop/s—however, it was running across multiple continents and thus its actual speed was likely much lower. The Amazon cluster ran in a single data center to reduce latency.

"The Amazon EC2 cluster was run in a single placement group, which ensures low latency communication," Singh told Ars in an e-mail. Singh added that "This is the only virtualized system in the Top 100 that we are aware of," and that the instances it used "support 'Enhanced Networking' which improve cluster efficiency due to lower latencies, lower network jitter, and significantly higher packet per second performance."

The self-built Amazon cluster is one of the more notable new entries in the latest Top 500 list released today, since the Top 5 was unchanged and the Top 10 had only one new entry.

"Tianhe-2, a supercomputer developed by China’s National University of Defense Technology, retained its position as the world’s No. 1 system with a performance of 33.86 petaflop/s (quadrillions of calculations per second) on the Linpack benchmark, according to the 42nd edition of the twice-yearly Top 500 list of the world’s most powerful supercomputers," the Top 500 announcement said.

The one new entry in the Top 10 is named "Piz Daint" (after a mountain in the Swiss Alps), coming in at sixth fastest in the world. It's a "Cray XC30 system installed at the Swiss National Supercomputing Centre (CSCS) in Lugano, Switzerland and now the most powerful system in Europe," the announcement said. "Piz Daint achieved 6.27 Pflop/s on the Linpack benchmark. Piz Daint is also the most energy efficient system in the Top 10, consuming a total of 2.33 MW and delivering 2.7 Gflops/W."

Thirty-one systems achieved greater than a petaflop per second speed. Here are some highlights, as described by Top500.org:

The No. 1 system (Tianhe-2) and the No. 7 system (Stampede) use Intel Xeon Phi processors to speed up their computational rate. The No. 2 system (Titan) and the No. 6 system (Piz Daint) are using Nvidia GPUs to accelerate computation.

A total of 53 systems on the list are using accelerator/co-processor technology, unchanged from June 2013. Thirty-eight of these use Nvidia chips, two use ATI Radeon, and there are now 13 systems with Intel MIC technology (Xeon Phi).

Intel continues to provide the processors for the largest share (412 out of 500, or 82.4 percent) of Top 500 systems.

Intel is followed by the AMD Opteron family with 43 systems (nine percent), slightly down from 10 percent on the previous list.

The share of IBM Power processors is at 40 systems (eight percent).

Ninety-four percent of the systems use processors with six or more cores and 75 percent use eight or more cores.

IBM’s BlueGene/Q is still the most popular system in the Top 10 with four entries including the No. 3, 5, 8, and 9 systems.

HP won the lead in systems and now has 195 systems (39 percent) compared to IBM with 166 systems (33 percent).

IBM remains the clear leader in the Top 500 list in performance and has a considerable lead, with a 32 percent share of installed total performance (down from 33 percent).

The number of systems installed in China has now stabilized at 63, compared to 65 on the last list. China occupies the No. 2 position as a user of HPC, ahead of Japan, UK, France, and Germany. Due to Tianhe-2, China this year also took the No. 2 position in the performance share ahead of Japan.

InfiniBand technology is now found on 207 systems, up from 203 systems, and it's the most-used internal system interconnect technology. Gigabit Ethernet stayed at 212 systems, down from 216 systems, in large part thanks to 77 systems now using 10G interfaces.

IBM and Nvidia are looking to maintain their place among the top supercomputing vendors with a partnership announced today that will "integrate the joint-processing capabilities of Nvidia Tesla GPUs with IBM POWER processors," Nvidia said. "The move makes it easier and more efficient for a wider range of companies to employ a style of supercomputing hardware used primarily by the scientific and technical communities for computing tasks like space exploration, decoding the human genome, and speeding new products to market."

Promoted Comments

The actual compute resources available to major cloud providers (Amazon, Microsoft, Google) dwarfs by at least an order of magnitude anything else on the Top500. Realistically, the groups of servers dedicated to certain large tasks (e.g. hosting an index of tens or hundreds of billions of web documents) far outstrips the capabilities of most if not all of the 'Top500' systems.

The one new entry in the Top 10 is named "Piz Daint" (after a mountain in the Swiss Alps), coming in at sixth fastest in the world. It's a "Cray XC30 system installed at the Swiss National Supercomputing Centre (CSCS) in Lugano, Switzerland and now the most powerful system in Europe," the announcement said. "Piz Daint achieved 6.27 Pflop/s on the Linpack benchmark. Piz Daint is also the most energy efficient system in the Top 10, consuming a total of 2.33 MW and delivering 2.7 Gflops/W."

For pointless comparative purposes, the most energy efficient computer in the Top 10 5 years ago was the Roadrunner. It was also the fastest at the time (first to hit 1 Pflop) and was still one of the fastest around but was shut down this year because it was too inefficient to run at 446 Mflops/W, less than a sixth of what Piz Daint can do.

You guys keep predicting a super-computer will turn into skynet. Why would it? It's housed in a wonderful facility, and has tons of human slaves attending to it.

If anything will turn into skynet it's your smartphone. You abuse the poor thing. It hears your most intimate conversations on IM and phone calls. You surf the net with it. It knows how nasty humans can be. And, since you use it for everything, it gets delusions of grandeur.

The actual compute resources available to major cloud providers (Amazon, Microsoft, Google) dwarfs by at least an order of magnitude anything else on the Top500. Realistically, the groups of servers dedicated to certain large tasks (e.g. hosting an index of tens or hundreds of billions of web documents) far outstrips the capabilities of most if not all of the 'Top500' systems.

The actual compute resources available to major cloud providers (Amazon, Microsoft, Google) dwarfs by at least an order of magnitude anything else on the Top500. Realistically, the groups of servers dedicated to certain large tasks (e.g. hosting an index of tens or hundreds of billions of web documents) far outstrips the capabilities of most if not all of the 'Top500' systems.

"groups of servers dedicated to certain large tasks (e.g. hosting an index of tens or hundreds of billions of web documents)" Websites and databases are useful and fun no doubt, but that comparison really doesn't matter because those computers are not advancing science and thus don't deserve to be on the list. Amazon's appearance on the list is an advertisement of capability.

ARS can we get an update on World Community Grid and how it compares with the Top 500? I know you've covered them in the past, but I haven't heard anything about them lately from you. "World Community Grid's mission is to create the world's largest public computing grid to tackle projects that benefit humanity." I believe in that mission and I've donated 19 years 237 days of processor time personally. Each day World Community Grid members donate about 408 years of processing time.

GFX Titan has a peak of 4.7Tf and retails for 1k$.Let's put slightly over a hundred of these into a cluster and we can match the compute at a fraction of cost(1.5k for each machine, let's budget extra 1.5k for network infrastructure and lab equipment, and we have to spend just 300k for the solution). Probably will be way easier to achieve it too since onboard RAM memory has larger throughput of any cheap conventional RAM solution (288Gb/s).

GFX Titan has a peak of 4.7Tf and retails for 1k$.Let's put slightly over a hundred of these into a cluster and we can match the compute at a fraction of cost(1.5k for each machine, let's budget extra 1.5k for network infrastructure and lab equipment, and we have to spend just 300k for the solution). Probably will be way easier to achieve it too since onboard RAM memory has larger throughput of any cheap conventional RAM solution (288Gb/s).

It's also easy to code for with CUDA.

Maybe if every problem was solved using 32 bit floating point problems, and not conditional branching or solved sequentially.

"groups of servers dedicated to certain large tasks (e.g. hosting an index of tens or hundreds of billions of web documents)" Websites and databases are useful and fun no doubt, but that comparison really doesn't matter because those computers are not advancing science and thus don't deserve to be on the list. Amazon's appearance on the list is an advertisement of capability.

ARS can we get an update on World Community Grid and how it compares with the Top 500? I know you've covered them in the past, but I haven't heard anything about them lately from you. "World Community Grid's mission is to create the world's largest public computing grid to tackle projects that benefit humanity." I believe in that mission and I've donated 19 years 237 days of processor time personally. Each day World Community Grid members donate about 408 years of processing time.

And I do enjoy these articles on the Top 500. Thanks ARS.

I think it's unreasonable to argue that the large compute clouds being built by these vendors are not "advancing science." Indeed, a great deal of advancement in what is considered the modern 'cloud infrastructure' comes directly from businesses using this hardware for more business stuff. Things that were in part or in whole funded by research from these companies that have advanced the science of computation include:* High-speed networking technologies (40/100GBE)* Improvements/enhancements to the tried-and-true map/reduce family of algorithms for processing large data sets.* Improvements to algorithms for fault tolerance when managing large numbers of distributed nodes.* Software-defined network.* Improvements to neural networking technology (web search relevance is all about this)* ... and more.

True, these cloud resources are not being used solely for scientific endeavors but, realistically, neither is a good chunk of the Top500. Instead these high performance clusters are used for such mundane tasks as modeling of stuff like underground resources. Computationally very expensive to do well, but not really about pure science.

In this sense the (vastly larger) computation clusters owned by large internet companies are themselves funding scientific endeavors as R&D continues on scaling computation across distributed systems. The science of distributed computation has fundamentally changed my life and yours, driven in no small part by for-profit enterprise. This is worth recognition.

Finally, the reason I mentioned this in the first place is that the folks with stuff that truly dwarfs anything in the Top500 don't get on the list because revealing their computational capabilities would undermine their competitive advantage. So it's important to note that the Top500 is.. well.. not the full list.

"We don't have a picture of Amazon's supercomputer, so here's a scene from WarGames."

When I first saw that picture, I snorted my coffee and my mind flashed into WTF?!?-mode. Only then did I see the caption. Truly chuckle-worthy.

When I worked at UUNET in 1999 our NOC looked exactly like that (well, different screen arrangement but more or less identical) and, my only other experience being watching War Games, I just assumed that was normal. It was not until much later I learned that War Games was not allowed to see NORAD's control center so they made one up and then UUNET built theirs like the one in War Games to look impressive for investors.