Liquid-Cooled Supercomputers, to Trim the Power Bill

Satoshi Matsuoka, project leader, with the Tsubame KFC. More powerful than an older supercomputer, it uses no more energy.

Jeremie Souteyrat for The New York Times

By ERIC PFANNER

February 11, 2014

TOKYO — Dropping a home computer into a vat of liquid would wreck it.

Yet some operators of supercomputers are submerging their machines in liquids, without causing any apparent damage, to keep them from overheating. Advocates say so-called immersion or submersion cooling could solve one of the biggest challenges of the digital economy: reducing the air-conditioning bills and environmental strains of the power-hungry servers and supercomputers that crunch ever-rising mountains of data.

A prototype supercomputer at the Tokyo Institute of Technology, which is submerged in a tank of mineral oil, was named in November in an industry ranking, the Green500, as the most energy-efficient machine of its kind. The computer, called Tsubame KFC, is 50 percent more powerful than an older supercomputer at the institute but uses the same amount of energy.

“The university administration said, ‘You’re not going to get any more power,’ ” said Satoshi Matsuoka, the project leader. “But we still wanted more performance.”

Japan is driven to reduce electricity consumption since the March 2011 earthquake and tsunami that caused a meltdown at the Fukushima Daiichi power plant. The country idled other nuclear facilities across the country, cutting electricity generating capacity sharply.

The use of oil or other liquids, rather than air-conditioning, to cool supercomputers and powerful servers is not limited to Japan. The technology used by the Tokyo institute was developed by a company in Austin, Tex., called Green Revolution Cooling.

Elsewhere, engineers are experimenting with different liquids to accomplish a similar task. Iceotope, a start-up based in Sheffield, England, submerges computers in liquid fluoroplastic, rather than oil. The company, whose clients include universities in Britain and Poland, in January raised 6 million pounds, or about $10 million, from venture capital investors backed by Schneider Electric, one of the biggest providers of data center cooling technology.

In Hong Kong, a company called Allied Control, which designs cooling systems, used submersion technology for a recently opened data center. The facility is operated by Asicminer, a company that works with the virtual currency Bitcoin, solving the algorithms that create the currency.

Unlike water, neither mineral oil nor the liquid fluoroplastics used by Iceotope and Allied Control conduct electricity. Therefore, experts say, there is no risk of short-circuiting the equipment or damaging it in other ways.

Peter Hopton, chief executive of Iceotope, said data centers that use air-conditioning could cut their energy bills and infrastructure costs in half by using submersive cooling. Not only do facilities that are cooled this way use less electricity, they also eliminate or reduce the need for costly air-conditioning and filtering equipment.

“We’re not talking about small numbers here,” he said. “We’re talking about millions in savings, every year.”

Some supercomputers consume tens of millions of dollars worth of energy annually, while the biggest corporate data centers have electricity bills of hundreds of millions of dollars, much of that going for air-conditioning.

Water is already used widely as a coolant in data centers and supercomputers, but usually it is piped through the facilities or the machines.

Piping water through a supercomputer or a data center is one thing. Submerging the devices in liquid is another.

Cray, the American supercomputer maker, used submersion cooling for one of its machines in the 1980s, earning it the nickname Bubbles. But the method was not widely used, in part because of concern about the high cost and the ozone-depleting effects of the coolants of that era.

Iceotope and the Hong Kong system use new kinds of fluids, including one from 3M called Novec 1230. 3M says it is not ozone-depleting and has been exempted from the list of volatile organic compounds issued by the United States Environmental Protection Agency.

Mr. Hopton said Novec, which he described as “liquid Teflon,” provided more cooling power than mineral oil, the fluid used by Green Revolution.

Christiaan Best, chief executive of Green Revolution, countered that mineral oil was almost as effective and considerably less expensive. Mineral oil costs less than $10 a gallon, while Novec 1230, which is more commonly used in sophisticated fire extinguishing equipment, costs about $70 a gallon, he said.

To test the viability of the new system, the Tokyo institute erected a small building, little more than a shed, to house Tsubame KFC just outside a much larger lab that contains an existing supercomputer.

Putting it next to a big heat-producing supercomputer gives the academics a chance to determine whether Tsubame KFC can function even when the oil warms up to summertime temperatures of 95 degrees Fahrenheit. They think it might be possible because modern semiconductors can operate at higher temperatures than their forerunners, Professor Matsuoka said.

Only a few modifications were necessary to enable Tsubame KFC to be submerged, he said. Moving parts like hard drives and fans had to be removed, for example.

Green Revolution’s mineral oil cooling method has already been used at several data centers, including facilities operated by the United States Department of Defense. Intel conducted a one-year study of the system and found that its servers had suffered no adverse effects, and power consumption was cut sharply.

Japanese computer experts consider energy efficiency an area in which their country’s supercomputer industry, long a source of national pride, can claim a technological lead. Other countries, like China, have rivaled or surpassed it in sheer computing power.

At SC13, a conference in Denver where Tsubame KFC was named the greenest supercomputer, a Chinese supercomputer topped a separate list that measured only raw computing power. But that computer uses more than twice as much energy as Tsubame KFC to do the same number of calculations — 4.5 gigaflops per watt for the Tsubame KFC, compared with 1.9 gigaflops per watt. (A gigaflop is one billion calculations per second.)

“Twenty or 30 years ago, when I started in this business, it was a race between muscle cars,” said Steve Conway, an analyst at IDC, a research firm. “Everyone wanted to build the biggest, fastest supercomputer. Now, it’s not muscle cars anymore. It’s a family sedan.”

So far, submersive cooling has been used more widely for supercomputers than for data centers. Mr. Best said that so far, academic institutions that operate supercomputers had been more willing to experiment with the technology than the corporations that run data centers.

“You can imagine, if we walk in and say, ‘Why don’t you take your data center and put it in oil,’ you have to have something pretty solid to point to,” Mr. Best said.