Intel tests oil immersion to cool servers

SAN JOSE, Calif. – Intel Corp. announced positive results from a year-long test of a novel system that cools servers by immersing them in mineral oil. The CarnotJet vat from Green Revolution Cooling (Austin, Texas) delivered a cooling power usage effectiveness (PUE) rating of 1.02-1.03 without impacting performance of the seven dual-socket Xeon servers, Intel said.

The results suggest the oil vats require only two to three percent additional energy than the servers themselves. By contrast, a typical data center runs at a cooling PUE as high as 1.6, meaning the non-IT systems require 60 percent in additional energy beyond the power needed to run the servers.

Some companies such as Facebook have opened new data centers in areas where they can use ambient air to cool their systems, eliminating the costs of air conditioning. But they are not the targets for the oil-immersion systems.

“If you are going to add a significant amount of capacity and have systems with a lot of density and don’t have good air flow in your data center, this is worth looking at,” said Mike Patterson, a senior thermal architect for data centers at Intel who supervised the test.

The Texas Advanced Computing Center is one data center that fits that description. It has been testing the same technology for several years. But oil immersion is not for everyone.

“We probably have ten different advanced cooling technologies we are analyzing…each has its pros and cons, and each market has different priorities,” said Patterson.

Intel dunked seven Xeon servers for a year in a CarnoJet vat from Green Revolution Cooling.

The Intel group is working on a proposal to study immersion cooling with a two-phase fluid that boils off and condenses, transferring heat. “There are a number of other liquid cooling approaches including cold plates for CPUs to plates for full servers, so we are very much in our path-finding phase,” he said.

For its part, Intel has no further plans to use the oil-immersion approach, in part, because it has plenty of space in its data centers where air flow is good. However Patterson’s group is analyzing the trade-offs of building an “oil-optimized platform.” In today’s servers “the heat sinks and fan control built into the platforms are for air cooling,” he said.

Intel conducted the oil-immersion test after overcoming some of its own skepticism.

“We asked a lot of tough questions and they had pretty good answers for all of them so we put a pilot together,” said Patterson. “There’s almost an emotional reaction [against oil immersion] at first, so people say they would never do that, but when they see the potential [energy] savings, they see there’s something there,” he said.Related stories:

thank you @docdivakar! you are very well informed, as always...would you be interested in giving a talk at emerging technologies symposium in Whistler next summer? details at www.cmoset.com, kris.iniewski@gmail.com

@iniewski: we are almost there! There are microchannel coolers in CPU's that use propylene glycol and water but so far no liquid nitrogen... HP Z420 and Z820 Workstations already use liquid cooling.
MP Divakar

@agk: none of the above! FYI, the JEDEC 50.1 procedure to calibrate semi chip packages for thermal resistance does call for immersing in an Oil bath. Obviously the board assemblies are cleaned prior to being mounted on the server chassis, so there shouldn't anything leaching but there could be some outgassing.
So what Intel is doing is nothing out of the ordinary. It is a staright forward process to calculate the required oil volume for a given temperature rise.
MP Divakar

Remotely Operated Vehicles (ROVs), and other similar systems that operate at great depths in the ocean, have been using this technology for decades. In our case, the mineral oil is also used to equalize the pressure inside the electronics housing with the ambient seawater pressure so that very strong and expensive enclosures are not required. The oil lasts forever and is not aggressive, it does not exhibit chemical compatibility problems with most electronic components. As with all technologies, there are nuances to master but nothing exotic. Mineral oil is also environmentally friendly in case of a spill.

I think that's a question of power density. The problem is often a limited power budget for the site. Filling as much of that budget with processing, storage and communications and not cooling is becoming paramount.