SANDIA LAB NEWS

April 9, 2010

New Cielo supercomputer 10 times faster than current NNSA platform

Sandia and Los Alamos National Laboratory (LANL) researchers have jointly awarded a contract to Cray Inc. to build a supercomputer that will have more than 10 times the capability of NNSA’s current platform — the Purple supercomputer at Lawrence Livermore National Laboratory (LLNL).

The machine will support computations at Sandia and LANL, as well as at LLNL

“Cielo will target extremely large problems that require petascale supercomputing,” says Sudip Dosanjh (1420),
codirector of ACES (Alliance for Computing at Extreme Scale), a New-Mexico based partnership between Sandia and LANL. “This is the culmination of a two-year collaborative effort. We look forward to working with Cray to create an order-of-magnitude increase in capability for key NNSA national security applications.”

Because Cielo will be dedicated to running the largest and most demanding workloads involving modeling and simulation, it will support large single jobs capable of utilizing the entire platform.

This increased capability is expected to increase understanding of complex physics and improve confidence in the predictive capability of stockpile stewardship.

Additional capabilities in 2011

Installation is projected for the third quarter of 2010, with additional capability planned for 2011.

Design of the machine was led by Sandia in cooperation with LANL. The two labs will share day-to-day responsibilities for operation of the platform, which will be housed at LANL’s Strategic
Computing Complex facility.

The selection of Cray — the industry partner chosen to build the approximately $54 million machine — was made through a competitive procurement process. The technical evaluation by members of the labs included design, procurement, and deployment.

The ultimate design goal for the machine — part of NNSA’s Advanced Simulation and Computing (ASC) program — “is for Cielo’s increased capability to achieve higher degrees of fidelity in the models and reduce the total time to solution," says Doug Doerfler (1422), Cielo system architect.

ASC’s modeling & simulation applications “perform extremely well on the Cray XT architecture,” he says. “The XT has demonstrated fast execution times and excellent scaling characteristics while also providing a reliable and robust environment for our users.”

Based on next-gen Cray architecture

Cielo will be based on Cray’s next-generation “Baker” architecture with a new high-speed interconnect named “Gemini” that, says Doug, “will provide a transparent transition for our users and give a significant boost in performance.”

Says NNSA Administrator Thomas D’Agostino, “Cielo will be an invaluable addition to our supercomputing program, which enables NNSA to ensure the safety, security, and effectiveness of the nuclear stockpile.”

The future will produce even greater challenges, says Doug, because Cielo — as good as it’s expected to be — may be the last of its line in providing major improvements in computing capabilities without a major investment in new computing codes.

"Supercomputers are at an inflection point due to the development of massively multicore and heterogeneous processor architectures,” Doug says. “This is a huge issue for our algorithm and application teams, and at this point in time it's not clear what the right solution is and how the codes should be written to support these future machines.”

Ground broken on Combustion Research Computation and Visualization building

More than 20 years ago, then-director of the Combustion Research Facility (CRF) Peter Mattern had a vision for a complex that would include both experimental labs and a facility dedicated to predictive modeling and simulation. On March 24, that dream started to take shape as reality as the first mounds of dirt were moved at a groundbreaking ceremony for the Combustion Research Computation and Visualization (CRCV) building.

ARCHITECTURAL RENDERING of the new Combustion Research Computation and Visualization (CRCV) building. (Rendering courtesy of Flad Architects)

“People have had this vision for a while, recognizing the need to develop predictive simulation as a way to move forward because you can’t measure everything,” said CRF director Bob Carling (8300). “Technology and innovation must evolve more rapidly than through experiments alone in order to meet our energy goals. Predictive modeling is a key enabler for the transportation industry, so this new computational facility will be an integral part of our efforts.”

He credited Andy McIlroy (1800), acting director for Materials Science and Engineering, and acting senior manager Dawn Manley (8350) with putting together the vision for the CRCV and seeing the project through to the end.

Cofunded by DOE’s Office of Science and Office of Energy Efficiency and Renewable Energy, designed by Flad Architects, and under construction by BN Builders, the 8,400-square-foot facility will offer offices, visualization equipment, collaboration space, and 2,000 square feet of dedicated space for high-performance computing clusters (see Lab News, Aug. 14, 2009, page 6). Construction should be complete by the end of the year.

First-principle direct numerical simulation

The CRCV facility will support high-fidelity numerical simulations and the building and testing of models that require highly intensive computational capabilities. Simulations will include first-principles direct numerical simulations of turbulent flames, which allow for the examination of fine-scale coupling between turbulence and chemistry interactions. Large eddy simulations will enable modeling of the entire engine intake and combustion chamber, as well as the smaller-scale transport and chemistry issues.

Div. 8000 VP Rick Stulen welcomed guests from DOE, Sandia/New Mexico, and the city of Livermore to the groundbreaking ceremony. Also in attendance were two former CRF directors, Peter Mattern and Bill McLean.

“The Combustion Research Facility represents a phenomenal partnership between very fundamental science and applied science, one that has lasted over 25 years and is unique within the national laboratories,” said Rick. “It is this combination that makes this a very special place.”

He described the predictive modeling and simulation capabilities of the CRF as essential to help meet the challenges laid down by President Obama: an 80 percent reduction in CO2 emissions by 2050 and a 25 percent reduction in the consumption of liquid petroleum. This was a theme that the other speakers carried through their talks.

“For a long time we’ve recognized that the path to success is through truly predictive combustion models,” said Eric Rohlfing, director of chemical sciences, geosciences, and biosciences for DOE’s Office of Basic Energy Science. “As much as it pains me to say this as an experimentalist, we’ll never measure everything. We need to come up with ways to understand the details and complex interplay between turbulence and chemistry in modern engines with new fuels. I’m delighted that we’ve been able to partner with our colleagues in the vehicle technology program to provide the funding for this building.”

A new way of managing construction

CRCV is not just a new facility at the California site, it also represents a new way of managing construction for Sandia. Dan Sanchez, an NNSA/Sandia Site Office (SSO) senior manager, spoke about the new oversight model that gives Sandia more control. He likened the change to a parent giving up car keys to a teenager.

“We are taking a backseat approach while providing guidance and all of the support that Sandia needs to be successful in this particular project,” he said. “This new oversight process is a reflection of our confidence at SSO in Sandia’s abilities to manage a large construction project.”

Fuel cell-powered mobile lighting system deployed at Academy Awards

A team led by Sandia National Laboratories made a cameo appearance at the 82nd Annual Academy Awards ceremony in Los Angeles. The fuel cell- powered mobile lighting system, introduced in the Oct. 23, 2009, issue of Sandia Lab News, shed a
little environmentally friendly light on the festivities.

Sandia, Boeing, the California Department of Transportation (Caltrans), Altergy Systems, Multiquip, Inc., and others developed a novel mobile lighting system as a clean, efficient alternative to traditional technologies powered by diesel fuel generators. These small, portable lighting systems are used primarily by highway construction crews, airport maintenance personnel, and film crews.

During the night of the Academy Awards ceremony, the unit provided lighting at an access point used by media and production personnel to the famed red carpet area, and also provided auxiliary power for a security metal detector. The unit was used in the days leading up to the event for construction of the red carpet that leads into the Kodak Theatre.

Zero-emission electric power source

The new prototype system features a fuel cell running on pure hydrogen, a zero-emission electric power source that is also very quiet. The fuel cell produces electricity for an advanced, power-saving Light Emitting PlasmaTM (LEP) lighting system and additional auxiliary power up to 1.5 kilowatts, which allows some equipment (such as drills, power tools, or security metal detectors) to be powered by the unit at the same time the system is providing illumination. The hydrogen was purchased from Air Products and dispensed from the company’s hydrogen refueling trailer.

By comparison, current mobile lighting uses diesel fuel generators that produce CO2, NOx (nitrogen oxides produced during combustion), and soot (particulate matter), making them less than ideal for the environment. In addition, diesel units are noisy and can create a safety hazard when construction personnel are distracted and cannot hear oncoming traffic.

Sandia project lead Lennie Klebanoff (8367) estimates that the deployment of a fuel cell-based mobile lighting unit could reduce diesel fuel consumption by nearly 900 gallons of diesel fuel annually (per unit), while also eliminating the NOx and soot emissions produced by diesel fuel generators. If the hydrogen used in the lighting system is generated from nonfossil fuel sources, then the replacement of a single mobile unit would reduce CO2 emissions by about nine metric tons per year.

Especially appealing for entertainment industry

Russell Saunders, whose company, Saunders Electric, Inc., has been providing temporary power facilities for the Academy Awards since 1953, says working with the mobile fuel cell lighting unit has been a positive experience due to its ease and flexibility. According to Saunders, the fact that the system meets film production sound levels, maintains zero exhaust emissions, and can be used both on indoor and outdoor film shoots makes it especially appealing for the entertainment industry.

In addition to the fuel cell, another key component of the system is the LEP technology contributed by Luxim, Lumenworks, and Stray-Light Optical Technologies.

Before this technology was introduced, mobile lighting units typically consumed 4.4 kilowatts. The LEP system only consumes about 2.3 kilowatts for the same light output, a reduced power requirement that saves energy and increases the system duration (operational time between refills). Because LEP uses approximately half the energy of standard systems, it further increases the efficiency of the fuel cell-powered system. This makes it a zero-emission electric power source.

The system incorporates two pressurized hydrogen tanks (purchased by Sandia from Structural Composites, Inc.), a trailer with an enclosure that houses the hydrogen tanks and fuel cell (provided by Multiquip), and a 5-kilowatt fuel cell (provided and installed by Altergy Systems). Multiquip and Altergy assembled the overall unit, while Sandia oversaw the design and technical plan.

Attracts interest of San Francisco International Airport

In addition to the film industry, the project has also attracted the interest of San Francisco International Airport (SFO) for airport maintenance work and Caltrans for nighttime highway construction. Unlike the diesel systems that traditionally power mobile lighting units, the fuel cell-powered mobile light can be used indoors. SFO plans to use the technology in its indoor terminal maintenance work.

Boeing provided original seed funding to build the prototype unit. A unit for SFO will incorporate a more sophisticated, technically ambitious hydrogen storage system that uses metal hydride storage tanks designed by Ovonic Hydrogen Systems.

Sandia’s work on a fuel cell-powered mobile lighting system reflects the Labs’ long history of exploring basic science for energy and transportation. From developing renewable means of producing hydrogen, to discovering the science behind hydrogen safety, to creating the building blocks of hydrogen and fuel cell systems, Sandia scientists and engineers are actively working to help hydrogen and fuel cells take their place in a sustainable energy future.

Additional funding sources are being sought so that more fuel cell-powered mobile lighting systems can be refined, built, and deployed, reducing both diesel
consumption and emissions. The ultimate goal is to displace diesel fuel-powered systems altogether. --Mike Janes