On May 26, 2005, a new supercomputer, a pioneering giant of its time, was unveiled at BNL at a dedication ceremony attended by physicists from around the world. That supercomputer was called QCDOC, for quantum chromodynamics (QCD) on a chip, capable of handling the complex calculations of QCD, the theory that describes the nature and interactions of the basic building blocks of the universe. Now, after a career of state-of-the-art physics calculations, QCDOC has been retired — and will soon be replaced by a new “next generation” machine.

Great Start

QCDOC was a “special purpose” machine dedicated to fundamental physics. Among its prime users were the scientists of the RIKEN BNL Research Center (RBRC) at BNL, for their studies of the physics taking place at Brookhaven’s Relativistic Heavy Ion Collider (RHIC) and other forefront physics. Funded by RIKEN, the Institute of Physical & Chemical Research in Japan, with infrastructure support from DOE’s Office of Science, QCDOC was built by BNL, Columbia University, IBM, the RBRC, and the University of Edinburgh.

Most specifically, QCDOC was designed to use lattice gauge theory — a method of approximating space-time in a lattice and then computing the dynamics of quarks and gluons, the fundamental constituents of matter, by their theoretical positions on the lattice. One of the founders of this approach is BNL’s Michael Creutz.

The $5-million QCDOC, which took three years to design and build, had 10 teraflops (Tflops) of peak computing power and could perform 10 trillion arithmetic calculations per second. When installed, it took pride of place over BNL’s 1998 supercomputer called QCDSP — for quantum chromodynamics on digital signal processors — which had 0.6 Tflops at peak speed. Together, these two supercomputers provided comparable or greater computing resources for lattice QCD than were available anywhere else in the world.

Surpassing Expectations

“We expected great things from QCDOC, and it surpassed all expectations,” said Nicholas Samios, RBRC director. “It has been the workhorse of three groups at BNL — the High Energy Physics Theory Group, the Lattice Gauge Theory Group, and the RBRC Computing Group that together have established BNL as a world leader in the QCD field. With QCDOC installed, we produced a series of ‘firsts’ in calculations that shed light on much of the breakthrough physics at RHIC as it explores the fundamental properties of matter and conditions of the early universe.”

Taku Izubuchi, Physics Department, a leader of the RBRC Computing Group using QCDOC, explained that until 2008 the entire lattice physics program of RBRC was carried out on the QCDOC and QCDSP supercomputers.

“Through the capabilities of these machines, we developed successful techniques to study kaon physics, nucleon physics, hadron electromagnetic effects, and other important problems in low energy QCD,” he said. “Scientists from Japan, the UK, and many U.S. institutions — we have all benefited from the unique combination of QCDOC and QCDSP.

“We also benefited greatly from the excellence of the Brookhaven support staff from the machine’s installation through technical upkeep, as well as in collaborations with the Lab’s Computational Science Center and New York Center for Computational Sciences,” Izubuchi added.

Among the many highlights of the QCDOC computations were the first lattice QCD computation with chiral quarks (quarks treated with a proper separation of mirror-image spin states, important for the light quarks in nature) and the first and only lattice QCD computation including both dynamical chiral quark effects and electromagnetism. Another was the first and only direct computation for a long-standing problem in certain decays of kaon particles that were key to understanding important aspects of the Standard Model of particle physics. For more, see Physical Review D78: 034503 (2009), arXiv:1106.2714.

Building on Past Success: Blue Gene, QCDCQ

The success of these supercomputers at Brookhaven preceded the much larger commercial machines that exploited the same technology but had greater economy of scale. One example is the IBM Blue Gene/P established in 2008 at Argonne National Laboratory. The U.S. QCD group, of which RBRC is a part, used nearly one-quarter of this huge supercomputer for its work.

Now, a new project is under way: RBRC, Columbia University, and BNL, together with the University of Edinburgh, a collaborator in the United Kingdom KQCD group, are developing an even more powerful supercomputer, QCDCQ (QCD with Chiral Quarks). This machine will have 75 to 150 Tflops of sustained speed — key to RBRC’s continuing international leadership in lattice QCD. The researchers will also work with colleagues at the T.J. Watson Laboratory of IBM — the same colleagues who transformed the technology of QCDSP and QCDOC into IBM’s Blue Gene product. So QCDCQ will benefit from advantages of IBM’s next-generation supercomputer, which will be available in prototype form at RBRC and Edinburgh between one and two years before larger installation of commercial versions are available.

When installed, these QCDCQ machines will provide the RBRC and the larger RBRC/UKQCD collaboration with spectacular research opportunities for understanding more of what RHIC shows us about the early universe and exploring new physics at present not accessible to lattice QCD, Izubuchi said.

And Beyond

“There’s another great value of QCDCQ,” said Izubuchi. “Just as QCDOC provided the basis of innovations for QCDCQ, the advances we’ll make using QCDCQ will inspire the next generation of supercomputers even further into the future.”

Brookhaven National Laboratory

Brookhaven National Laboratory is a multipurpose research
institution funded by the U.S. Department of Energy. Located on
Long Island, NY, Brookhaven operates large-scale facilities for
studies in physics, chemistry, biology, medicine, applied
science, and advanced technology. The Laboratory's almost 3,000
scientists, engineers, and support staff are joined each year by
more than 5,000 visiting researchers from around the world.

One of ten national laboratories overseen and primarily funded by the Office of Science of the
U.S. Department of Energy (DOE), Brookhaven National Laboratory conducts research in the physical,
biomedical, and environmental sciences, as well as in energy technologies and national security.
Brookhaven Lab also builds and operates major scientific facilities available to university, industry
and government researchers. Brookhaven is operated and managed for DOE's Office of Science by Brookhaven
Science Associates, a limited-liability company founded by the Research Foundation for the State
University of New York on behalf of Stony Brook University, the largest academic user of Laboratory
facilities, and Battelle, a nonprofit applied science and technology organization.