Simon Fraser University (SFU), in partnership with Compute Canada and regional partner WestGrid, installed a new, advanced research computing (ARC) system, Cedar, at the SFU Data Centre at SFU’s Burnaby Mountain campus. Named after the Western Red Cedar, British Columbia’s official tree, Cedar is one of four national new supercomputers in Canada that will provide over 11,000 Canadian researchers with access to the latest technology in advanced research computing and expertise. The full investment across Canada is valued at $75-million in funding from the Canada Foundation for Innovation (CFI), provincial and industry partners. These investments are addressing urgent and pressing needs and replacing aging high-performance computing systems across Canada.

SFU is a distinct leader in advanced research computing, and the development of the SFU Data Centre directly connects with the University’s vision of being Canada’s engaged university defined by it dynamic integration of innovative education, cutting-edge research, and far-reaching community engagement. Originally built in 1969 to post-disaster specifications, the SFU Data Centre building reflects the unique approach to architecture at the time. It was designed to match Arthur Erickson’s vision for SFU’s Burnaby campus, and the University took over the building from the original occupants, BC Hydro, in 2008. One of the unique benefits of locating the SFU Data Centre on Burnaby Mountain is that it sits on top of an outcrop of sandstone bedrock, which makes it one of the best seismic locations in the lower mainland.

From day one, staff from the Vice-President, Research’s Office, IT Services, Finance, and Facilities Services departments dove in head first to realize the dream of providing a more effective and efficient advanced research computing infrastructure for the country. They collaborated with staff from Compute Canada, located from coast to coast to work through the intricacies of a complex RFP process, and overcame delays in construction and delivery issues in order to build the SFU Data Centre and install Cedar on time and on budget. Since the launch of Cedar in April 2017, the installation was ranked in the top 100 supercomputers in the world for both power and energy efficiency (www.top500.org).

Critères

Please submit one paragraph describing how the proposal fulfills each of the evaluation criteria.

Transférabilité

The collaborative nature of providing a national, strategic research computing service reflects its transferability. The success of this project was fully and completely dependent on a small group of brilliant and dedicated people, the true heroes who transcend the technology, that would happily share the blueprint they developed with other institutions. Part of the blueprint included repurposing an existing building rather than commissioning a completely new building and working with the local utility company to find ways to be more energy efficient. SFU worked closely with BC Hydro on the redevelopment of the Water Tower Building which resulted in a $500,000 incentive which is the largest incentive that has been awarded in their Power Smart Program. Also, the rear-door heat exchange Cedar utilizes is now being used by Graham—Compute Canada's system at the University of Waterloo.

Impact sur la qualité

The two key desired outcomes for this project were:

Outcome: Install an advanced research computing system (ARC) as part of Compute Canada’s technology refresh program that will provide over 11,000 Canadian researchers with access to the latest technology in advanced research computing and expertise.

Result: The installation of Cedar was completed on time and on budget. Cedar's HPC (high performance computing) environment, its massive storage system and its OpenStack partition integrate seamlessly with Arbutus—Compute Canada's cloud system at the University of Victoria—and other systems in the national cloud federation. The installation also benefits the next generation of researchers—students, post-doctoral fellows, young faculty members—by providing them with access to leading edge technology such as Intel's OmniPath interconnect system and Nvidia’s P100 graphics processor.

Outcome: Repurpose existing physical infrastructure to develop a data centre that connects with the University’s vision of being Canada’s engaged university defined by it dynamic integration of innovative education, cutting-edge research, and far-reaching community engagement.

Result: The SFU Data Centre building was originally built in 1969 to post-disaster specification, and SFU took over the building from BC Hydro in 2008. The 22,000 sq ft renovation was designed with efficiency and sustainability in mind and included installing 510 tons of mechanical cooling via modular chillers and a free cooling operation via three 120-ton fluid coolers. The Cedar installation was ranked one of the top 100 most powerful supercomputers in the world by Top500 (86th in June 2017 and 94th in November 2017) and top 100 energy-efficient supercomputers by Green500 (12th in June 2017 and 17th in November 2017). The mechanical portion of the project also received a Silver Award for Construction Excellence in 2017 from the Vancouver Regional Construction Association (VRCA).

Impact sur la productivité

Initiated in 2015 and completed in April 2017, the development of the SFU Data Centre and installation of Cedar is part of an initiative led by Compute Canada to address the urgent and pressing need of replacing aging high-performance computing systems across Canada which comprised of the following:

27 data centres

50 systems

200,000 cores

2 petaFLOPS of computing power

20 petabytes of parallel storage

The new national system, which Cedar is an integral part of, will be comprised of the following:

5-10 data centres

300,000 cores

12 petaFLOPs of computing power

50+ petabytes of parallel storage

Cedar provides 11,000 Canadian researchers, including more than 3,000 faculty members across all academic disciplines, with the platform they need to pursue discovery and innovations in deep learning, artificial intelligence, image processing, and simulations ranging from cosmology to population health to molecular dynamics. Success include:

A data centre capable of housing 10 Megawatts of power. For comparison, SFU’s Burnaby Mountain campus consumes 9 Megawatts as a whole. Each of Cedar’s 17 high density racks is capable of generating the same amount of heat as four average household barbeques.

A PUE ratio worth bragging about. For every megawatt used for computing, the SFU Data Centre only uses 0.07 megawatts to cool and support the computing. To put this into perspective, conventional data centres have a PUE ratio of 2, so for every megawatt of processing power an additional megawatt is required for the cooling and support.

The 8,000 sq ft of uninterrupted data centre floor space. Cedar currently occupies 25% of this leaving room for future opportunities which included moving SFU’s administrative systems to the SFU Data Centre in November 2017. The legacy administrative data centre was housed in a 3000 sq ft facility with approximately 40 racks and reduced by 62.5% to 15 racks in a virtualized environment.

Cedar’s large memory nodes. It has up to 3 terabytes, on-node storage, and GPU nodes for powerful computing. It’s designed to run multiple simultaneous jobs of up to 1280 CPU cores each.

Cedar provides special node types to serve as wide a spectrum of computational needs as possible. E.g., it has large memory nodes with 3 TB of memory and GPU nodes with four graphical processing units for accelerated computing. Cedar is designed to run multiple simultaneous jobs of 1024 processes and more.

Support of SFU’s big data initiative. Cedar is built for big data with 10 petabyte parallel storage system and extraordinary computing power and can support researchers who are collecting, analyzing, or sharing immense volumes of data.

A leading researcher at SFU, Fiona Brinkman, Professor, Molecular Biology and Biochemistry, explained how Cedar would impact her work in an interview in April 2017. “Stable access to a secure, powerful computing infrastructure is critical in today’s world. It is even more critical if we are to effectively use microbial DNA as a key tool to better track and control infectious diseases. Thanks to international travel we are increasingly connected globally—and so are disease-causing microbes. Computing resources like this will be increasingly important to more quickly analyze the growing amount of data necessary to track and control disease.”

Innovation

The innovative aspect of this project includes the following:

Collaboration with other institutions and suppliers across Canada;

Repurposing existing physical infrastructure resulting in a 22,000 sq ft renovation that included converting office space into a mechanical room and installing an evaporative cooling system with over 3,000 feet of chilled water distribution piping;

Use of a 240V/415V electrical system which is more efficient than the 120V/208V system; and

Providing Canadian researchers, regardless of their geographic location, access to the latest ARC resources and expertise as part of a national platform. Cedar will serve many of Canada’s world-class researchers working in diverse fields, including genomics, advanced manufacturing, green technology and personalized medicine.