Peregrine System

Peregrine is NREL's flagship high-performance computing (HPC) system. Learn whether
it has the capabilities to meet your research modeling, simulation, and visualization
needs.

System Architecture

Peregrine is a high-performance computing (HPC) system which is configured to run
compute-intensive and parallel computing jobs. It is a cluster comprised of over 2500
nodes (servers) that run the Linux operating system (Red Had Linux or the derivative
CentOS distribution), with a peak performance of 2.26 PetaFLOPS.

The nodes are connected to each other and to storage by a high-speed FDR InfiniBand
network.

The system has NFS file systems for home directories (/home) and application software
(/nopt) as well as high-speed parallel Lustre file systems for both temporary (/scratch)
and longer-term (/projects) storage. These file systems are mounted on all nodes.

Peregrine has four login nodes, each of which has Intel E5-2670 Xeon processors and
64 GB of memory. In addition to mounting the /home, /nopt, /projects and /scratch
file systems, the /mss file system is mounted on all login nodes.

Peregrine has 2592 dual-socket compute nodes, each with 16 or 24 processors (cores).
Together, these nodes have 6,912 Intel Xeon E5-2670 "SandyBridge" processor cores,
24,192 Intel Xeon E5-2695v2 "IvyBridge" processor cores, and 27,648 Intel Xeon E5-2670v3
"Haswell" cores for a total of 58,752 Intel Xeon processor cores. These processors
provide a total of about 1.68 PetaFLOPS. Peregrine also has 576 Intel Phi many-core
co-processors with an aggregate performance of about 582 TeraFLOPS. In total Peregrine
is capable of 2.26 PetaFLOPS.

Number of Nodes in Peregrine

Processors

# cores/chip

# cores/node

Memory/node

Peak (DP) performance per node

88

Intel Xeon E5-2670 "Sandy Bridge"

8

16

32 GB

332.8 GF

56

Intel Xeon E5-2670 "Sandy Bridge"

8

16

256 GB

332.8 GF

288

Sandy Bridge + 2 Xeon Phi co-processors

8

16

32 GB

2.3 TF

288

Intel Xeon E5-2695v2 (Ivy Bridge)

12

24

64 GB

260.8 GF

720

Intel Xeon-2695v2 "Ivy Bridge:

12

24

32 GB

460.8 GF

1152

Intel Xeon E5-2670v3 "Haswell"

12

24

64 GB

883.2 GF

Access to compute nodes is managed by a job scheduler, which makes nodes available
either for either "batch" or interactive use.

Peregrine has two nodes that each have an NVIDIA Quadro 6000 GPU card and 384 GB of
memory. These nodes support the OpenCL and CUDA programming models and support hardware-accelerated
remote visualization of data that resides on the parallel file systems (/scratch and
/projects) using VirtualGL/Turbo VNC.