The astro_long|short|devel nodes are based on a Dell shoe-box design with a dual 10-core ivy-bridge CPU. The plain nodes are Dell C6220II, while the nodes with Xeon-Phi cards are Dell C8220x.

Analysis Hardware

Name

CPUs

RAM

GPUs

Local Scratch Space (if any)

Notes

astro01.hpc.ku.dk

8 cores: 2x Xeon X5550 @ 2.67GHz

72 GB - 9 GB / core

4x Tesla C1060

- Identical to a GPU node.

- L2: 256KB/core, L3: 8MB, newest supported extension: SSE4.2.

astro04

48 cores: 8x Xeon X5680 @ 3.33GHz

247 GB - 5.14 GB / core

/sc1: 66TB, /sc2: 77TB

- Using ScaleMP software across four 2-socket nodes.

- L2: 256KB/core, L3: 12MB, newest supported extension: SSE4.2.

- Not accessible from outside. Only via astro06!

astro06.hpc.ku.dk

32 cores: 4x Xeon E5-4650 @ 2.70GHz

768 GB - 24 GB / core

Quadro K2000

/scratch: 2.7TB RAID0 SSD

- GPU for remote HW rendering through VirtualGL.

astro07.hpc.ku.dk

astro08.hpc.ku.dk

astro09.hpc.ku.dk

32 cores: 4x Xeon E5-4650 @ 2.70GHz

768 GB - 24 GB / core

Quadro K2000

- GPU for remote HW rendering through VirtualGL.

Notes

astro04 is a virtual NUMA (non-uniform memory access) server composed of 4 nodes each having 2x6 cores, 72 GB memory, 24 disk slots, 2 raid controllers, and a dual QDR infiniband connection. The four nodes are uplinked to a dedicated QDR switch and are running the virtual machine software ScaleMP, which effectively present the four nodes as one machine. ScaleMP uses some of the system memory for caching purposes, decreasing the available memory from 288 GB to 247 GB.

The latency between CPU cores on different nodes is very different relative to core on the same node (0-11, 12-23, 24-35, 36-47). It is therefore an advantage to launch programs with, for example, taskset:

% taskset -c 0-11 idl

will launch an IDL session restricted to running on core on the first node.

The main advantage of astro04 is that it has fast, low latency, local storage /sc1 and /sc2, and a 10gbit connection to the storage server.

Storage

The home directory (/users/astro) is a GPFS system. We have a shared 16TB quota.

Archive (/users/astro/archive) is a symbolic link to the two ZFS filesystems exported from astro05 as NFS volumes. Each archive system can be found under /users/astro/archive0 and /users/astro/archive1, and each user will either have a directory under archive0 or archive1, but should always refer to it as /users/astro/archive/<username>.

archive0 has 215TB spread over 90 3TB disks. It uses RAIDZ2 for redundancy in stripes of 16+2 disks.

archive1 has 172TB spread over 37 6TB disks. It uses RAIDZ2 for redundancy in stripes of 16+2 disks with 1 hot spare.

The storage server, astro05, is serving both the two archive filesystems, and a similar sized filesystem for DARK. It has an intel Xeon E5-1650, 6-core CPU, 3.2GHz, 128GB of memory for efficient caching, and a 10 gbit/s uplink to the core switch.

Network

External connection: The local HPC center is a Tier-1 CERN node and has a dual 10 gbit/s connection. In practice we have reach 100 MB/s ot-of-house by tranferring several large files in parallel