Brian,
There are usually three or four categories of storage:
1) /home - small, just enough to keep source files and compile code
2) /scratch/local - distributed disks within a cluster for local
writing (think Gaussian)
3) /scratch/global - a high-performance (and higher cost) parallel
file system accessible by all nodes
4) /archive - a very large pool of spinning disks which receives data
from /scratch/global when a run (or set of consecutive runs) is
"complete." The idea is to clear off the expensive parallel system for
other run-time use, but that you still want to hold the data for some
future need.
I would keep your /home and /scratch/global separate.
The /scratch/global solution you pick will very much depend on how you
want it connected to your clusters. By definition (of your cluster
suite) you cannot have a system that relies on IB as not all of your
systems have IB. This leaves GbE as the only global means of
connection. If at all possible, I would dedicate a GbE interface on
all nodes who access /scratch/global.
--
Brian D. Ropers-Huilman, Director
Systems Administration and Technical Operations <bropers at msi.umn.edu>
Supercomputing Institute for Digital Simulation and Advanced Computation
599 Walter Library +1 612-626-5948 (V)
117 Pleasant Street S.E. +1 612-624-8861 (F)
University of Minnesota Twin Cities Campus
Minneapolis, MN 55455-0255 http://www.msi.umn.edu/