On Wed, 2 Jul 2008, Carsten Aulbert wrote:
> OK, we have 1342 nodes which act as servers as well as clients. Every
> node exports a single local directory and all other nodes can mount this.
Have you considered using a parallel file system ?
> What we do now to optimize the available bandwidth and IOs is spread
> millions of files according to a hash algorithm to all nodes (multiple
> copies as well)
There have been many talks of improving performance by paying
attention to the data locality on this very list. Are you not able to
move the code to where the data is or move the data to where the code
is ?
F.e. using a simple TCP connection (nc, rsh, rsync or even http) to
transfer the file to the local disk before using it is probably more
efficient than the way you use NFS is you deal with small files (as
they have to be written to some local storage). The setup and
tear-down costs of the NFS connection (automounter, mount, unmount)
simply doesn't exist in this case; the transfer of data on the wire
happens the same way. Or you could even get around the limitation of
storing it locally by using a ramdisk to temporarily store the files
(if you have the free memory...) - from what I understand they are
read then used immediately and not needed again in a short time frame
so it makes no sense to store them for longer, a perfect application
for a tmpfs.
--
Bogdan Costescu
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de