I will build computing cluster (about 64 nodes) and I'm looking for some informations. I want to connect them with Infiniband and install Linux. I tried to google it a little bit, but I couldn't find any informations how to set up a simple working configuration.

I want to use it to access NAS and for interprocess communication (MPI, etc.)

Is it easy to set up Infiniband on Linux? How to do it? Or maybe there is better solution?

5 Answers
5

The Infiniband itself is not a big deal to set up. You have to load the driver for your device and if you don't want to use anything else then IP the ipoib kernel module.

1 IP OVER INFINIBAND
2
3 The ib_ipoib driver is an implementation of the IP over InfiniBand
4 protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib
5 working group. It is a "native" implementation in the sense of
6 setting the interface type to ARPHRD_INFINIBAND and the hardware
7 address length to 20 (earlier proprietary implementations
8 masqueraded to the kernel as ethernet interfaces).

After this you are able to use your device as a normal ethernet device.

Easy? cheap? not really, but definitely possible. Since you're not going for the budget solution (that's six figures for the metal only), you'd probably want to use support from a paid, enterprise, OS.

NAS and IPC have different performance characteristics. I would seriously consider FC for NAS, or perhaps FCoE or even iSCSI if you have gigE. Bonding is good if you have sufficient ports available, but do your calculations first, some multi-port NICs can't sustain the aggregate transfer rates required. Personally, I would use separate switches and networks for FCoE (layer 1) and TCP/IP (layer 3) traffic.
–
CyberFonicMay 29 '09 at 23:52