More specifically, Open MPI still uses a TCP layer for its run-time setup/teardown. That TCP can be regular ethernet, IPoIB, or any other emulation layer. Ralph's referring to the oob_tcp_if_in|exclude flags allows you to specify using your IPoIB devices, pure ethernet devices, etc.