The buffer sizes are already "tuned". Btw, what is optimal for the send values
("btl_tcp_min_send_size" and "btl_tcp_max_send_size")? A high value (just few segmentations on MPI
level) or a low value? We changed it both ways, but it had nearly no effects.

Jon Mason wrote:

> That script should optimally tune your NIC. If you are still not
> satisfied with the performance, Chelsio should have people available to
> help. Since the TOE module is not opensource, there is not much anyone
> else can do. You can try tweaking any module parms that are exposed.
> Checkout the modinfo output for that module.

The NIC is well tuned, as the results on TCP-level show. There seems to be a bottleneck in Open MPI.

Jon Mason wrote:

> You can also try the new iWARP support in OMPI 1.3. The perf for that
> should be much better.

Yes i will try it, but i can't offer a unstable version of Open MPI in a productive system. So still
it's not released official the users have to work with 1.2.6

Steve Wise wrote:

> So OMPI experts, what is the overhead you see on other TCP links for
> OMPI BW tests vs native sockets TCP BW tests?