I ran a test of tcp using NetPIPE and got throughput of 850 Mb/s at
message sizes of 128 Kb. The latency was 50 us. At message sizes above
1000 Kb, the throughput oscillated wildly between 850 Mb/s and values as
low as 200 Mb/s. This test was done with no other network traffic. I
then ran four tests simultaneously between different pairs of compute
nodes and saw a drastic decrease in performance. The highest stable
(non-oscillating) throughput was about 500 Mb/s at a message size of 16
Kb. The throughput then oscillated wildly, with the maximum value
climbing to 850 Mb/s at a message size greater than 128 Kb and dropping
to values as low as 100 Mb/s. The code I am using (VASP) has 100 to
1000 double complex (16 byte) arrays containing 100,000 to 1,000,000
elements each. Typically, the arrays are distributed among the nodes.
The most communication intensive part involves executing an MPI_alltoall
to redistribute the arrays so that node i contains the ith block of each
array. The default message size is 1000 elements (128 Kb), so according
to the NetPIPE test, I should be getting very good throughput when there
is no other network traffic. I will run a NetPIPE test with openmpi and
mpich2 now and post the results. So, does anyone know what causes the
wild oscillations in the throughput at larger message sizes and higher
network traffic? Thanks!