MTU Issues

Jumbo Ethernet frames can increase performance by a factor of 2-4 on 10G paths on older hardware. Even on modern hardware, we see 9.9Gbps vs 5Gbps for UDP, and 9.9Gbps vs 9.3Gbps for TCP. For 40G paths, the performance gains are even greater. Single stream tests of TCP were observed to be 25Gbps vs. 10Gbps. UDP testing showed an improvement of 15Gbps vs. 8Gbps.

Use of Jumbo frames has two advantages: One is that for a given data throughput, the packet rate is less for jumbo frames vs. standard frames, requiring less CPU for packet processing. The other advantage is that recovery in data throughput after a loss event is proportional to the maximum segment size - so with jumbo frames you get a ~6x faster recovery rate from loss events. For more information on the benefits of Jumbo Frames, see: http://staff.psc.edu/mathis/MTU/

ping can be used to verify the MTU size. For example, on Linux you can do:

Note thaat there are some downsides to jumbo frames as well. All hosts in a single broadcast domain must to be configured with the same MTU, and this can be difficult and error-prone. Ethernet has no way of detecting an MTU mismatch - this is a layer 3 function that requires ICMP signaling in order to work correctly. (Unfortunately some sites block ICMP, which breaks path MTU discovery. If tracepath fails, that is likely what is happening).

Therefore a good approach is often to create a new jumbo frame enabled subnet for your high-speed data transfer hosts.