Kb75

Emulab FAQ: Troubleshooting: Why is the per-packet latency so high (variable) on the pc3000 nodes?

Emulab FAQ: Troubleshooting: Why is the per-packet latency so high (variable) on the pc3000 nodes?

The pc3000 nodes have Intel Pro1000 Gigabit adaptor cards which implement "interrupt throttling" which can add quantity and variability to the latency of packets. Interrupt throttling effectively caps the interrupt rate (either transmit or receive) presented to the CPU. See the README file that comes with the driver (/share/linux/e1000* or /share/freebsd/em*) and the referenced Intel application note http://www.intel.com/design/network/applnots/ap450.htm . Please note that the absolute and packet timer descriptions in the README file are backwards, in particular the descriptions of RxIntDelay and RxAbsIntDelay would lead you to believe that the default values have no effect when in fact they serve to limit receive interrupts to about 8000/sec (RxAbsIntDelay == 128).

To disable interrupt throttling on all interfaces in Linux, put the following as one line in your /etc/modules.conf (/etc/modprobe.conf in Fedora Core images) file:

For FreeBSD, while you can change the per-interface settings with sysctl (hw.em?.*_int_delay), you cannot overcome the global 8000 ints/sec. that is compiled into the kernel. To override that, you will need to modify the driver and build a new kernel. In /usr/src/sys/dev/em/if_em.c look for: