I am trying to improve performance of my servers while under load. I am on a 1Gbit line and don't have an issue sending or receiving at capacity while idle.

These servers fetch, process, and then return data to users. rx:tx is almost 1:1. I noticed issues with my downstream while serving traffic to users and have been able to reproduce the issue with iperf3/wget. I've seen the issues on the web servers, and also load balancers (haproxy) sitting in front of them.

If I run iperf3 -P 4 -R (4 parallel streams receiving data) or wget alone, it goes at 1Gbit. If I run them both at the same time total server usage drops under 500Mbit and bounces all over the place. Stopping one causes the other to go back up to 1Gbit.

Transmitting works just fine @ 1Gbit while under load, as well as downloading within the datacenter.

The machines are running dual Xeon L5630 with a mix of Broadcom and Intel NICs. I've tried a variety of tweaks to improve performance nothing seems to help.

I've been told that the issue is with processing interrupts quick enough. And in the process of researching this I have determined there's no one size fits all solution for performance optimization. I would love if I could figure this out myself, but would be happy to hire someone as well.

Does anyone have recommendations as to where to start?

Is it most likely interrupts? I loaded up irqbalance, then turned it off and assigned nic queues to different cores by hand.

Tried tweaking ring buffers and sysctl parameters and countless things. Not sure where to go from here.