Time_wait

Would an excess amount of these cause a server load issue? Past few days ive been having some issues with load and I was thinking it was a network issue. Scanning over the netstat Ive blocked some IPs via a hardware firewall (fortigate 300a) and load came down but new IPs have returned. Ive turned off keepalive in httpd.conf but now the TIME_WAIT has increased.

Kinda goes back and forth, with KeepAlive on that number is around 40-70

IPs are random, never the same so i dunno if im having some network problems or not kinda stumped. Just know the load is running a little rich for a robust system, procs are low, memory and swap is low, everything else looks alright. When load goes up a lot of httpd processes pop up but goes away just as quick as it showed up doesnt last long few seconds maybe but quick enough to raise the load to 12 and drops back down to 2, when they do thats all I see in netstat is a bunch of TIME_WAIT