I'm running the latest LAMP package with PHP-APC. The server is located behind a protected proxy.
When I run netstat -anp | grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n I get 18099 which is normal for the nature of the site. They all are legitimate connections hitting the server from 1 IP.

I have apache.conf prefork_module tuned and have changed it many times with no effect of solving the hanging.

I have watched all the proccess to see if any peak, none do. The disk IO at peak is 67% idle. I have plenty of available ram and it never swaps. The server itself runs great then it just acts like it wants to take a 5 second break and when it returns it can handle what it has missed during the 5 seconds just fine.

There is nothing in the apache error.log that would suggest anything.

I feel something is getting limited and I cannot pin point what it is. I kinda want to say its something with the amount of connections I have like its hitting some sort of connection limit. I was wondering if you guys had any ideas what it could be or something I can run to see.

Update

The server reponds fine when visiting it directly via IP not through the domain. So when I test at the same time when it hangs for the 5 seconds the domain doesn't respond but the IP will respond fine so it makes me think it's the server OS doing it

I did manage to see [apache2] <defunct> just one or two in there not sure if that is enough to cause the hang but it only shows up when it does hand and apache doesn't give me anything about it in the error log.

Update 1/20/2013

My datacenter is going to build a new server and have me switch over.
I have two other servers identical on the same rack same setup same ammount of traffic and this is the only one giving me a issue.

I believe I ran it correct. dpaste.com/884153 it says SMART overall-health self-assessment test result: PASSED but when you scroll down it says some errors not sure what that is.
–
Abby EJan 19 '13 at 22:10

I had an issue just like this about 5 years ago. The entire machine would just lag incredibly hard for 5 seconds (and you could do nothing at that time). It turned out that something in the motherboard had gone bad, and the warranty was still active, so I didn't look into it any further. Anyway, you might want to check there, if the drive isn't the problem.

I have contacted my hosting company and see what they have to say.
–
Abby EJan 19 '13 at 23:12

The server reponds fine when visiting it directly via IP not through the domain. So when I test at the same time when it hangs for the 5 seconds the domain doesn't respond by the server will respond through the IP so it makes me think it's the server OS doing it.
–
Abby EJan 19 '13 at 23:59