[solve] cpu usage high

i'm hosting a site that gets 1M+ visits a day in litespeed right now.
the site is a combination of php and cgi and i noticed that my cpu's hover around 20-30 percent w/ load avg around 8-15. server has 32 cpu cores of intel romley btw. litespeed is configured with lsphp + suexec.

the server has a ton of ram (64gb), but it is only utilizing 2-3 percent of it and i was wondering if i haven't configured litespeed correctly for this. disk is SSD w/ disk i/o less than 2 percent;. mysql is hosted on separate server.

yes but when load avg starts to reach 15+ (indicating more traffic), the site starts to slow down and response time of site starts to become greater than 4+ secs, when normally it's 1 second or so.
could i be hitting some max connections settings (either litespeed and/or php, kernel, etc) and it is placed in queue?

it seems your cgi scripts take most cpu cycles. so your application is more of cpu bound.

take a look at real-time stats in LiteSpeed Admin Console (Actions=> Real-Time Stats) and paste a screen shot here (as NiteWave requested). It should tell us whether your application processes are piling up and where bottleneck if any, etc.

i don't think we really have a busy time, traffic is constant throughout as we also get lot of international traffic.
uplink is 100mb. do you think we are hitting a bottleneck in network?
i see we use 30mb down and 30-50mb up.