You are here

Reducing the load on a small VPS by 80% in 5 minutes

A few days ago I noticed the average load on a newly setup VPS was too high relative to its workload. The server was noticeably sluggish. Load was high and a page the wiki it was running would occasionally take 10 seconds to load.

I didn't have too much time to kill but I decided I would at least try picking off the lowest hanging fruit before upgrading to a beefier, more expensive plan which would cost a few hundred dollars extra a year.

When I noticed swap usage was relatively high, I suspected the server was constantly moving memory in and out of swap.

One of the first things you should try when you're tuning the performance of a server is to reduce memory usage so that it doesn't use as much swap and can use more of your memory for disk buffering. Accessing buffered blocks from memory is a gazillion times faster than accessing them from disk.

So how do you reduce memory usage without reducing functionality? By tuning the trade off between memory and CPU time.

One example of this trade off in practice is pre-forking, which is a common technique used by many systems that can improve user-perceived performance. The principle is that by starting the process in advance of when it's needed then that bit of overhead is not experienced by the user who would otherwise need to wait for the process to start up before it can serve him.

The catch is that keeping spare processes around requires memory.

When you have a limited amount of real memory, then any performance gains from pre-forking can quickly evaporate due to reduced IO performance.

On this server, we didn't really have that much activity for things like mail or web, but many subsystems were configured (by default) to pre-fork at least 5 worker processes in advance.

This is a reasonable configuration for a dedicated server with lots of memory and hundreds of users, but not ideal for that cheap VPS you just setup for your workgroup.

I reduced memory consumption by over 200MB by simply decreasing the number of preforked worker processes for apache (20-25MBs per process), spamassassin (30MBs per process), and saslauthd. Average load dropped immediately by about 80%.

Total time spent picking this low hanging fruit: 5 minutes.

Comments

I substantially reduced the memory usage on my own VPS a few weeks back by moving from the Apache Prefork mpm to the Worker mpm, and by disabling FastCGI so I didn't have idle php processes sat around using up memory.

If I recall that's the main reason prefork is so widely used. Worker is known to be more memory efficient but not everything is thread-safe. In particular I remember this created a problem with PHP, so you had to run it outside of Apache in its own little pre-forking application server (e.g., with FastCGI).

Personally I prefer lighttpd over Apache. There's a bit of a learning curve involved but it's not only super-fast and efficient, it also has a very powerful configuration system which is in my opinion superior to Apache. These days the only reason I would use Apache is backwards compatibility on a server where memory resources are not an issue.

I've also played with nginx, and it's nice too. Slightly faster than lighttpd in many cases. But on the other hand, lighttpd is far more flexible and configurable.

We had similar issues with porting rewrite rules to lighttpd from Apache and found the solution in lighttpd's mod_magnet, which allows you to script these things in Lua. Sure there's a learning curve involved but you can pretty much get anything done. I like solving these sorts of challenges so for me it was fun.

Also, lighttpd rewrite rules are getting much more powerful so in the future I bet you'll only need dive into Lua for the really complicated stuff.

Due to copy-on-write and the way virtual memory works, that does come as a surprise to me. Pre-forking shouldn't significantly increase the consumption of RAM and thrash your server that much, at least not for as little as 5 idle-just-forked processes.

I'd say it was good that it seems to have fixed your problem, but I don't believe pre-forking to be the real issue here.

So with copy-on-write semantics pre-forking shouldn't necessarily be an issue but I think that depends a lot on where you fork and what the child process does afterwards. Also, I'm guessing with dynamic languages like Perl in which spamassassin is implemented the performance hit from running multiple instances of the Perl runtime could be significantly greater.

I used your tip about tuning the trade off between memory and CPU time and it helped speed up our server. Pre-forking wored wonders. We have lots of workers that all run automotive crm software and we all noticed the difference after I altered the server.

Liked your article dude. I tried the trade off technique using prefork, reduced Apache conf to bare minimum and removed all unwanted modules but did not get the expected results out of it. Had to move to worker model. Although both prefork and worker caused the load average to shoot up to 20, the worker model was able to withstand 250 concurrent users against 15 concurrent users from prefork.

It would be extremely helpful if you can share tips on how to reduce the load average which is caused by Apache coz at this stage I would like to stick with Apache instead of switching over to Lighttpd or nginx although both are better than Apache any time.

I'm trying to use amazon ec2 for my MT4 usage. I'm a rookie in this kind of thing. I've seen your suggestion to maximise the performance so would like to seek your help to set something to improve the load.

I don't understand all these things you mentioned. Would you please email me the procedures to set it? Thanks.

you wrote:

For anyone wondering where to look to tweak Apache Prefork conditions, you'll find the settings in /etc/apache2/apache2.conf as follows:

But IMO unless you have really low traffic A micro server just isn't going to cut it! In my experience if you want half decent reliability then a Small is the starting point. Once you start starving that of resources then it might be time to consider tuning... My 2c anyway...

As others have said elsewhere, perhaps it might be time to try one of the other lighter weight servers such as Nginx or LigHTTPd.

when 150 visitors online on my site then server load going up to 2.5 it is increased my vps limits and when 200 visitors online the server load at 1.6, does the server depand upon the visitors or any other activities of website

Every few days my VPS server with my host provider runs out of memory and I cannot access anything. I have to try to reboot it from my end. If this does not work I have to contact the host team, and then sometimes trying to access the server causes a Firewall to Block me out from my machine. Can I ask a couple of Questions regarding VPS Servers

My VPS Server along with CPanel Licences costs me about €36 per month.

I have 5 websites and 5 Cpanels set up for those sites.

My VPS server has 1GB of memory I am told. But every 3-4 days I have to reboot it to keep it visible.

Would you have any suggestions why this happens. My hosting company keep saying I need to buy more memory, if that is the case everytime I ad a new account to my hosting plan I will need to buy new memory and if you are hosting many websites then your bill to a hosting company will be astronomical?

All Help Greatly Appreciated.

This is the largest website I have on the server. http://www.kitchens4u.ie - OH and BTW this will not be accessible until Monday 14th October as they have suspended me over the weekend.

Thanks for the post. My FreeBSD VPS (512MB RAM, dual-core) was swapping like crazy (almost all of the 512M on 1st virtual HD and some of the 2G swap on 2nd virtual HD).

Mysqld was running in its default state (w/o a config file), probably due to a recent fiasco with ports after updating from 8.2 > 9.2. Copied /usr/local/share/mysql/my-small.cnf to /usr/local/etc/my.cnf. Copied the "mpm_prefork_module" section of /usr/local/etc/apache22/extra/httpd-mpm.conf into the httpd.conf file. Set "$max_servers = 2;" in /usr/local/etc/amavisd.conf and rebooted.

*NO* swapping!

Now, don't criticize ... sloppy management, yes. But I just keep this box in memory of my former business ;-)

Hey, just grabbed a VPS from a decent provider.
I left my VPS running for a couple days just to configure it and what not(not restarting). I found that when I logged in to check my RAM usage, it was at 48%. I'm running the Turnkey Wordpress appliance. Ideally after getting WordPress configured, I also want to run a Supybot with minimal usage, around 20 users.

The VPS stats are as follows:
2 x Intel E5 CPUs 2.3GHz
1024 mb RAM

After getting the VPS configured, would running a Supybot(for IRC) be obtainable?
Any suggestions about WordPress would be appreciated.
Thanks.

As per this article (and many of the comments) you can try tuning Apache to trade off CPU vs RAM. In the TurnKey appliances Apache is using it's (very general) default Debian config. I'm no expert but from what I hear there are lots of things that you can do to tune it to get maximum performance.

Another option is to use an alternative webserver such as LigHTTPd or Nginx. They are both significantly lower resource usage but much less user friendly (IMO) to configure. TurnKey has both a LigHTTPd or Nginx webstack appliances (with PHP & MySQL) but you'll need to install WordPress yourself.