May 30, 2008

HOWTO: Using nginx to accelerate Apache on Cpanel server

Nginx - the small, lightning fast and very efficient web server is usually used to serve static content or as a reverse proxy/load balancer for the Apache or other relatively slow backends. So it would be natural to use nginx as a frontend for Cpanel's Apache. It would save a substantial amount of memory and CPU time usually sucked by the numerous Apache children spoonfeeding content to the clients.

I always had this in mind, but until recently had no time to look closely at implementing it. Then I saw a forum post with a sample script for generating the nginx configuration file based on Cpanel account info, and then an onslaught of visitors on a shared Cpanel server I admin slowed it to a crawl, and I was forced to delve into the innards of Cpanel. As a result of this investigation I wrote the "nginx on Cpanel" HOWTO presented below.

Installing Apache module

First of all, when nginx is used as a reverse proxy to Apache, the visitors' IPs received by Apache are wrong - all requests to Apache come from nginx, so the main server IP will be logged.To make Apache log the real IPs of the visitors instead of the main server IP, a special Apache module (mod_rpaf) is needed.Download, untar, cd to the newly created directory and run this command as root:/usr/local/apache/bin/apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.cThat will install the module into the Apache module directory.

Then go to WHM, Main >> Service Configuration >> Apache Configuration > Include Editor > Pre Main Include and add this section there, replacing LIST_OF_YOUR_IPS with the list of IP addresses managed by Cpanel:

LoadModule rpaf_module modules/mod_rpaf-2.0.so

RPAFenable On# Enable reverse proxy add forwardRPAFproxy_ips 127.0.0.1 LIST_OF_YOUR_IPS# which ips are forwarding requests to usRPAFsethostname On# let rpaf update vhost settings# allows to have the same hostnames as in the "real"# configuration for the forwarding ApacheRPAFheader X-Real-IP# Allows you to change which header mod_rpaf looks# for when trying to find the ip the that is forwarding# our requests

Apache configuration changes

Then we need to move Apache to another port, let's take 81 for example. You can simply edit it in the "Tweak Settings" page in WHM, replacing 0.0.0.0:80 with 0.0.0.0:81 or, doing it command line way, edit /var/cpanel/cpanel.config and change port 80 in apache_port assignment to 81:

apache_port=0.0.0.0:81

Run /usr/local/cpanel/whostmgr/bin/whostmgr2 --updatetweaksettings as advised at the top of that file.Check /usr/local/apache/conf/httpd.conf for any occurences of port 80, and run /scripts/rebuildhttpdconf to make sure httpd.conf is up to date.

It also makes sense to reduce the number of Apache children, as nginx will take care of spoonfeeding the data to the clients connecting via the slow network links, freeing Apache children to do their backend work. Edit /usr/local/apache/conf/httpd.conf and replace prefork.c section with this (note that I used very modest values here, and your mileage may vary):

Run /usr/local/cpanel/bin/apache_conf_distiller --update --main to pick up the changes, and then /scripts/rebuildhttpdconf to make sure your changes are in.Note that you will need to watch Apache extended server status at the peak load times to have an idea how many Apache children your server needs by default.

You'll also need to update the Apache port in /etc/chkserv.d/httpd and restart chksrvd with /etc/init.d/chksrvd restart

Generating nginx config files

The final step - we have to build the nginx config file based on the domains hosted on your server.It is done by the simple script which will generate two configuration files for nginx - main one here: /usr/local/nginx/conf/nginx.conf and the include file with all virtual hosts: /usr/local/nginx/conf/vhost.conf

#!/bin/sh

cat > "/usr/local/nginx/conf/nginx.conf" <<EOFuser nobody;# no need for more workers in the proxy modeworker_processes 1;

error_log logs/error.log info;

worker_rlimit_nofile 8192;

events { worker_connections 512; # increase for more busy servers use rtsig; # you should use epoll here for Linux kernels 2.6.x}

Run /usr/local/nginx/sbin/nginx -t to check the configuration, and then /usr/local/nginx/sbin/nginx to start nginx. You are set!

If you don't care about the bandwidth consumed by the virtual hosts and are willing to lose the correct bandwidth calculation over the increased server performance, you can uncomment the <location> lines below the WARNING comment and watch the server picking up the speed. Beware of the two gotchas here: the sub domains most likely will not work as they have document root pointed to a different place, and as nginx doesn't support .htaccess files for the performance reasons, they won't be obeyed for the file types listed.

Obviously, the config file must be regenerated every time a new domain is added. The deleted and suspended domains should work just fine though.

Conclusion

You may ask, is it really worth the trouble? Here's a graph of the average load on the server where nginx was installed as a reverse proxy for Apache as described in this post, can you guess where the switch to nginx happened?

OK, I read that thread but it seems the guy is just trying to downplay this HOWTO. Rewriting of the Location headers with embedded Apache port is handled by the proxy_redirect directives, so there is nothing to worry about here.

He also asserts "the guide has no configuration generation scripts" but that's what this HOWTO is all about - the said script is the main part of it!

In fact I already have a working nginx setup that has the same features as his paid installation service provides - slightly patched nginx that processes the static files but passes the dynamic content to Apache. I just don't have time to wrap it up.

No, or at least not using the technique described in this HOWTO. Apache is needed to handle the dynamic files (PHP, shtml, cgi). Theoretically both shtml and PHP could be processed by nginx, but not CGI. Also, nginx doesn't support .htaccess files Cpanel uses.

Hello,is this tested with latest version of whm/cpanel?I cant find anywhere prefork.cin conf files neither settings declared in it.When i add that to apache conf then sites didnt loaded at all.Also i noticed there is no need for update command,since apache port change works same moment after i put 81 into tweak settings.So i downloaded latest stable version of ngix and install but when i put that text into config i getting error when i type test command about unable to proccess cat directive unkown command.If i delete that line,that i get same error again but then with line where is rsig command.So tehnicly i wont need to change anything beside apache port and nginx config file since i dont need logging of ip adresses and changing prefork settings wont work anyway.So question is why prefork section doesnt work and why i getting error with nginx configuration.

Ok now i managed to install it but there are still some questions remain.On my version of centos and whm chksrvd is called chkservd.What to do with fork section on apache,and what is most important,how to make htaccess to work?When i put htaccess from apache it does work but then image from that domain not loading properly-some loading and some not.Any idea why is that happening considering this is not nginx install it is just proxy mode.

Yes, apparently the service is called chkservd now. I'll update the guide.

> What to do with fork section on apache

Looks like you are not using prefork MPM. Then you have to update whatever MPM configuration your Apache is set up with. You can also leave the configuration unchanged, but in that case Apache will still consume the same amount of memory as before, removing the main advantage of the nginx proxying.

> how to make htaccess to work?When i put htaccess from apache it does work but then image from that domain not loading properly-some loading and some not.Any idea why is that happening considering this is not nginx install it is just proxy mode.

No idea here. Do you see this problem with the images only, or with the pages, too? Did you enable the handling of static file types by nginx (lines labeled with WARNING in the script)?

Nginx doesn't support .htaccess and .htpasswd files (and all associated Apache config directives for the start). All authentication stuff will still be handled by Apache - Nginx will pass the authentication tokens from the browser to Apache. So you have to control the authentication stuff through Cpanel/WHM or by editing .htaccess files.

> Looks like you are not using prefork MPM. Then you have to update whatever MPM configuration your Apache is set up with. You can also leave the configuration unchanged, but in that case Apache will still consume the same amount of memory as before, removing the main advantage of the nginx proxying.

Actualy it seems you puted wrong name into if-it should mpm_prefork.c and not prefork.c.I dont know how did you got only prefork.c since i have mpm_prefork since ages.So i puted mpm_prefork and it works fine now.

> No idea here. Do you see this problem with the images only, or with the pages, too? Did you enable the handling of static file types by nginx (lines labeled with WARNING in the script)?

No i didnt enable handling of static files.

>Nginx doesn't support .htaccess and .htpasswd files (and all associated Apache config directives for the start). All authentication stuff will still be handled by Apache - Nginx will pass the authentication tokens from the browser to Apache. So you have to control the authentication stuff through Cpanel/WHM or by editing .htaccess files.

Actualy i does suport htpasswd files but not htaccess.I will try to repeat install of nginx this time with proper fork installation so let see what happening.Also,i tried different apache configuration-instead suexec and dso and prefork i puted mpM_worker and mod_fcgi for apache and fast cgi for php.But the problem with fast cgi is it doesnt support well zend optimizer so script with zend doesnt work properly.But for someone who doesnt use zend application coulg come handy since it eliminate toomany connection problem(with it there is no slowdown and timeout)

Also one more thing:it seems there is no need to update vhost configuration when new host is added on apache.I removed one domain from vhost and it still worked fine.It seems nginx even if there is no details about domain in vhost it will redirect it properly.I think key is in ip adress,so instead puting details for each domain it should be enough to put ip info like this one:server{listen 127.0.0.1:80;server_name DOMAIN www.DOMAIN;limit_conn zonetwo 200;

Houston we have problem again.When i installed on server two,i had to revert to apache beacuse traffic was marked as 100% proxy traffic for some reason on all sites.So i install it to server one and there was'nt that problem,until i checked two sites where that problem showed again while other sites are ok even on same ip and domain account.Mod rpa is installed and properly configured.I noticed there is upstream response error for that two sites,and i fixed that by increasing proxy values to 64 128k But even after that and reseting nginx i still getting all traffic marked as proxy traffic.Before i mentioned there was a problem with half pictures not showing.I resolved that problem by increasing number of workers to 2048(since i saw error about it in log)So i wonder could be toomuch workers problem with those two sites where all traffic is showed as proxy in trade script(that is script which tracks traffic)

But if i add new domain (virtualhost) by WHM then i should run your script manualy to generate vhost.conf for nginx? If, yes is there possibility to do that automaticlly when we add new domain (account) to WHM?

cuci:Yes, accessing by IP for the dedicated IP accounts doesn't work with the version of the config file generator I've published. However http://domain should always work.

You can make dedicated IP addresses work by altering the script a little: - change "listen 80;" to "listen $IP:80;" - add another line after two proxy_redirect lines:"proxy_redirect http://$IP:81 http://$IP;"

Łukasz > But if i add new domain (virtualhost) by WHM then i should run your script manualy to generate vhost.conf for nginx? If, yes is there possibility to do that automaticlly when we add new domain (account) to WHM?

Yes, you'll have to regenerate the nginx config on every domain addition. You can automate this by using Cpanel hooks:http://www.cpanel.net/support/docs/hooks.htm#postwww

This is a great tutorial, follow the steps and everything seem to be working. Just have a little problem with file downloading, mostly on slow connection.

On broad band connection i can download file (zip, rar) just fine. but on slow connection. the download get disrupt and the download could never get finish. It vary, for the same file some time it stop at 10% complete, sometime it at 30%. What setting should i change to get it working correctly with slow connection

/usr/local/nginx/proxy_temp should be owned by "nobody" user with 0700 permissions. If you want to be on the safe side, you can do "chmod 777 /usr/local/nginx/proxy_temp" to allow all users to write to that directory.

This is a great tutorial. I am having a little problem on the last step. I have create nginx.sh and copy-pasted the script to generate configuration files and ran it by giving 755 permission but I am getting these errors:

./nginx.sh: line 3: /usr/local/nginx/conf/nginx.conf: No such file or directory/bin/cp: cannot create regular file `/usr/local/nginx/conf/vhost.conf': No such file or directoryConverting domain.com for user./nginx.sh: line 59: /usr/local/nginx/conf/vhost.conf: No such file or directoryConverting domain.com for user./nginx.sh: line 59: /usr/local/nginx/conf/vhost.conf: No such file or directory

One suggestion though that's worked out great for me - instead of generating the vhost config file for nginx based on cPanel data, why not just rip all the subdomain data from Apache's config itself, which would then have the proper document roots?

Sushant, it looks like nginx is installed in some other place than /usr/local/nginx on your server. You should correct the config file path in the script accordingly.

Anonymous @ 28 May, 2009Yes, that would work, if you're able to write a good parser for httpd.conf. Personally I think it's a hassle, and a proper way would be to grab the domain data directly from /var/cpanel/userdata. It's stored in YAML so parsing it would be easier than httpd.conf, and not that ambiguous, too.

You can use any Apache MPM with this guide, there is nothing specific to the prefork MPM except the StartServer/MaxClient tweaking part which obviously needs to be adjusted accordingly. Or you can leave it alone, it wouldn't affect the site operation, except for a small waste of memory on the extra Apache worker threads.

Hello, thx for how-to. One question: why not set apache listen to 127.0.0.1:80 and nginx to IP:80, that way they both coexist smootly, but apache does not listen on external interface which can be security advantage over configuration where apache listen on external interface's port 81 or whatever.

I have a note - vhost generation script uses wrong $ROOT dir when root dir not within public_html.DOMAIN can be in /home/ACCOUNT/ or /home/ACCOUNT/public_html/SUBDIR - but in all cases $ROOT will be /home/ACCOUNT/public_html.

That's broke many things.Better to get real $ROOT from cPanel configs or maybe from httpd.conf ?