Monday, 29 December 2014

After running "cap production deploy" I had a mysterious case of "ActionControllerException:Mysql2::ErrorCan't connect to MySQL server on '127.0.0.1' (111)". The settings in database.yml were correct. I could verify that the database was, indeed, accessible by using these settings in MySQL Workbench to ssh tunnel to the database on the VPS from my laptop. So what was going on?

It turned out, that the application was deployed in development, not production despite the capistrano command and the settings in deploy/production.rb. And in database.yml I had incorrect settings in the development block - they were supposed to work only on my laptop but not at the VPS.

It was a great relief to find the explanation of that mysterious error. But how could be that "cap production deploy" produced a deployment in a development environment, rather than in production? I don't have 100% certainty as I changed a number of things while trying to find the culprit but the most likely reason was the following line in /etc/nginx/nginx.conf:

server {
rack_env development;

Once I changed that setting and re-started nginx the problem disappeared. I am relieved but I still find the whole thing a bit troubling. Somehow I cannot escape the feeling that the rails stack is really messy and fragile - you can specify the same setting (in this case the environment) in a number of different places (in fact in different layers of the stack) so that you get inconsistency and unexpected behaviour. God knows what other settings are now over-ridden but haven't yet manifested themselves in an error. This is a bit frightening.

Anyway - you have to live with it if you want to use rails. For those who are suspecting a similar problem in their deployment:

The best way to tell which environment your application is running in is to look at the logs (/home/deploy_user/my_app/shared/log). In my case the file development.log had a current timestamp while production.log was stale (a few weeks old). That sort of nailed it. However, the situation could be a bit more complicated if you are running different instances of the application at different ports - they could run in different environments. Than you need to look up the logs and find which one has the error message that you are getting via the browser.

location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
# Uncomment to enable naxsi on this location
# include /etc/nginx/naxsi.rules
}

}

The error message I found in

/var/log/nginx/error.log

was the same I found earler: "directory index of "/home/deploy/appmate/current/public/" is forbidden". Back than I resolved the issue by changing the privileges of the parent directories. So there was no reason I had to do that again!

With a bit of googling I found the culprit in the location block. By following this recipe. I removed $uri/ from the try_files command. I started getting 404 instead of 403. When I removed the whole location block all error messages disappeared.

Monday, 11 August 2014

As a part of a New Year's resolution I started learning Ruby on Rails (it was more than 5 years since I last picked up a new programming language!). Ruby and Rails are really fun to learn especially if you start with a toy application. But I found Rails deployment to be a bit more involved than Java deployment. I don't mean using a platform such as Heroku - this really removes all the grunt out of it. I mean deploying Rails while keeping control over the entire stack - i.e. starting from bare metal.

Deploying Rails

There are tons of tutorials on this subject but the clearest I could find is the one written by Chris Oliver. I followed it quite literally. I chose the Ubuntu 12.04 and rvm and I've been using the most recent versions of the other packages as available from the Ubuntu 12.04 repositories:rnginx/1.6.0mysql/5.5.38-0ubuntu0.12.04.1Phusion Passenger version 4.0.48*** LOCAL GEMS ***bigdecimal (1.2.4)bundler (1.6.2)bundler-unload (1.0.2)executable-hooks (1.3.2)gem-wrappers (1.2.4)io-console (0.4.2)json (1.8.1)minitest (4.7.5)psych (2.0.5)rake (10.1.0)rdoc (4.1.0)rubygems-bundler (1.4.4)rvm (1.11.3.9)test-unit (2.1.2.0)On the dev machine I installed, among others,capistrano (3.2.1)Despite following the tutorial to the letter I was getting HTTP 500 after deployment.

Setting the correct path for passenger_ruby

I posted a question on Chris's blog and he kindly suggested that Nginx is the likely culprit. After inspecting/var/log/nginx/error.logI found the following error message

You've set the `PassengerRuby` (Apache) or `passenger_ruby` (Nginx) option to '/home/deploy/.rvm/rubies/ruby-2.1.2/bin/ruby'. However, because you are using RVM, this is not allowed: the option must point to an RVM wrapper script, not a raw Ruby binary. This is because RVM is implemented through various environment variables, which are set through the wrapper script.
To find out the correct value for `PassengerRuby`/`passenger_ruby`, please read:
https://www.phusionpassenger.com/documentation/Users%20guide%20Apache.html#PassengerRuby
https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#PassengerRuby

The error message is handily descriptive and checking the above links makes it clear that I had to execute

$ which passenger-config

to find the path to the passenger-config executable. Running it produced:

About Me

In my day job I am the Head of Biocomputing at the MRC National Institute for Medical Research in London, UK. I am using this blog mainly as a place to put "notes to self" about various technical issues that I encounter. Needless to say, any opinion in this blog is my own and may not reflect that of my employer.