from ServerBeach to Amazon AWS/EC2

I’ve had a dedicated server at ServerBeach for about 5 years, overall they were pretty solid, one time the hard drive died but they were able to mount the dead drive in read-only mode so I could recover all my data. After they were acquired by Peer 1, there were some connectivity issues, but after a few tickets it was pretty solid again.

My ServerBeach server was so old that it was one of the EOL (end-of-life) servers, which means they no longer stocked spare parts for them, which is kindda not good. Ray and I have been hosting our Ruby on Rails projects on EC2 for quite a while, OnMyList is also hosted on EC2 with software load-balancing, multiple app and database instances. I had always planned to consolidate things and move my ServerBeach setup to AWS/EC2, but I never got around to it. About a month ago ServerBeach sent me an email saying that they had to reassign the IP addresses on the server, that is a bit of a pain coz I would have to change the DNS on all the domains, so that pretty much gave me enough incentive to move.

Ray and I started with the Intrepid Ubuntu AMI by Eric Hammond. To get a static IP, I allocated an Elastic IP Address, you get one Elastic IP for free per instance, and you can assign it to any instance. We also use Elastic Block Store (EBS) for persistence storage, the entire /home, the MySQL database, and most of the important configuration files in /etc are symlinked to the EBS volume. With EBS you can easily create snapshots for backups by using the ec-create-snapshot command included in the EC2 tools.

After this you can add a line to /etc/fstab to mount the EBS volume to a mountpoint.

Everything on the server was pretty much installed from APT, we went with BIND9 for DNS. Rails apps are hosted with Passenger as usual. God is used for monitoring. We host the primary DNS for all our domains on the instance, and secondary DNS’ are hosted at EditDNS. I chose EditDNS because they support SRV records, and we need that for Jabber. All our domains use Google Apps, there is no reason to run my own mail server when Google provides superb reliability, spam filtering, secure IMAP and SMTP services, the gmail web interface is also way better than any open source webmail package I’ve seen. Pretty much all the hosted domains share the same zone file, which makes DNS management super easy. If you got BIND9 setup but the server ain’t answering remote queries, check to see that the port (53) is enabled, with EC2 you have to manually open up ports, most ec2 setup instructions open up port 22 for SSH and 80 for HTTP, if you need anything else you need to authorize them yourself:

ec2-authorize default -P udp -p 53

For Rails apps we setup a deployer user to make things really simple, the Capistrano recipe is pretty straight forward. We use remote_cache with Git to make deploys very very fast.

To migrate user data over from my old server, I first created the users on the new system manually, I didn’t feel comfortable copying over the /etc/passwd, shadow files, and what not, also I had to get rid of quite a few old accounts. After the users and home directories have been created, I used rsync via SSH to copy everything over, I ran rsync with quite a few exclusions, like I wanted to exclude all the Maildir’s and the spamassassin bayesian database files coz I didn’t need them anymore, and a lot of them were pretty huge. If you run rsync in archive mode (rsync -a) the permissions and ownerships are sync’ed automatically to the users and groups with the same names. I also had to migrate Apache1 virtual host configs to Apache2 ones, but that was pretty straight forward. SSL setup was also pretty trivial in Apache2, I moved the hostname over and was able to reuse my InstantSSL certificate.

To migrate MySQL databases, we just did mysqldump’s and loaded them on the new server. I realize this probably wouldn’t work if you’re migrating a high traffic site, but fortunately we could tolerate a few hours of downtime.

After everything was running smoothly, some of our domains had longer TTL values and people were still going to the old IP. So I setup Pound on the old server to forward all HTTP and HTTPS traffic over to the new IP, worked like a charm. I went with Pound because that was the only one available in APT on the old Debian sarge server. Next time I do a migration like this, I will make sure to lower the TTL of the domains a couple of days pre-migration, but even then I am assuming DNS’ obey the specified TTL’s, and most probably don’t. I use OpenDNS and it was pretty easy to expire their cache manually.

It’s been a few weeks and the instance has been rock solid, amazingly it is much higher performance than the ServerBeach dedicated server, but since the server was over 5 years old, that is not saying much. Cost-wise, the EC2 instance is about $20 cheaper than a dedicated host (I was paying about $90 a month for the server + 2 additional IPs). Ray and I have been using EC2 heavily for over a year, so to scale horizontally with more instances is pretty trivial, and it’s far quicker than getting another dedicated server setup with a hosting provider.

There is one step which i am still not getting about this ec2 / ebs stuff. With a simple webapp, iâ€™ll have some log, the webapp itself (maybe rails, maybe php), and my database

so iâ€™d probably setup maybe 2 ebs volumes (webapp+logs and database). then iâ€™d copy my database and webapp to the volumes, create symlinks (or change the config), tweak the config of my instance, upload my ssl keys for the webserver and so onâ€¦.

butâ€¦ when i stop/restart my instance i have persitance logs, webapp and databaseâ€¦ but still my whole config is gone, my ssl keys are gone, my installed services are goneâ€¦ shouldnâ€™t there be a way to store the main volume itself on ebs or am i missing something?