Main menu

Migrate sites from one Aegir to another

30 December 2011

We recently needed to migrate all our sites on one physical server to another server, there were more than 200 sites, and they were all hosted with Aegir. The old server was to be decommissioned, so we had to move all of Aegir's data about the site to the new server import into a new Aegir master on the new server. We also needed to do this with as small amount of downtime as possible.

In the end we migrated all the sites with about 30 seconds of downtime each, here's how:

The setup before

For clarity, here's a simplified diagram of the infrastructure before the migration:

You can see we have two servers behind a firewall, the firewall basically is there to translate the public IP address into an internal IP address of the actual web server.

We set up the two webservers to be as identical as possible, in terms of Aegir installed and the names of the platforms hosting the sites on them etc. but this process is actually quite flexible, so for example, we actually had a dedicated DB server on our old infrastructure, but not in the new.

We installed the code I'm about to describe on both machines, into ~aegir/.drush/ and additionally we ensured that the aegir user on the new machine could ssh into the old machine as the aegir user with its ssh key. We also created a Drush alias for the 'old_hostmaster' by creating the file: ~aegir/.drush/aliases.drushrc.php and adding the entry:

Reducing downtime

To reduce the amount of downtime during the server migration, we employed the apache module: mod_proxy. This allows the apache server on the new webserver to function as a reverse proxy server. Traffic arriving at the new server gets forwarded to the old one and visitors don't notice any difference.

Setting up 200 or more virtual host files with all the correct details is a pain though, and so naturally we wrote a Drush command to do it for us.

First we need to get a list of sites on an Aegir platform, which we can do like this:

// This is a bit of a fragile way to return the data, but I couldn't seem to // get Drush to pass the structured data back properly, so we'll do this, // which works. drush_print_r(serialize($sites));}

Because this is a Drush command itself, we can call this on a 'remote' server. So we can run a command from the new server, and go get all the sites we need to migrate from the old server.

This is a fairly straightforward command in which we go an get a list of sites on a particular platform on the old server, and then create a provisionConfig_computerminds_proxy for site and write it. The provisionConfig_computerminds_proxy class looks like this:

We ran this Drush command before starting the migration, and actually because our new server had a public IP address of its own, we could test beforehand that access one of the sites domains at that IP, we actually accessed the site on the old server.

Switching IPs

Once we had our servers set up we were then good to start the migration process, the first stage was to re-assign the public IP address of the sites to the new server, instead of being attached to the old, this was the traffic flow:

Note, that if you don't have the ability to re-assign public addresses between servers, then you could switch the DNS entries over to the new IP, and just wait until all the traffic is hitting the new server and not the old one.

Doing the actual migration

The actual migration process is pretty simple, we take advantage of Aegir's built-in Drush commands to do all of the heavy lifting.

Here is the complete Drush command to migrate one site from the old server to the new one.

Note that we copy the context from the old server, so that settings stored about the site are also copied over to the new server. We reset a few of those, such as the database server.

We wanted our site's Aegir generated vhost to be used in place of the proxied one we created earlier, so we implemented a couple of Drush hooks to hook into the above command and remove the proxy vhost when it is run, and put it back if the command to migrate the site fails:

Finally, to be able to migrate all the sites we have a simple Drush command that gets a list of sites that could be migrated and compares that to the sites on the current server, and offers to migrate them:

/** * Drush command to migrate a lot of sites from an old Hostmaster to this one. */function drush_computerminds_migrate_migrate_computerminds_all() { // Get the remote sites. $remote_sites = computerminds_migrate_get_all_sites('@' . COMPUTERMINDS_OLD_HOSTMASTER_NAME, COMPUTERMINDS_PLATFORM_NAME);

// Get the local sites. $local_sites = computerminds_migrate_get_all_sites([email protected]', COMPUTERMINDS_PLATFORM_NAME);

We had to add a --limit option because Drush creates so many log messages that the parent Drush process asking all the other processes to do the work ran out of memory.

The actual command we ran to do the migration was:

drush @hostmaster migrate-computerminds-all --limit=50

And we just kept running it until all the sites had been migrated. Each site will in turn be taken off-line, by removing the proxy vhost, and then backed up, migrated, and imported onto the new server. This took about 30 seconds per site for us. After a few hours or so our setup looked like this:

We were then free to remove the old server after confirming that no traffic was being routed to it. It should be noted that apart from installing this code onto the old server, we've not actually changed anything on it, so if something goes wrong we could just switch the IP back in the firewall and go back to serving sites from the old server until we fixed the problem.

About the author

Steven has a wealth of experience building Drupal-based websites and tools associated with them. He is a maintainer of the Aegir project and has contributed to many other projects on Drupal.org. He is currently our infrastructure lead and is a lead architect on many of our projects.

He is a Master of Mathematics graduate from the University of Warwick and got involved in Drupal creating a website for the Maths society there.