Building for Production: Web Applications — Backups

Introduction

After coming up with a recovery plan for the various components of your application, you should set up the backup system that is required to support it. This tutorial will focus on using Bacula as a backups solution. The benefits of using a full-fledged backup system, such as Bacula, is that it gives you full control over what you back up and restore at the individual file level, and you can schedule backups and restores according to what is best for you.

Solutions such as DigitalOcean Droplet Backups (snapshot backups of your entire Droplet) are easy to set up and may be sufficient for your needs, if you only require weekly backups. If you opt for DigitalOcean Backups, be sure to set up hot backups of your database by following the Create Hot Backups of Your Database section.

In this part of the tutorial, we will set up a Bacula to maintain daily backups of the required backups of the servers that comprise your application setup (db1, app1, app2, and lb1), defined previously in our recovery plan—essentially, this is a tutorial that shows you how to use Bacula to create backups of a LAMP stack. We will also use Percona XtraBackup to create hot backups of your MySQL database. Lastly, we will use rsync to create a copy of your backups, on a server in a remote data center. This will add two servers to your setup: backups and remotebackups (located in a separate data center).

Let's get started.

Install Bacula on Backups Server

Then follow the Organize Bacula Director Configuration (Server) section of this tutorial: How To Back Up an Ubuntu 14.04 Server with Bacula. You will need the Director Name when setting up the Bacula clients (on the servers you want to back up). Stop when you reach the Install and Configure Bacula Client section.

Note that we will be using the RemoteFile pool for all of the backups jobs that we will be setting up. With that said, you may want to change some of the settings before proceeding.

Install Bacula Client on Each Server

Install the Bacula client on each server that you want to back up (db1, app1, app2, and lb1) by following the Install and Configure Bacula Client section of this tutorial: How To Back Up an Ubuntu 14.04 Server with Bacula. Stop when you reach the Add FileSets (Server) section.

Note that you will need the FileDaemon Name (usually the hostname appended by "-fd") and the Director Password (the password that the Bacula server will use to connect to each client) from the bacula-fd.conf file on each server.

Add Bacula Clients to Backups Server

On backups, the Bacula server, add a Client resource to the /etc/bacula/conf.d/clients.conf file for each server that you installed the Bacula client on.

Open the clients.conf file:

sudo vi /etc/bacula/conf.d/clients.conf

Here is an example of the Client resource definition for the database server, db1. Note that the value of Name should match the the Name of the FileDaemon resource and the Password should match the Password of the Director resource, on the client server—these values can be found in /etc/bacula/bacula-fd.conf on each Bacula client server:

Create a similar Client resource for each of the remaining Bacula client servers. In our example, there should be four Client resources when we are finished: db1-fd, app1-fd, app2-fd, and lb1-fd. This configures the Bacula Director, on the backups server, to be able to connect to the Bacula client on each server..

Create XtraBackup Script

Percona XtraBackup is ready to create hot backups of your MySQL database, which will ultimately be backed up by Bacula (or DigitalOcean Backups), but the hot backups must be scheduled somehow. We will set up the simplest solution: a bash script and a cron job.

Create a bash script called run_extra_backup.sh in /usr/local/bin:

sudo vi /usr/local/bin/run_xtrabackup.sh

Add the following script. Be sure to substitute the user and password with whatever you set up when you installed XtraBackup:

Save and exit. Running this script (with superuser privileges) will delete the existing XtraBackup backup at /data/backups/full and create a new full backup. More details about creating backups with XtraBackup can be found in the Perform Full Hot Backup section of the of the XtraBackup tutorial.

Make the script executable:

sudo chmod +x /usr/local/bin/run_xtrabackup.sh

In order to properly backup our database, we must run (and complete) the XtraBackup script before Bacula tries to backup the database server. A good solution is to configure your Bacula backup job to run the script as a "pre-backup script", but we will opt to use a cron job to keep it simple.

Create a cron configuration file (files in /etc/cron.d get added to root's crontab):

sudo vi /etc/cron.d/xtrabackup

Add the following cron job:

/etc/cron.d/xtrabackup

30 22 * * * root /usr/local/bin/run_xtrabackup.sh

This schedules the script to run as root every day at 10:30pm (22nd hour, 30th minute). We chose this time because Bacula is currently scheduled to run its backup jobs at 11:05pm daily—we will discuss adjusting this later. This allows 35 minutes for the XtraBackup script to complete.

Now that the database hot backups are set up, let's look at the Bacula backup FileSets.

Configure Bacula FileSets

Bacula will create backups of files that are specified in the FileSets that are associated with the backup Jobs that will be executed. This section will cover creating FileSets that include the required backups that we identified in our recovery plans. More details about adding FileSets to Bacula can be found in the Add FileSets (Server) section of the Bacula tutorial.

Application Server Backup Jobs

For our application servers, we will create two backup jobs named "Backup app1" and "Backup app2". The important thing here is that we specify the correct Clients (app1-fd and app2-fd) and FileSet (Apache DocumentRoot).

Load Balancer Server Backup Job

For our load balancer server backup job, we will create a new job named "Backup lb1". The important thing here is that we specify the correct Client (lb1-fd) and FileSet (SSL Certs and HAProxy Config):

Now our backup Jobs are configured. The last step is to restart the Bacula Director.

Restart Bacula Director

On the backups server, restart the Bacula Director to put all of our changes into effect:

sudo service bacula-director restart

At this point, you will want to test your client connections and backup jobs, both of which are covered in the How To Back Up a Server with Bacula tutorial. That tutorial also covers how to restore Bacula backups. Note that restoring the MySQL database will require you to follow the Perform Backup Restoration step in the Percona XtraBackup Tutorial.

Review Backups Schedule

The Bacula backups schedule can be adjusted by modifying the Bacula Director configuration (/etc/bacula/bacula-dir.conf). All of the backup Jobs that we created use the "DefaultJob" JobDef, which uses the "WeeklyCycle" schedule, which is defined as:

Full backup on the first Sunday of a month at 11:05pm

Differential backups on all other Sundays at 11:05pm

Incremental backups on other days, Monday through Saturday, at at 11:05pm

You can verify this by using the Bacula console to check the status of the Director. It should output all of your scheduled jobs:

Feel free to add or adjust the schedule of any of your backup jobs. It would make sense to modify the schedule of the application servers to occur at the same time that the Percona XtraBackup script is executed (10:30pm). This will prevent the application and database backups from being inconsistent with each other.

Set Up Remote Backups

Now we're ready to set up a remote server that will store copies of our Bacula backups. This remote server should be in a geographically separate region so you will have a copy of your critical backups even if there is a disaster in your production data center. In our example, we will use DigitalOcean's San Francisco (SFO1) region for our remotebackups server.

We will explain a simple method to send our backups from our backups server to our remotebackups server using public SSH keys, rsync, and cron.

On the remotebackups server, create a user that will be used for the rsync login.

Next, on the backups server, generate a password-less SSH key pair as root. Install the public key on the remotebackups user that you just created. This is covered in our How To Set Up SSH Keys tutorial.

On the backups server, write up an rsync command that copies the Bacula backup data (/bacula/backup) to somewhere on the remotebackups server. Rsync usage is covered in our How To Use Rsync tutorial. The command will probably look something like this:

After you set all of this up, verify that there is a copy of your backups on the remotebackups server the next day.

Other Considerations

We didn't talk about the disk requirements for your backups. You will definitely want to review how much disk space your backups are using, and revise your setup and backups schedule based on your needs and resources.

In addition to creating backups of your application servers, you will probably want to set up backups for any other servers that are added to your setup. For example, you should configure Bacula to create backups of your monitoring and centralized logging servers once you get them up and running.

Conclusion

You should now have daily backups, and a remote copy of those backups, of your production application servers. Be sure to verify that you are able to restore the files, and add the steps of restoring your data to your recovery plans.

Tutorial Series

This 6-part tutorial will show you how to build out a multi-server production application setup from scratch. The final setup will be supported by backups, monitoring, and centralized logging systems, which will help you ensure that you will be able to detect problems and recover from them. The ultimate goal of this series is to build on standalone system administration concepts, and introduce you to some of the practical considerations of creating a production server setup.

May 28, 2015

This tutorial will demonstrate how to plan and set up a sample production application from start to finish. Hopefully, this will help you plan and implement your own production server environment, even if you are running a different application on a completely different technology stack. Because this tutorial covers many different system administration topics, it will often defer the detailed explanation to external supporting articles that provide supplemental information.

May 28, 2015

In this part of the "Building for Production: Web Applications" series (2 of 6), we will deploy our example PHP application, WordPress, and a private DNS. Your users will access your application over HTTPS via a domain name, e.g. "https://www.example.com", that points to the load balancer. The load balancer will act as a reverse proxy to the application servers, which will connect to the database server.

May 28, 2015

In this part of the "Building for Production: Web Applications" series (3 of 6), we will devise a recovery plan. A recovery plan is a set of documented procedures to recover from potential failures or administration errors within your server setup. Creating a recovery plan will also help you identify the essential components and data of your application server setup.

May 28, 2015

In this part of the "Building for Production: Web Applications" series (4 of 6), we will set up the backup system that is required to support our recovery plan. This tutorial will focus on using Bacula as a backups solution. The benefits of using a full-fledged backup system, such as Bacula, is that it gives you full control over what you back up and restore at the individual file level, and you can schedule backups and restores according to what is best for you.

May 28, 2015

In this part of the "Building for Production: Web Applications" series (5 of 6), we will add monitoring to improve our awareness of the state of our servers and services. Monitoring software, such as Nagios, Icinga, and Zabbix, enables you to create dashboards and alerts that will show you which components of your application setup need attention. The goal of this is to help you detect issues with your setup, and start fixing them, before your users encounter them.

May 28, 2015

In this part of the "Building for Production: Web Applications" series (6 of 6), we will set up centralized logging for our production application setup. Centralized logging is a great way to gather and visualize the logs of your servers. Generally, setting up an elaborate logging system is not as important as having solid backups and monitoring set up, but it can be very useful when trying to identify trends or problems with your application.