How to Create DigitalOcean Snapshots Using Packer on CentOS 7

Introduction

Packer, by Hashicorp, is a command-line tool for quickly creating identical machine images for multiple platforms and environments. With Packer, you use a configuration file, called a template, to create a machine image containing a preconfigured operating system and software. You can then use this image to create new machines. You can even use a single template to orchestrate the simultaneous creation of your production, staging, and development environments.

In this tutorial, you'll use Packer to configure an Nginx web server on CentOS 7. You'll then use Packer to create a snapshot of this Droplet and make it immediately available in your DigitalOcean dashboard so you can use it to create new Droplets.

Prerequisites

Before you can get up and running with Packer you will need a few things.

Once downloaded, install the unzip utility and use it to unzip the package contents into the /usr/local directory, the recommended location to make Packer available to all users.

sudo yum install -y unzip

sudo unzip -d /usr/local packer_0.12.2_linux_amd64.zip

CentOS already includes a program called packer, and while you could simply type out the full path each time you run a command, a more efficient method to work around this issue is to create a symlink that maps packer.io to /usr/local/packer. Make the symlink in the /usr/local/bin folder with the following command:

sudo ln -s /usr/local/packer /usr/local/bin/packer.io

Verify that the installation was successful by checking that packer.io is available on the command line:

packer.io

A successful installation will output the following:

Output

usage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
fix fixes templates from old versions of packer
inspect see components of a template
push push a template and supporting files to a Packer build service
validate check that a template is valid
version Prints the Packer version

Packer is now installed and working on your machine. In the next step you will set up a project directory and configure the template to produce a basic CentOS snapshot.

Step 2 — Configuring the DigitalOcean Builder

We want Packer to create a Droplet, install some software and configuration files, and then turn that Droplet into an image we can use to create new machines. Packer uses a configuration file called a template that contains all the details that tell Packer how to build an image. We write this configuration using JSON, a common format for configuration files.

In Packer-speak, a builder is a JSON object that contains the blueprint for the image you want Packer to create. Using the digitalocean builder, you are going to instruct Packer to create a 512 MB CentOS 7.3 Droplet that will be launched in the NYC1 region.

Create and change to a new directory which will hold the template and configuration files we'll create in this tutorial:

mkdir ~/packerProject

cd ~/packerProject

Now that you have a project directory, open a new file called template.json, in your text editor:

vi ~/packerProject/template.json

Each builder needs to go into the builders section of template.json. Add this section now and include the digitalocean builder by placing this code into the file:

Packer connects to Droplets using the ssh_username value. This value needs to be set to "root" in order for Packer to work properly.

Save template.json and exit your text editor.

The preceding block of code contains the minimum amount of configuration needed to create a DigitalOcean Droplet but there are additional configuration options available, as shown in the following table:

Key

Value

Required

Description

api_token

String

Yes

The API token to use to access your account. It can also be specified via environment variable DIGITALOCEAN_API_TOKEN, if set.

You now have a valid template, but your API token is hard-coded in your template. This is a bad practice and a potential security risk. In the next step you will create a variable for this token and move it out of template.json.

Step 3 — Creating and Storing User Variables

Packer lets you create and store the values of variables in a separate file. This file can then be passed to Packer via the command line when you are ready to build your image.

Storing variables in a separate file is an ideal way to keep sensitive information or environment-specific data out of your template. This is crucial if you intend to share it with team members or store it in a public facing repository such as GitHub.

Even if you will only be saving a local copy, it's a Packer best practice to store variables outside of a template.

Create and open a new JSON file in the packerProject directory to store this information:

vi ~/packerProject/variables.json

Now, add a my_token variable and set its value to your DigitalOcean API token:

~/packerProject/variables.json

{
"my_token": "YOUR_DIGITALOCEAN_API_TOKEN"
}

Save variables.json and exit your editor.

Now let's configure our template to use variables. Before you use the my_token variable, or any other variable, you first need to tell Packer the variable exists by defining it in a variables section at the beginning of the template.json file.

Open template.json in your editor:

vi template.json

Add a new variables section above the builders section you previously defined. Within this new section, declare the my_token variable and set its default value to an empty string:

~/packerProject/template.json

{
"variables": {"my_token":""},
"builders": [
...
}

Variables defined in the variables section are available globally.

Next, replace your API token in the builders section with a call to my_token:

As you can see, calls to user variables must use a specific format: "{{ user `variable_name` }}. The quotes and the backticks are required, as are the double curly braces.

Save the file and exit the editor.

You now have a working template that produces a basic snapshot and a separate variables file to store your API key. Before you validate and build your image, let's add a provisioners section to our template which will configure Packer to install and set up the Nginx web server on the machine before creating the image.

Step 4 — Configuring Provisioners

The provisioners section is where Packer installs and configures software on the running Droplet before turning it into a machine image. Like builders, there are different types of provisioners you can use to configure a Droplet.

In order to configure Nginx, you are going to use Packer's file provisioner to upload configuration files to the server, and then use the shell provisioner to execute an installation script that uses those files. The file provisioner lets you move files and directories to and from a running machine before it is turned into an image. With the shell provisioner, you can remotely execute shell scripts on that machine.

Provisioners execute in the same order in which they appear within the template. This means putting the file provisioner first since your shell scripts need the uploaded files.

Add a provisioners section immediately following the builders section in template.json and set the two provisioners you will use:

The file provisioner requires a source, which points to a local file path, and a destination, which points to an existing file path on the running machine. Packer can only move files to destinations that already exist. For this reason, we generally upload files to the /tmp directory.

Configure the file provisioner by adding the highlighted lines to template.json:

Scripts must be listed individually, which allows you to control the execution order of the scripts.

The provisioners section of your template is complete. Save the file and exit Vim.

Now let's create the shell scripts and configuration files that Packer will use to create your image.

Step 5 — Adding Configuration Files and Installation Scripts

We want our image to ship with a fully-configured Nginx installation, with the proper configuration files and a default web page. In this section, you'll create these files from some predefined configuration based on the tutorial How To Set Up Nginx Server Blocks (Virtual Hosts) on Ubuntu 16.04, as Nginx configuration is beyond the scope of this tutorial.

We'll provision the server with Nginx by creating and uploading three separate configuration files that are handled by a single installation script.

First, create a new directory within the project folder to store the configuration files.

mkdir ~/packerProject/configs

Change to /configs to create your Nginx configuration files:

cd ~/packerProject/configs

First, you need a default web page to serve from your new domain. Create the file index.html.new:

vi index.html.new

In this new file, insert the following:

~/packerProject/configs/index.html.new

HELLO FROM YOUR TEST PAGE

Next, you need an Nginx configuration file that defines the server block for your domain that defines the listening port and the location of your web pages for the domain. Create a file called newDomain.conf:

In this example, we're using example.com as a placeholder value. When you create a new machine from your image, you'll have to log in to the new machine and change this file to reflect the actual domain or IP address that points to the machine.

Finally, you want Nginx to load your domain's configuration from a new directory, /etc/nginx/vhost.d/. This means editing the main Nginx configuration file.

Create nginx.conf.new:

vi nginx.conf.new

We'll use a default Nginx config file, but we'll modify it to include our specific site configuration. Put the following contents into this file:

Your template is finished and you're now ready to validate and build your snapshot.

Step 6 - Validating and Building the Droplet

It's time to test your template using Packer's validate subcommand. Once your template validates successfully you will build your Droplet and create the snapshot.

Change to the root of your project:

cd ~/packerProject

The validate subcommand will check your template for valid syntax and configuration options:

packer.io validate -var-file=variables.json template.json

The -var-file flag reads variables.json and sets the value for my_token within template.json.

You'll see the following output:

Output

Template validated successfully.

If there is something wrong with template.json you will get an error message. This message will vary depending on the error but most can be fixed by double checking syntax and correcting any typos.

The build subcommand runs the build that you defined in the builders section of your template. In other words, it tells Packer to build your Droplet and then create a snapshot of that Droplet in your DigitalOcean dashboard.

Call packer.io build to build the Droplet and create the snapshot:

packer.io build -var-file=variables.json template.json

Note that the -var-file flag operates in the exact same manner for both the build and validate subcommands.

Troubleshooting

Occasionally, you may run into an issue that isn't adequately explained by the error message. In these scenarios, you can extract more detail about your build by enabling debug mode, inspecting the Packer logs, or both.

Debug mode provides builder-specific debugging information for each step in a remote build. Enabling debug mode for a DigitalOcean build will also produce a temporary private key in your project folder which you can use to connect to and inspect a running Droplet before it is turned into a snapshot.

You can enter debug mode by passing the -debug flag to packer.io build on the command line:

packer.io build -debug --var-file=variables.json template.json

If you are unable to diagnose the issue in debug mode you can try enabling the Packer logs. These logs are primarily used to debug local builders, but they may provide helpful information on remote builds as well.

To enable the Packer logs, set the PACKER_LOG environmental variable to any value except "0" or an empty string:

PACKER_LOG=1 packer build --var-file=variables.json template.json

Logs will print to the console unless you also set the PACKER_LOG_PATH environmental variable.

If you are still having problems, you may want to try reaching out to someone in the Packer community.

Conclusion

Now that you are comfortable with the basics of Packer, you may be interested in building on this foundation.

Try adding a second builder to your template to create a local testing environment alongside your DigitalOcean snapshot. The virtualbox-iso builder, for example, produces images for VirtualBox, a free, open-source virtualization product used by both enterprises and hobbyists. You can define a post-processor to the VirtualBox image and create Vagrant environments that mirror your DigitalOcean snapshots. This will allow you to test website changes locally before pushing them to a live Droplet. You can learn more in the Vagrant post-processor documentation.

Or you may want to connect your web server to a database. Add a second digitalocean builder and use the only key in your provisioners section to apply different provisioning to each build.