How To Install Puppet To Manage Your Server Infrastructure on Debian Jessie

Introduction

Puppet, from Puppet Labs, is a configuration management tool that helps system administrators automate the provisioning, configuration, and management of a server infrastructure. Planning ahead and using config management tools like Puppet can cut down on time spent repeating basic tasks, and help ensure that your configurations are consistent and accurate across your infrastructure. Once you get the hang of managing your servers with Puppet and other automation tools, you will free up time which can be spent improving other aspects of your overall setup.

Puppet comes in two varieties, Puppet Enterprise and open source Puppet. It runs on most Linux distributions, various UNIX platforms, and Windows.

In this tutorial, we will cover how to install open source Puppet in an Agent/Master setup. This setup consists of a central Puppet Master server, where all of your configuration data will be managed and distributed from, and all your remaining servers will be Puppet Agent nodes, which can be configured by the puppet master server.

Prerequisites

To follow this tutorial, you must have root access to all of the servers that you want to configure Puppet with. If you do not have an existing infrastructure, feel free to recreate the example infrastructure (described below) by following the prerequisite DNS setup tutorial.

Before we get started with installing Puppet, ensure that you have the following prerequisites:

Private Network DNS: Forward and reverse DNS must be configured, and every server must have a unique hostname. If you do not have DNS configured, you must use your hosts file for name resolution. We will assume that you will use your private network for communication within your infrastructure.

Firewall Open Ports: The Puppet master must be reachable on port 8140.

Example Infrastructure

We will use the following infrastructure to demonstrate how to set up Puppet:

Hostname: host1

Role: Generic Debian

Private FQDN: host1.douglasqsantos.com.br

Private IP: 192.168.25.152

Hostname: host2

Role: Generic Debian

Private FQDN: host1.douglasqsantos.com.br

Private IP: 192.168.25.153

Hostname: host3

Role: Generic CentOS

Private FQDN: host3.douglasqsantos.com.br

Private IP: 192.168.25.154

The puppet agent will be installed on all of these hosts. These hosts will be referenced by their private network interfaces, which are mapped to the “.douglasqsantos.com.br” subdomain in DNS.

Once you have all of the prerequisites, let's move on to creating the Puppet master server!

Create Puppet Master Server

Create a new Debian Jessie, using “puppet” as its hostname. Add its private network to your DNS with the following details:

Hostname: puppet

Role: Generic Debian

Private FQDN: puppet.douglasqsantos.com.br

Essentially, you need to add an “A” and “PTR” record, and allow the new host to perform recursive queries. Also, ensure that you configure your search domain so your servers can use short hostnames to look up each other.

Using “puppet” as the Puppet master's hostname simplifies the agent setup slightly, because it is the default name that agents will use when attempting to connect to the master.

If you don't want to use the DNS server configuration you can configure the hosts file and add the entries as follow.

Install NTP

Because it acts as a certificate authority for agent nodes, the puppet master server must maintain accurate system time to avoid potential problems when it issues agent certificates–certificates can appear to be expired if there are time discrepancies. We will use Network Time Protocol (NTP) for this purpose.

Let's install the ntpdate packet

apt-get update && apt-get install ntpdate -y

First, do a one-time time synchronization using the ntpdate command:

ntpdate pool.ntp.org

Your system time will be updated, but you need to install the NTP daemon to automatically update the time to minimize time drift. Install it with the following apt command:

apt-get update && apt-get -y install ntp

It is common practice to update the NTP configuration to use “pools zones” that are geographically closer to your NTP server. In a web browser, go to the NTP Pool Project and look up a pool zone that is geographically close the datacenter that you are using. We will go with the Brazil pool (http://www.pool.ntp.org/zone/br) in our example, because the servers are located in a Curitiba datacenter.

Open ntp.conf for editing let's add the time servers from the NTP Pool Project page to the top of the file:

vim /etc/ntp.conf
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/lib/ntp/ntp.drift
# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
# You do need to talk to an NTP server or two (or three).
server 0.br.pool.ntp.org
server 1.br.pool.ntp.org
server 2.br.pool.ntp.org
server 3.br.pool.ntp.org
# pool.ntp.org maps to about 1000 low-stratum NTP servers. Your server will
# pick a different set every time it starts up. Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst
# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details. The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust
# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255
# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines. Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

Save and exit. Restart NTP to add the new time servers.

systemctl restart ntp

Now that our server is keeping accurate time, let's install the Puppet master software.

Install Puppet Master

There are a variety of ways to install open source Puppet. We will use the debian package called puppetmaster-passenger, which is provided by Puppet Labs. The puppetmaster-passenger package includes the Puppet master plus production-ready web server (Passenger with Apache), which eliminates a few configuration steps compared to using the basic puppetmaster package.

The Puppet master, Passenger, Apache, and other required packages are now installed. Because we are using Passenger with Apache, the Puppet master process is controlled by Apache, i.e. it runs when Apache is running.

Before continuing, stop the Puppet master by stopping the apache2 service:

systemctl stop apache2

Next, we want to lock the version of Puppet.

Lock the Version

Changes from version to version can occasionally cause your Puppet environment to stop working properly. For this reason, you will want to maintain a consistent Puppet version across your entire infrastructure. If you decide to upgrade to a newer version, make sure that you upgrade your master before any agent nodes, as the master cannot manage agents that have a higher version number.

Let's look up the version of our Puppet installation with the following command:

puppet help | tail -n 1

During the time of writing, the output from the previous command is Puppet v3.7.2. We can use apt's pin feature to lock our Puppet install to 3.7.*, which will prevent apt from upgrading Puppet to a higher major release.

Add the following lines to lock the puppet, puppet-common, and puppetmaster-passenger packages to the 3.7.* (change this to match your installed version):

Set Up Names and Certificates

Puppet uses SSL certificates to authenticate communication between master and agent nodes. The Puppet master acts as a certificate authority (CA), and must generate its own certificates which is used to sign agent certificate requests. We will setup the master's certificates now.

Delete Existing Certificates

Delete any existing SSL certificates that were created during the package install. The default location of Puppet's SSL certificates is /var/lib/puppet/ssl:

rm -rf /var/lib/puppet/ssl

Configure Certificate

When creating the puppet master's certificate, include every DNS name at which agent nodes can contact the master at. In the case of our example, we will use “puppet” and “puppet.douglasqsantos.com.br”, the short hostname and FQDN, respectively.

Edit the master's puppet.conf file. It will look something like the following:

It is important to use assign certname to “puppet” because the Apache/Passenger configuration expects the certificate to be named “puppet.douglasqsantos.com.br”. If you decide that you want a different certname setting, be sure to edit the Apache config file (/etc/apache2/sites-available/puppetmaster.conf) to change the name of the SSL certificate path.

Save and exit.

Generate New Certificate

Now create new CA certificates by running the following command:

puppet master --verbose --no-daemonize

You will see several lines of output saying that SSL keys and certificates are being created. Once you see Notice: Starting Puppet master version 3.7.2, the certificate setup is complete. Press CTRL-C to return to the shell.

Our Puppet master service is almost ready to be started. Let's take a look at the master configuration first.

Configure Puppet Master

The main puppet configuration file, puppet.conf, consists of three sections: [main], [master], and [agent]. As you may have guessed, the “main” section contains global configuration, the “master” section is specific to the puppet master, and the “agent” is used to configure the puppet agent. Aside from the changes we made earlier, the defaults will work fine for a basic setup.

The configuration file has many options which might be relevant to your own setup. A full description of the file is available at Puppet Labs: Main Config File (puppet.conf).

If you want to edit it, run this command:

vim /etc/puppet/puppet.conf

Let's take a look at the main manifest file.

Main Manifest File

Puppet uses a domain-specific language to describe system configurations, and these descriptions are saved to files called “manifests”, which have a .pp file extension. The default main manifest file is located at /etc/puppet/manifests/site.pp. We will cover some basic manifests later, but we will create a placeholder file for now:

touch /etc/puppet/manifests/site.pp

Start Puppet Master

We are now ready to start the Puppet master. Start it by running the apache2 service:

systemctl start apache2

Your Puppet master is running, but it isn't managing any agent nodes yet. Let's learn how to install and add Puppet agents!

Install Puppet Agent

The Puppet agent must be installed on any server that the Puppet master will manage. In most cases, this includes every server in your infrastructure. As mentioned in the introduction, the Puppet agent can run on all major Linux distributions, some UNIX platforms, and Windows. Because the installation varies on each OS slightly, we will only cover the installation Debian servers.

Instructions to install Puppet on other platforms are located in the Puppet Labs Docs – be sure to follow the “Install Puppet on Agent Nodes” section for your OS of choice.

Note: It is assumed that all of your Puppet nodes, including agent nodes, are configured to use your DNS. If you are creating a brand new server, make sure to add it to your DNS before installing the Puppet agent otherwise use the /etc/hosts file to set up the configuration.

Configure Agent

Before running the agent, we must make a few configuration changes.

Edit the agent's puppet.conf. It will look exactly like the Puppet master's initial configuration file.

Assuming that the Puppet master is reachable at “puppet”, the agent should be able to connect to the master. If the master is not available at “puppet”, you will need to add the Puppet master's FQDN. We recommend configuring this regardless (substitute the FQDN with your own):

The Puppet agent is ready to run. Do so by running the following command:

systemctl restart puppet

If everything is configured properly, you should not see any output. The first time you run the Puppet agent, it generates an SSL certificate and sends a signing request to the Puppet master. After the Puppet master signs the agent's certificate, it will be able to communicate with the agent node.

Note: If this is your first Puppet agent, it is recommended that you attempt to sign the certificate on the Puppet master before adding your other agents. Once you have verified that everything works properly, then you can go back and add the remaining agent nodes with confidence.

Lock the Version

As with the Puppet master, we will want to use the Versionlock feature to lock the version of the Puppet agent:

Let's install the Versionlock

yum install yum-versionlock -y

Now we need to lock the puppet version

yum versionlock puppet

Your Puppet version is now locked.

Configure Agent

Before running the agent, we must make a few configuration changes.

Edit the agent's puppet.conf. It will look exactly like the Puppet master's initial configuration file.

Assuming that the Puppet master is reachable at “puppet”, the agent should be able to connect to the master. If the master is not available at “puppet”, you will need to add the Puppet master's FQDN. We recommend configuring this regardless (substitute the FQDN with your own):

The Puppet agent is ready to run. Do so by running the following command:

systemctl restart puppet

If everything is configured properly, you should not see any output. The first time you run the Puppet agent, it generates an SSL certificate and sends a signing request to the Puppet master. After the Puppet master signs the agent's certificate, it will be able to communicate with the agent node.

Note: If this is your first Puppet agent, it is recommended that you attempt to sign the certificate on the Puppet master before adding your other agents. Once you have verified that everything works properly, then you can go back and add the remaining agent nodes with confidence.

Sign Request On Master

The first time Puppet runs on an agent node, it will send a certificate signing request to the Puppet master. Before the master will be able to communicate and control the agent node, it must sign that particular agent node's certificate. We will describe how to sign and check for signing requests.

List Current Certificate Requests

On the Puppet master, run the following command to list all unsigned certificate requests:

puppet cert list

If you just set up your first agent node, you will see one request. It will look something like the following, with the agent node's FQDN as the hostname:

Note that there is no + in front of it. This indicates that it has not been signed yet.

Sign A Request

To sign a certificate request, use the puppet cert sign command, with the hostname of the certificate you want to sign. For example, to sign host1.douglasqsantos.com.br, you would use the following command:

puppet cert sign host1.douglasqsantos.com.br

You will see the following output, which indicates that the certificate request has been signed:

The Puppet master can now communicate and control the node that the signed certificate belongs to.

If you want to sign all of the current requests, use the -all option, like so:

puppet cert sign --all

Revoke Certificates

You may want to remove a host from Puppet, or rebuild a host then add it back to Puppet. In this case, you will want to revoke the host's certificate from the Puppet master. To do this, you will want to use the clean action:

puppet cert clean hostname

The specified host's associated certificates will be removed from Puppet.

View All Signed Requests

If you want to view all of the requests, signed and unsigned, run the following command:

puppet cert list --all

You will see a list of all of the requests. Signed requests are preceded by a + and unsigned requests do not have the +.

Getting Started with Puppet

Now that your infrastructure is set up to be managed with Puppet, we will show you how to do a few basic tasks to get you started.

How Facts Are Gathered

Puppet gathers facts about each of its nodes with a tool called facter. Facter, by default, gathers information that is useful for system configuration (e.g. OS names, hostnames, IP addresses, SSH keys, and more). It is possible to add custom facts if you need other facts to perform you configurations.

The facts gathered can be useful in many situations. For example, you can create an web server configuration template and automatically fill in the appropriate IP addresses for a particular virtual host. Or you can determine that your server's OS is “Ubuntu”, so you should run the apache2 service instead of httpd. These are basic examples, but they should give you an idea of how facts can be used.

To see a list of facts that are automatically being gathered on your agent node, run the following command:

facter

How The Main Manifest Is Executed

The puppet agent periodically checks in with the puppet master (typically every 30 minutes). During this time, it will send facts about itself to the master, and pull a current catalog–a compiled list of resources and their desired states that are relevant to the agent, determined by the main manifest. The agent node will then attempt to make the appropriate changes to achieve its desired state. This cycle will continue as long as the Puppet master is running and communicating with the agent nodes.

Immediate Execution on a Particular Agent Node

It is also possible initiate the check for a particular agent node manually, by running the following command (on the agent node in question):

puppet agent --test

Running this will apply the main manifest to the agent running the test. You might see output like the following:

Now save and exit. The inline comments should explain the resource that we are defining. In plain English, this will make ensure that all agent nodes will have a file at /tmp/example-ip, with -rw-r--r-- permissions, and text that contains the node's public IP address.

You can either wait until the agent checks in with the master automatically, or you can run the puppet agent --test command (from one of your agent nodes). Then run the following command to print the file:

cat /tmp/example-ip

You should see output that looks like the following (with that node's IP address):

Here is my Private IP Address: 192.168.25.152.

Specify a Node

If you want to define a resource for specific nodes, define a node in the manifest.

Now Puppet will ensure that a file at /tmp/hosts will exist on host2 and host3. You may want to run the puppet agent --test command (from host2 or host3), if you do not want to wait for the scheduled Puppet agent pull.

Note that if you do not define a resource, Puppet will do its best not to touch it. So if you deleted these resources from the manifest, Puppet will not delete the files it created. If you want to have it delete the files, change ensure to absent.

These examples don't do anything useful, but they do prove that Puppet is working properly.

Manisfest to Add User

Let's install the sdtlib module on the puppet Master

puppet module install puppetlabs-stdlib

Now let's create a manifest file that will make sure that every puppet agent has an user called support

vim /etc/puppet/manifests/site.pp
# /etc/puppet/manifests/site.pp
# The host3 will only get the information about the new users
node 'host3' {
import "users.pp"
}
# The host2 will get the information about the new groups and users.
node 'host2' {
import "groups.pp"
import "users.pp"
}
# All the nodes that are not explicit defined will get the following configuration
node default {
import "users.pp"
}

Now after the nodes get the new information from the master they will import the configuration that are explicit for each one, this is only another approach that I thought is a good way to maintain the configuration clear.

Using a Module

Now let's use a module. Modules are useful for grouping tasks together. There are many modules available in the Puppet community, and you can even write your own.

On the Puppet master, install the puppetlabs-apache module from forgeapi:

puppet module install puppetlabs-apache

Warning: Do not use this module on an existing Apache setup. It will purge any Apache configurations that are not managed by Puppet.

Now edit site.pp Now add the following lines to install Apache on host2:

Save and exit. Now the next time Puppet updates host2, it will install the Apache package, and configure a virtual host called “dqs.local”, listening on port 80, and with a document root /var/www/html.

On host2, run the following command:

puppet agent --test

You should see a bunch of output indicating that Apache is being installed. Once it is complete, go to host2's private IP address. You should see the default Apache welcome page.

Congrats! You have used your first Puppet module!

Creating a Simple Module

Here I will show how to create a simple module to configure the NTP server for the agents.

Here we are using the ntp.conf.$operatingsystem that will get the operation system of the node and replace with the correct file, because we are running CentOS and Debian.

Let's create the files, starting with the Debian one.

vim /etc/puppet/modules/dqs-ntp/files/ntp.conf.Debian
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/lib/ntp/ntp.drift
# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
# You do need to talk to an NTP server or two (or three).
server 0.br.pool.ntp.org
server 1.br.pool.ntp.org
server 2.br.pool.ntp.org
server 3.br.pool.ntp.org
# pool.ntp.org maps to about 1000 low-stratum NTP servers. Your server will
# pick a different set every time it starts up. Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.debian.pool.ntp.org iburst
server 1.debian.pool.ntp.org iburst
server 2.debian.pool.ntp.org iburst
server 3.debian.pool.ntp.org iburst
# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details. The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust
# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255
# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines. Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

vim /etc/puppet/manifests/site.pp
# Importing the new module
import 'dqs-ntp'
# Defining the configurations that will be applied to host3
node 'host3' {
import "groups.pp"
import "users.pp"
# This include directive will call the class dqs-ntp-server and apply all the configuration inside
include dqs-ntp-server
}
# Defining the configurations that will be applied to host2
node 'host2' {
import "groups.pp"
import "users.pp"
# This include directive will call the class dqs-ntp-server and apply all the configuration inside
include dqs-ntp-server
}
node default {
import "users.pp"
}