The contents of my brain, listed for all to see.

Automated Network Appliance Configuration Backups with Expect

An often overlooked aspect of managing network appliances like switches and routers is the necessity of configuration backups. Changes to these devices are usually infrequent, making backups easy to forget after finalizing a configuration. What’s more, network appliances often lack built-in tools for automatically copying configuration to a secure backup location. The problem becomes even more cumbersome when you consider that changes on one appliance often means changing several others. For example, if my server needs to change VLANs, a configuration change on the switch is required at minimum, but will likely be accompanied by a change to firewall rules on a router and policy changes on a security appliance. A simple VLAN reassignment is deceptively complex.

I’ve always believed backups should be automated; you simply can’t trust a user (or an administrator, for that matter) to always remember to perform them. With that in mind, I’ve implemented an automatic backup solution that overcomes these challenges by leveraging the one thing these devices have in common: a command-line interface.

Most network devices, most notably but not exclusively Cisco devices, store their configuration as a simple list of commands; if you copy the commands, you’ve created a backup. Manually copying and pasting “show run” output, while effective, is not ideal due to the forgetfulness factor. What’s the alternative then? Automatically copying and pasting output. Enter the ‘expect’ command.

I’ve had a lot of success using ‘expect’ to create backups. The ‘expect’ tool is described as “programmed dialogue with interactive programs.” In other words, we can use expect with a CLI to automate tasks that were designed to be performed interactively by a user. By leaning on ‘expect’, we gain several advantages:

No need to maintain a TFTP server, the traditional backup method for Cisco devices.

Working with Expect

After ‘expect’ is installed, remote into your network appliance and perform a “show run” command. Take note of all the key presses along the way, including user authentication input and returned output; these are the actions you’ll have to teach ‘expect’ to perform.

Next, create a wrapper script that will be execute an ‘expect’ script among other things. Utilizing a wrapper increases re-usability and helps protect login credentials.

As you can see, we’re simply passing arguments to the ‘expect’ command and redirecting the output to a file. The first argument is the path of the expect script which describes how to interact with the device; you’ll likely need one expect script for each device type. The remaining arguments are values that will be passed to ‘cisco3560.exp’.

This particular script is about as simple as it gets but it clearly demonstrates the basic concepts. After setting some variables and spawning an SSH session, there are only two things we need to do: ‘send’ input and ‘expect’ output.

We use the ‘send’ function for inputting keystrokes including carriage returns (\r). We must be explicit; forget a carriage return and expect will hang.

We use the ‘expect’ function to tell the script what output it’s supposed to receive so it can move on. If unexpected output is returned, ‘expect’ will hang until the timeout value is reached. We can describe expected text literally or through regular expressions. We can also define procedures based on the output. Look at this notated example from the ‘expect’ man page:

Defining procedures in this manner still only skims the surface of what ‘expect’ can do. With mastery, we can create very flexible and useful scripts that can assist us in performing tasks or completely automate interactive processes. Have a look at this article for a much more comprehensive look at ‘expect’.

Making Readily-Restorable Backups

Let’s get back to our backup. Remember that our wrapper script is capturing the raw output from the ‘expect’ command including all of the prompts and key strokes. If our goal is to create a readily-restorable backup, we need to edit the resulting file and get rid of text we don’t need. Fortunately, manipulating text is trivial. Let’s add to our wrapper script:

The amount of text manipulation you need to do perform varies from device to device. A Cisco 3560 switch produces just a few unnecessary lines, and always in the same place, so I’ve chosen to simply delete those lines. Other devices might require more complex text maniuplation. I recommend always saving a copy of the raw output along with the formatted file. This ensures that, if the output changes slightly and we delete something important, we can reference the raw file that still has everything we need.

All that’s left is to incorporate this script into our backup routine. In production, my wrapper script contains a loop that will connect to every 3560 we have on the network and create separate files for each. I also have scripts to handle a pair of Cisco SGE2010s, a half-dozen Cisco 2960s, and an old TrendNET gigabit switch. If there’s a CLI interface for it, ‘expect’ can work with it!