Azure Automation: Leveraging Runbooks

I've had the opportunity to build a platform to ingest 3rd party data from multiple sources from scratch. The platform is designed and built entirely on Azure Cloud using a number of resources available within Azure. As is the case with many Agile projects, my team designed the platform to consists of three identical environments: DEVELOPMENT (DEV), UAT, and PRODUCTION (PROD).

The replication of environments along with the need to maintain strict configurations for all components across the platform encouraged us to automate as much of the platform deployment as possible. This would reduce the complexity and amount of time taken to deploy a new environment, as well as track the configuration changes to the platform.

Runbooks

Runbooks are script-like files that are based on Windows PowerShell. Runbooks are used to deploy, provision, and manage the various resources available in Azure (e.g. create resource groups, provision HDInsight cluster, or provision a VM). They are created and managed within an Azure Automation account (Creating a Runbook). Runbooks can be run manually or automatically by attaching a schedule.

There are two main components of Azure Automation runbooks that we found to be useful:

Deploying child runbooks from a master runbook

Versioning the runbooks with Git

Structure the Runbooks for Multiple Environments

As mentioned previously, the platform consisted of three identical environments: DEV, TEST, and PROD. Therefore, we adopted some best practices.

Define the order in which the runbooks should be executed
a. The admin_rg runbook is intended for resources that will exist 24/7, such as resource groups or virtual networks (VNet).

Calling Child Runbooks

In order to reduce the complexity, time, and potential human error, the entire platform can be deployed from a single runbook. As shown in the animation below, the master runbook is used to deploy both the create_rg and storage runbooks. However, notice that the master runbook will wait until create_rg has finished before calling the storage runbook.

When writing your "master" runbook you must first define your parameters. In this case I defined a set of parameters for each child runbook that I was calling:

The storage runbook depends on a successful completion of the create_rg runbook before it can be started. The -wait option on the Start-AzureRmAutomationRunbook cmdlet will force the master runbook to wait for the child runbook to finish before moving on.

I won't go into detail on how to version your runbooks in VSTS since Microsoft already does a pretty good job on this. However, I would like to highlight a few aspects regarding this solution. Firstly, the development workflow is a bit obscure (I wasn't able to use Powershell ISE). The workflow I find works best is:

Make your changes/updates to the runbooks in the Azure portal
a. This allows you to test the runbooks as you develop.
b. Be prepared for a slow feedback loop.

Identify good incremental stopping points, and SAVE your runbook.

Copy the entire runbook content. I recommend against manually adding your changes to the repository file, as you're more likely to introduce human error.

Ippon Technologies is an international consulting firm that specializes in Agile Development, Big Data and
DevOps / Cloud.
Our 300+ highly skilled consultants are located in the US, France and Australia. Ippon technologies has a $32
million
revenue and a 20% annual growth rate.

Newsletter

For our news update, subscribe to our newsletter! You can unsubscribe anytime.

Recent Posts

We are IPPON

Proud of our passion for technology and expertise in information systems, we partner with our clients to deliver innovative solutions for their strategic projects.
Through communication, curiosity and team spirit, we offer the best solutions to our customers.