On-Demand Test Environments With ANSIBLE and Shippable

- By Ambarish Chitnis on December 11, 2017

One of the biggest challenges to implementing an end-to-end Continuous Delivery pipeline is making sure adequate test automation is in place. However, even if you have automated across your entire Test Suite, there is a second challenge: How do you manage Test infrastructure and environments without breaking the bank?

If you want to move towards Continuous Delivery, you need to execute a majority of your tests for each code change in a pristine environment that is as close to your production environment as possible. This ensures that code defects are identified immediately and every code change is therefore 'shippable'. However, creating these environments and updating them with each new application version or every time the config changes adds a lot of overhead. If you're testing an application with many tiers or microservices, the complexity increases since each tier might need to be tested indepedently in its own environment against specific versions of other tiers.

The utopia of Test Automation is the following:

Environment definitions are represented by infrastructure-as-code tools like Ansible, Terraform, Puppet, or Chef. The provisioning scripts are committed to source control and versioned, so you can go back to an earlier state if needed.

All (or at least a good majority) of your tests are automated and either committed to source control or hosted on services such as Nouvola, Sauce, etc.

You have a completely automated deployment pipeline that automatically spins up a production-like Test environment for every code change, triggers your automation, and if all tests succeed, destroys the environment. If tests fail, the right folks are notified and the environment is kept live until someone can debug the failures.

The first step is already happening in most organizations. The DevOps movement encouraged Ops teams to start writing scripts to provision and manage environments and infrastructure, and multiple vendors support this effort quite effectively. The second step is still a challenge in many organizations, but this is really something that needs executive buy-in and a commitment to automation, even if it slows down product development for a while.

This whitepaper presents a method of implementing the third step - spinning up test environments on-demand and destroying them automatically after the automated test suite is executed.

The Scenario

To make things simpler, we'll skip the CI step which builds and tests the application Docker image and pushes it to Amazon ECR. This can be accomplished by following instructions for CI: Run CI for a sample app

Our example follows the steps below:

1. A service definition, aka manifest, is created, including the Docker image and some options

2. A test environment is provisioned using Ansible under the cover. Ansible config files are templatized using environment variables defined in

3. The manifest is deployed to the Test environment and functional test suite is triggered

4. If tests pass, the test environment is destroyed using Ansible and test owner is notified.

5. If tests fail, the test owner is notified and the environment is not destroyed. The Test owner can always destroy the environment manually after he/she has extracted the information they need about the failure.

Before we start

Ideally, some familiarity with Ansible is desirable, though not required.

If you're not familiar with Shippable, here are some basic concepts you should know before you start:

Configuration: The Assembly Lines configuration for Shippable resides in a shippable.yml file. The repository that contains this config in your source control is called a Sync Repository, aka syncRepo. A syncRepo is added through your Shippable UI to add your Assembly Line.

Jobsare executable units of your pipeline and can perform any activity such as CI, provisioning an environment, deploying your application, or running pretty much any custom script. A simple way to think of it is, if something can execute in the shell of your laptop, it can execute as a Job.

Resourcestypically contain information needed for Jobs to execute, such as credentials, pointer to a cluster on a Container Engine or an image on a Hub, or any key-value pairs. Resources are also used to store information produced by a job which can be then accesses by downstream jobs.

Integrations are used to configure connections to third-party services, such as AWS, Docker Hub, GKE, Artifactory, etc.

How the sample application is structured

Our sample repositories are in GitHub:

The sample application that we will run functional tests on is a voting app that is built using Python Flask and Redis. The source for the front end (Flask) can be found in the vote_fe repository and the backend (redis) in the vote_be repository. The shippable.yml in these repositories contains the CI configuration to build and deploy their Docker images to their public repositories on Docker Hub.

At the end of Step 1, you should have two images published in your Docker registry integration.

Step 2: Create the service definition

A. Define the resource in shippable.yml file.

shippable.yml file can be committed in one of the app repositories or to a separate repository. We have used a different repository devops-recipes/on_demand_test_environmentsin our sample. The repository containing your jobs and resources ymls is called aSync repository and represents your workflow configuration.

After the cluster is created, we use Shippable platform resources and API to persist important cluster metdata such as the ARN and public IP of the cluster in a params resource test_info_odte and the cluster resource test_env_ecs_odte.

The ansible-ecs-provision playbook calls two roles to provision the ECS cluster.

Step 4: Deploy the application to the test ECS environment

deploy_app_test_odte is a deploy job which creates the service and task definition in the ECS cluster and starts the service. Since it needs to run after the ECS cluster is created in the workflow, prov_test_vpc_odteis specified as an input.

Step 5: Run functional tests on the test cluster

Add thedeploy_app_test_odte job to your shippable.yml file. This job extracts the public DNS of the ECS cluster from the test_info_odte params resource and passes it to the script that runs some tests using the public DNS.

Itis arunShjob that lets you run any shell script. Since it needs to run after the application is deployed in the workflow, test_env_ecs_odteis specified as an input. In addition, we also provide the manifest job as an input to the job.