Eleven Steps to Virtualize Your CI Test Environment

As more organizations take advantage of the flexibility and scalability of a cloud-based, OpenStack, Continuous Integration (CI) environment for development, they naturally look to extend those benefits to the rest of the product lifecycle. But the test lab poses significant challenges for this process, including a rapidly changing code base that must perform on multiple operating systems and cloud management platforms.

Spirent assessed the issues and gathered requirements to provide our developers with on-demand, multi-node Openstack test environments where they could, in a whiteboard-like fashion, configure different network functions and customize them across different distributions.

The main objectives were ease and speed of test lab setup, security, repeatability, and cost. The solution had to integrate with our existing product build process, which spanned various adhoc, DevStack-based OpenStack installations (i.e. Havana, Icehouse, Juno) with differing levels of support and documentation, and provide a seamless handoff to the product test environment, regardless of operating system, OpenStack environment, or cloud management distribution.

The result was a real-world, technical implementation of a highly-parallelized OpenStack CI network testing solution.

The Tools

The test rigs perform both functional and performance tests as well as probing for security weaknesses and vulnerabilities. For performance testing we use a cloud based test lab hosted at Nephoscale that has access to the latest Spirent traffic generation equipment. For functional and regression testing the nested virtualization capabilities provided by Ravello Systems are leveraged to run KVM and ESXi Virtual Machines on AWS.

Ravello offers all the functions of our physical test lab yet only uses virtual machines and networks deployed on a public cloud infrastructure. Using their blueprint technology and API SDK, we have empowered our developers to build their own multi-node OpenStack lab environments in the cloud with different distributions at the push of a button. Machines typically live for a few hours at a time, and we can scale out to hundreds of machines, depending on the combination of operating systems and networks to be tested.

We use Mist.io to provision OpenStack host machines through the NephoScale API to kick start the RDO deployment process and to provide real-time monitoring of the testing procedures. We configure custom metrics on the host machines and the guest VMs. These trigger alerts and automated responses when needed, such as to provision additional compute and network nodes. Mist.io also provides a unified dashboard for managing multiple OpenStack installations (Havana, IceHouse, Juno, Kilo).

Since some of the products under test include sensitive intellectual property, we implement strong security controls. The Openstack test labs running on Nephoscale and AWS through Ravello are completely fenced off from external access and from each other. This allows us to implement parallel test labs for our customers with minimal concern for data leakage.

Step 4: After successful unit testing, the developer commits the changes to the repository for source code version control, which triggers a series of jobs. Value: New content isn’t committed to the code base until it’s been tested, and any update can be easily backed out if it introduces quality issues. Automation reduces errors, accelerates the schedule. Tools: Git Hook, Jenkins

Step 5: A code quality enforcement tool reports on compliance with best practices for the development language to a dashboard for review and drill-down to violation details. Value: Code quality problems are caught early, maintaining the integrity of the code base. Tools: Sonar

Step 8: Test data is collected and analyzed into a data warehouse, including performance and security findings correlated with metrics on infrastructure utilization such as CPU, RAM, PCI Bus, network, storage, and power. Tools: Spirent Velocity

Step 9: Different OpenStack environments in Ravello are saved as blueprints for future reference. Value: The ability to quickly run instances of Openstack from existing blueprints and compare those with environments with patches applied.Tools: SaaS platform: Ravello Systems

Step 10: Defects are recorded and tracked for troubleshooting and resolution. Value: All issues are documented to reduce the possibility of a defect falling through the cracksTools: JIRA

Based on OpenStack, we built a workflow to continuously test our products without overbuilding the hardware infrastructure or spending time provisioning multi-node Openstack environments with different flavors for each developer. Each of our developers quickly spin up their own multi-node Openstack labs as needed, run them on AWS, execute automated tests including parallel tests on multiple Openstack setups, and shut them down when they are done. This new workflow process has improved quality and allows us to support multiple distributions used by our customers.