CASE STUDY

Speaking of Sandboxes

Posted by Roni Dover August 9, 2017

I came to the Jenkins Users Conference in Tel Aviv last month (JUC) expecting to do a lot of explaining around the concept of sandboxes, what they are and why they are needed. Surprisingly, I found that sandboxes were already a big part of the conversation at the conference, so I spent most of my time listening instead. To be fair, only a handful of people used the term 'sandboxes'. Some referred to them as environments, other as 'production replicas’. Similarly, I heard about many diverse, seemingly unconnected use cases, from applications testing to physical network topologies. These different use cases and terms all revolve around a similar pain that surfaced again and again in the conversation: How to create production like environments and make them easy to use for testing and dev without adding a ton of complexity. How to make use of these environments across the devops pipeline, from dev to test - from CI to CD.

So, what are sandboxes, and why do we need them?

I think of sandboxes as the missing abstraction for creating on-demand, environments based on your production which are scoped for a specific usage context. Basically, making both production and its subsets something easily accessible to everyone by making it reproducible via automation. The three basic layers of sandbox environments are the apps (in testing these are the apps under test), the required infrastructure and the orchestration code – the magic gluing everything together.

One of the reasons sandboxes are becoming so important is that tests are evolving. In the presentation I gave at this conference, I discussed how the term ‘continuous integration’ should be considered as broader than simply continuously running unit tests. Continuous integration, to me, means continually integrating the code with different aspects of production – load, infrastructure, different browsers and viewports, security restrictions as well as failures and server crashes. Unit testing as code validation are only a small part of the continuous integration spectrum.

As organizations ‘move right’ from continuous integration towards continuous deployment, the need for better and more varied integration tests now becomes critical. By accurately testing different aspects of production, much of the uncertainly and risk of the final deployment act is removed and the organization can transition to an automated release process. Unlike unit tests, which run in isolation on any MV, this varied spectrum of tests requires continually and accurately recreating aspects of production. These tests require sandboxes.

There seem to be many issues with making sandboxes work. Some of the people I've spoken with at JUC have already implemented custom solutions for setting up environment using various tools and products, but are struggling to release it for the rest of the team in a way that would be standardized, controlled and easy to use. There is no shortage of tools which can help hand string production environments together, making the life of an admin or devops engineer much easier. However, packaging this orchestration, and defining the environment specifications and parameters in a way that an IT or devops engineers would be comfortable letting anyone in the organization use, is something else entirely.

This conflict is at the heart of devops these days. On the one hand, maintaining control and ensuring standardization, on the other hand realizing that the role of the devops team is to enable others - not to do the work for them. Devops teams cannot afford to become a bottleneck by being personally responsible for each environment. Devops after all does not mean every developer is now responsible for ops. Having each team member create their own environment using whatever means they see fit would result in hard to support non-standard environments, not to mention in a mess of orphaned networks and forgotten VMs.

A maintainable devops process requires, therefore, to turn the pyramid on its head. To provide sandboxes as ‘self-service’ to users - but maintain restrictions and policies on their consumption.

The complexity of production environment and the problems in recreating it for testing was a popular topic at JUC. Even organizations who have succeeded in creating a stable continuous deployment process find it hard apply that process to dev/test environments. Most of the time, the way production is modeled is a big impediment. Different dev/test activities require different levels of similarity to production, in terms of scale, configuration, data and external services production environment may depend on. It’s difficult to create an environment modeling that is both single sourced and based on the push to production assets and at the same time flexible and easily parameterized.

The user should be able to selectively scale up/down certain elements of the environment or even mock or replace some of them altogether. Some integrated cloud solutions make this even more frustrating by closely coupling infrastructure with the application. This is makes it very difficult, for example, to shift between running an app on multiples instances and running many apps on once instance without massive changes. Simple yet flexible modeling is a make or break feature for any sandboxing platform.

My takeaway from JUC was that sandboxes are ever present and increasingly an important part of the conversation in the devops world. They are also a big pain in many organizations with the strain falling directly on the devops team. This may seem a bit disparaging, yet I think it’s a good sign and an indication of the growing maturity of devops and testing processes.

To be effective, sandboxes need to be available ‘as-a-service’ to all users, modeling is key as is implementing way for the devops team to enforce policies for consumption and standardization. At Quali, we are excited to be working on improving our own sandboxing platform based on these principles. Judging by the conversations I’ve had at JUC TLV, cloud sandboxes are going to get even more interesting in the coming year.