Test Systems: The Soft Underbelly of System Security

Test systems are the guts of your overall system design. Test systems embody an incredible amount of the history of how your team’s code and development practices turn into the tools, applications, and services you provide to your users. Most importantly, these systems show how your systems, tools, and applications mature and refine, and every brilliant and embarrassing decision your team made to create your product can often be found there.

But the contents of test environments, whether it’s test frameworks, CSV files, or a rainbow of test data, can be very sensitive, and more importantly, your test systems could contain a decade or more of information related to the development of your systems, tools, and applications. In spite of the potential sensitivity of the data in these systems, the often ad hoc, or ShadowOPS, nature of their development, frequently means that test systems do not undergo the same security scrutiny as other parts of your development environment or production systems, and this means that the risk associated with them probably outstrips the regular investment that has been made to secure them. So from a security point of view, it’s time to right the balance — to toughen up the soft underbelly of your test systems so they don’t undermine your test, dev, and production environments.Test systems constitute all of the resources, often under-served by operations and security teams, that are used by software testers (QA, SDET, SRE, customer service, product managers, etc.) to test and hone the products they want to release. These can include local virtualized servers (as used with tools like Docker, VMWare, or VirtualBox), CI systems like Jenkins and the test resources connected to them, demo or staging systems for manual testing or acceptance, and test databases for integration and load testing. Non-unit testing resources are the most susceptible since real resources are likely used in preference to in-memory, mock resources.

Properly built test systems are situated inside de facto isolated systems so a system-under-test (SUT) can be controlled by the layer around it (e.g., a CI system like Jenkins or Travis CI, or language-specific test frameworks). Controlling this layer is important as is preventing access to SUTs and the systems and credentials. Building sandboxes is not quite enough, however: you need to build high walls around the sandboxes.

If you integrate tools to manage your test systems and share them with an operations team’s configuration management tools or deployment process, you can use the test systems to improve operations practices and tools while improving the repeatability and documentation of the development and the accountability of the test systems. In short, you can use the same authentication tools in test and production instead of providing anonymous access, which is an all-too-common practice.

If you improve the security of your test systems, you can also help your Development and QA teams by providing a safe, well-monitored environment in which you can test and deploy updated applications, test applications to recently released security patches, and improve the overall understanding of the way your projects provide access to the outside world. In other words, improved security improves the QA ability of your systems by providing another valuable perspective on the development and execution of your systems. Increasing the isolation of a system should always lead to increased security, knowledge, and testability of that system — goals that will help to unite the QA/SRE (Site Reliability Engineer), Operations, and Security teams.

Given the importance of security, then, it is critical that you follow defined policies and procedures to ensure that your test systems are as secure as possible and don’t expose existing vulnerabilities or create new ones. With that in mind, here are three main areas where you can improve the security of your test systems, along with specific recommendations for each.

1. Set Up a Perimeter

Keep test resources inside a VPN and consider isolating test VMs in a VPN that is separate from production or demo systems. Ideally, a VPN containing test-specific resources should have no access to or interaction with development or production systems or their credential systems.

Require authenticated access to all test environments. Do not allow any anonymous access. Anonymous or default access is only appropriate for well-developed local/local-virtualized testing against test data (i.e., non-real data).

Control the security groups of your test systems, and be as explicit as possible. All inbound and outbound traffic should be over known ports. Do not simply allow inbound and outbound access on all ranges; for test systems, be specific about what ports are open and what is communicating over them. This improves your ability to monitor and test the system being tested as well as the testing environment, thereby helping to achieve the goals of both the security and the testing teams.

Monitor all access to and communication with the test system’s networks, and set up sane alert rules in a platform such as Threat Stack to ensure that your test systems remain properly configured. Do not disable your security perimeter by ignoring logging access and networking information for test systems. This information enables you to learn about and define the KNOWN aspects of your systems, so do not turn them into unknowns. Be MORE specific about configuration in test systems, not less, especially when it comes to security.

2. Monitor Your Test Systems and Set Up Alerts

Alert on platform actions such as those provided by AWS CloudTrail. Examples of activities to monitor include:

Backing up instances

EBS volume copying and snapshotting

Backing up test or development databases

Increase security alerting for login and communication events into test systems and networks. Assume that there should be no regular direct human access to automated test infrastructures. While no human should regularly be scp’ing or FTP’ing anything to or from automated test systems, your test and QA teams should be considered responsible and trusted to access these systems, but they should still be monitored for any manual interaction.

Increase security event logging in test systems, but be careful not to increase pages for test systems.

3. Improve Your Tests; Improve Your Security

NEVER use default credentials. Default and root credentials are highly vulnerable targets, so disable them for any test system. Always run tests with known access credentials. This is especially true if you ever place production data on a test system to help debug major production issues. (See the next bullet.)

Never use production data if you can help it; instead use generated, production-like data. To speed up this process, use database migration tools if your development teams are already using them.

Shared/hosted test resources are more exposed than production systems if they are placed under identical scrutiny because of their common direct access and manual configuration nature. Save any unauthenticated testing for your local test development, not for shipped tests for your CI pipeline. These systems can often contain valuable information in the form of old, forgotten credentials, and state actors are known to regularly breach systems and monitor actions and files on test systems to improve their ability to move laterally to connected systems.

Use secure methods to transfer sensitive configuration information to your test environments, and prevent plain text secrets or passwords from being committed to source control or configuration management systems, where this information can easily be leaked. Use systems such as Travis CI’s encrypted environment variables or Jenkins’ credential store to dynamically inject variables into your build to reduce the possibility that secrets such as passwords are being checked into test code. This approach will also frequently improve the local development of tests and test infrastructure by making it easier to run tests in a variety of environments with diverse credentials, or even in local testing scenarios.

Encrypt or secure test database and system backups as you would any production backup.

Deprecate and remove any proof of concept test systems. There is a strong risk of credential leaking with any manually configured systems. If possible, move test systems into configuration management when graduating test systems out of the proof of concept, and ideally before a wide internal testing user base begins developing tests against a new test environment. Isolate ALL systems used to develop new test systems or patterns during development, and destroy them once more permanent testing resources are put in place (e.g., once tests have been integrated into a CI pipeline).

Always patch your test systems. These are just as vulnerable as any other system, and you should not develop or release code to unpatched servers. Testing software that needs to run in known unpatched environments should be tested on hardware-isolated networks — which is not an ideal use case for the cloud. (Note: If you don’t think you can patch your production systems, stop reading this immediately. Then you can start learning how to patch and upgrade ALL your systems, platforms, and tools, and how make it easier to patch, test, and deploy these changes to your production systems.) At the very least you should begin using your test systems to help stage security patches to your production systems. (Test systems are strangely perfect for things like testing!)

Do not test with plaintext credentials. Instead, use one-time or temporary credentials during test runtime if possible. The following services work extremely well to provide temporary credentials:

The Vault Project, which offers temporary credentials for a number of services, such as:

AWS

Databases: MySQL, PostgreSQL, MSSQL, MongoDB

EVEN SSH!

Final Words . . .

The lack of appropriate knowledge or lax security practices can cause your test systems to become points of weakness or vulnerability. Follow the recommendations outlined above to toughen the security of your test systems, to ensure that you don’t expose existing vulnerabilities or create new ones, and to pass on the benefits of your test systems to your Dev and Ops environments.

Apollo Catlin is a Senior Operations Engineer on Threat Stack’s Operations team. With 10 years of experience in quality assurance, automation, and build engineering, Apollo understands the challenges of building, deploying, and maintaining high-quality platforms and environments for large, distributed applications. His experience delivering software in a variety of industries (including financial, consumer electronics, ecommerce, health care) has given him a strong background for helping Threat Stack clients build stable and secure platforms for their applications.