Testing Automated Configuration Management Tools

I have written many times about the need to test a server configuration after making a change with a CM tool. What I have started to see is that this testing is as much about security and auditing as it is about validating the change. Here is an example to illustrate .

Policy Testing Example

Let’s say you want to ensure that the xinetd is not set to autostart on your Linux servers. This is your policy. First you need to run the command to make that change to put your servers in compliance with your policy. Then you need to test that the change has been made successfully. You also want to verify that the server remains in policy. These steps can be done with different tools.

Both the serverspec test and the Nessus scan are external tests that only run when we manually launch them or run them on a schedule. We could instead use an IDS agent running on the server that automatically and repeatedly tests itself. For example, in OSSEC (an IDS) we could use a rule like this:

If OSSEC detects a violation of our no xinetd policy it will notify us.

Conclusion

I find it interesting that three very different tools, severspec, OSSEC, and Nessus can be used to test the change made by the CM application. As I wrote about in my post on idempotency, I would like a CM application that allowed my to specify the state of my servers (my policy) and which would then make the changes needed to achieve that state, and continuously monitor them to make sure that they remain in that state.

In the mean time, there are some interesting options for infrastructure testing worth exploring. My point here is to show that there are many tools that can accomplish this, some of which you may not think of as infrastructure or integration testing tools.

2 replies

In my view, this would not fill the role of testing Ansible initiated changes. I favor independent validation of changes and a test-first approach. For example, a process more like this:

Identify business need for a change.

Have change approved by the CAB.

Write test for the change using something like ServerSpec

Document the change in your CMDB or other tool (maybe JIRA?)

Write Ansible code to affect the change

Test the change in the staging environment

Push to production once QA has signed off

Obviously all of the artifacts are tracked in version control. Ansible is code and I wouldn’t allow any code going into production that hasn’t been tested. Companies may have different Change Management and Release Management pipelines but I think this captures they key points.