Testing the effectiveness of your security procedures

A while ago, I talked about procedures and how they are different from policies. Basically, a policy states what your security strategy is, while a procedure is a set of steps for implementing the various parts of that strategy. So, for example, your policy might be (in part) that all user passwords are 15-plus characters long, and one of the procedures for that might be adding that to your Windows Group Policy.

But how do you know that your procedures are working? Well, of course, one way is to conduct vulnerability assessments that test those procedures. In the case of my example above, most vulnerability scanners can examine both your Group Policy and the systems affected by it to ensure compliance with your security policy.

But what if your procedures deal with disaster preparedness or recovery? There isn't, as yet, any way to automate checking that those procedures are properly in place. Another way to test must be used.

In my organization, we do what we call “tabletop exercises." We develop a policy and a set of procedures, and then when we believe we have the procedures enumerated correctly, we sit down at a conference table with all the relevant parties -- the people who will be carrying out the procedures, and sometimes other interested parties -- and talk through the event and the procedures. It's assumed that the procedures are written consistently enough that every single applicable scenario does not have to be walked through each time the entire plan is tested, which should be at minimum once a year.

For example, we might start out by saying, “There's been an earthquake, and the building that houses this office is in ruins. How do we recover the business?” We run through the entire procedure almost like a script for a play, noting any places where there are snags or the procedure breaks down. For instance, Joe might be the person who initiates the phone calling tree, but we might not have built a procedure for what happens if Joe can't connect to Mary, the next person on the tree. We therefore annotate the procedure and add verbiage to cover that scenario. Ideally, if the procedures must be changed, they should be tested again immediately if at all possible (perhaps with a different group of people, if your organization is large and more than one group is affected by the same set of procedures).

As with any security training, it's not always easy to schedule time to do these tabletop exercises. If you can combine them with another sort of training, or make it into a team-building exercise, it may be easier to get the key employees on board. But as I said, these non-automated procedures must be tested at least once a year, because otherwise, they are only words on paper.

Mary Ursula Herrmann

Mary Ursula Herrmann is a Network Security Analyst living in Juneau, AK. She has worked in Information Security for over 15 years, and obtained her CISSP in 2005.