What are good security metrics?

By Jason Miller

Nov 03, 2006

WILLIAMSBURG, Va. ' 'No government agency can say with confidence that the Chinese are not inside all their computers.'

That one comment from Alan Paller, the research director of the SANS Institute of Bethesda, Md., sparked the discussion of how well government agencies secure their IT systems and measure the effectiveness of their cybersecurity controls.

'The Chinese doctrine calls for a cyberattack as part of the next war with the U.S.,' Paller said earlier this week at the Executive Leadership Conference sponsored by the American Council for Technology and the Industry Advisory Council. 'Every major nation has a substantial cyberespionage initiative. It does matter that they and terrorist organizations are doing this because we are not doing very well in stopping them.'

Paller said that Congress and the administration pay too much attention to how agencies meet certain aspects of the Federal Information Security Management Act. He said the number of systems certified and accredited, awareness training, configuration management and annual testing don't go far enough to ensure agency IT systems are secure.

'Agencies spend about $1 billion on reports and most are never used or updated,' Paller said. 'There are self-test reports that are not updated regularly. There is no measure of competency for training, just did the employee take the course, and many agencies say they have a configuration management policy in place, but it doesn't mean anything.'

Paller added that agencies should do continuous monitoring and update their certification and accreditation process regularly.

'I would argue that the C&A process would not survive a week because of changes to hardware and software,' Paller said. 'Agencies need to find the ways people cause problems and fix those patterns in their systems that enabled the attack.'

Ron Ross, the National Institute of Standards and Technology's senior computer scientist, said agencies should instead figure out what the most important systems are and harden them first.

'The problem is agencies treat all systems equally,' he said. 'The higher-risk systems must be protected against the lower-risk systems. It could be an architecture issue.'

To improve agency security, NIST will release its second version of Special Publication 800-53, 'Recommended Security Controls for Federal Information Systems,' in December, which will give agencies 17 minimum requirements for all federal systems to meet.

These requirements will 'stop 95 percent of the attacks,' Ross said. 'The other 5 percent are so nasty they cannot be stopped. But if you are stopping 95 percent of the attacks, then you can focus on the other 5 percent and stop chasing your tail.'

Ross added that NIST's standards and guides are not a panacea, but provide a foundation for securing systems. Policy decisions and political wills also play a key role in the strength of agency cybersecurity.

'The idea is to get as many effective controls versus vulnerabilities as possible within your resources,' Ross said. 'Then you cut the risk to your mission.'