Program Auditing Standards

In the IT certification industry, program maintenance is just as important as program development. An organization might develop a successful certification program, but if that program isn’t continually reviewed and updated, it could become obsolete faster than you can say Betamax.

At the National Organization for Competency Assurance (NOCA), certification programs that are accredited by the National Commission for Certifying Agencies (NCCA), which is part of NOCA, aren’t required to review or update their programs at any particular frequency. However, according to Wade Delk, executive director of NOCA, these programs are required to become reaccredited every five years, and when applying for reaccredidation, aspects such as program auditing are considered, he said.

For example, organizations must make sure their programs continually meet the demands of certification holders in the field. This requires reviewing the job needs and skills on a continual basis. “The way to set up a quality cert program, you need to be continually looking at the job at which they are providing a certification to,” Delk said. “It requires being continually updated and reviewed. You want to put together an updated job analysis every five or six years. You need to make sure that when you go back and audit that you look at the field. What’s going on in the industry? What changes are happening? What are the new skill sets in the field?”

Once necessary skills are determined, program managers must make sure their exams test for those skills as well. “In essence you would want to make sure that in the test itself those test questions are performing correctly,” Delk said. “You need to be looking at the test and potentially the different versions of the test and do this over periods of time. If you have different versions of the test make sure you equate them so those scores match.”

At Information Systems Audit and Control Association (ISACA), which provides the Certified Information Systems Auditor (CISA) and Certified Information Security Manager (CISM) certification programs, the CISA is completely reviewed and updated every five years, said Richard Brisebois, chair of ISACA’s CISA Certification Board and principle for IT audit services at the office of the auditor general of Canada. The process takes a full year, he said.

“We do a complete review at least every five years. Our latest review is going to be effective June 2006,” Brisebois said. “The five-year review is fairly comprehensive. It’s a worldwide project. Before we start we do an extensive survey to find out what are the tasks that a CISA does today. We make sure the program reflects what a CISA actually does. What are the tasks a CISA is asked to do? What is the knowledge that is needed to perform those tasks? How much time do you spend on each of those tasks?”

Along with the program’s extensive survey, CISA also holds several focus groups worldwide, Brisebois said. “The task force takes the results of surveys and the focus groups and revisits the delineation. Afterward, we do another worldwide survey to confirm the delineation.”

Delk said the frequency a program should be fully audited and updated depends on the size and scope of the certification. However, individual aspects of a program might need to be reviewed more frequently. “You want to make sure the policies you have are updated. How you deal with your eligibility criteria, you want to make sure that’s current to,” Delk said. “There is a tremendous amount a certification program must do continually. If you’re not advancing the practice, then you will sort of be stuck at that point in time.”

Delk also said the certification exam should be updated more often than other aspects. “Certainly for reviewing your own test, depending on how many candidates you have, at least every year have the exam committee review the test,” he said. The sample questions also need to be reviewed annually. “You always have to build more and more questions. If you have an item bank of 600 questions, eventually those are going to wear out.”

At ISACA, the CISA review manual is updated annually, Brisebois said. “Every year we have an updated CISA review manual. We add new questions to the bank, especially in emerging technologies,” Brisebois said. “If you don’t change anything in five years, your program wouldn’t be up to date.”