Certifying Experience: The New Frontier

With mounting concerns about the economy, finding and retaining qualified IT talent is more critical now than it’s been in nearly a decade. And with a shortage of qualified senior-level IT executives, hiring managers continue to rely on certification as a gauge of competency.

Until recently, however, virtually all IT certifications have been knowledge-based, meaning they certify an individual’s recollection of a body of knowledge by requiring a passing grade on a test. Unfortunately, these certifications rarely confirm an individual’s ability to use this knowledge successfully in practice.

A new class of certification programs for IT professionals addresses this limitation by certifying individuals based on their demonstrated competence as successful practitioners in actual engagements.

The Shift in the Nature of IT Competence

From the earliest days of IT, there has been an inexorable trend to reduce cost and increase flexibility. It manifests itself in three ways: consolidation, standardization and commoditization.

More importantly, both ultimately lead to commoditization, as vendors reap the economic benefits of larger markets and users conclude that the benefits of buying faster, cheaper and more reliable “black boxes” off the shelf outweigh whatever benefits they might obtain by customizing their own components.

This trend has several implications for the skills, knowledge and experience IT practitioners need to be successful. In the lower layers of the IT department, the most valuable skills will be those that address the need for cross-enterprise integration with multiple solutions. But the most valuable IT staff will be those who are most comfortable working closely with the business side.

What does this mean for how we define competence as an IT practitioner in today’s business environment, and how can an organization be sure someone has that competence?

Practice, continuing professional development and certification are the cornerstones of acquiring, maintaining and confirming competence. In particular, new certification methods are necessary to effectively demonstrate these new kinds of competencies.

Knowledge, Skills and Experience

Competence can be described as having knowledge, skills and experience. Strictly speaking, knowledge is something you know, a skill is the ability to do something, and experience is what you get by applying knowledge and skills in real-world situations.

Ultimately, experience provides a practical context for your skills and knowledge, turning theory into practice. One can be knowledgeable without having skills or experience, or have a great deal of experience without actually acquiring skills or knowledge.

Nowadays, general skills tend to remain useful much longer than knowledge of specific technologies. At some point, though, a skill can become obsolete.

Let’s consider some simple examples. Riding a bicycle is a skill. Riding a bicycle through a particular kind of terrain might be a refinement of that skill. Having knowledge about the mechanics of bicycle design and the dynamics of bicycle riding does not, by itself, make one a skilled bicycle rider, though it can be helpful in improving that skill.

Now, one might have knowledge about particular brands or models of bicycles and their suitability to different terrains. One might have knowledge about the kinds of terrain one would encounter when riding from one place to another. However, the experience of riding, perhaps repeatedly, from one place to another will refine and augment both one’s skills and knowledge about riding in general and riding this route in particular.

Confirming Competence

Any process for confirming competence must have several properties to be useful. It has to be practical and scalable, objective and consistently repeatable, and resistant to fraud.

Possession of knowledge is demonstrated by correctly answering questions. But knowledge by itself is probably the least predictive measure of future competence since it doesn’t mean you can apply it effectively in diverse and novel situations. Further, while question-based tests are practical and objective, they also are the most susceptible to fraud.

Demonstrable skills are probably a better predictor of future competence, but again, demonstrating individual skills in isolation does not imply that one can choose and integrate them effectively in diverse and novel situations. Furthermore, while tests of specific skills can be made objective and generally are difficult to cheat on, they present a major development and administrative challenge.

Fortunately, the possession of and ability to effectively apply both knowledge and skills can be confirmed by what many of us believe is the best predictor of future competence: demonstrated past competence.

It was based on this analysis that The Open Group chose to develop experience-based certification programs for IT architects and IT specialists.

About The Open Group and Certification

The Open Group is a consortium of IT vendors and users, formed in 1996 by the merger of X/Open and the Open Software Foundation (OSF). Multiple forums allow members to contribute to open standards in a variety of technology domains. One of the most active forums is The Open Group Architecture Forum (TOGAF), with more than 180 members from all over the world, representing a wide variety of industry sectors. In 1994, the membership decided a standard enterprise architecture framework was needed. This led to the development TOGAF and an affiliated certification program.

There are two ways an architect can become TOGAF certified: by taking TOGAF certified training or by passing a TOGAF certified examination. The training and examination tests must address a thorough and complete knowledge of the elements of TOGAF.

As TOGAF went through several successive revisions, members of the Architecture Forum asked, “How do you tell if someone is really an architect, in practice, not just in theory?” and considered the problem of experience-based IT architect certification independent of TOGAF. Several of the forum’s members operated architecture profession programs, and certification often was part of the professional development and career paths of participating members.

These programs had comparable criteria and processes, but differed in many details and were essentially proprietary. The Architecture Forum recognized the value of industry-wide, vendor-independent standard certification criteria and asked The Open Group to initiate a project to define such a standard.

In early 2004, IBM and HP began collaborating on a detailed proposal to The Open Group. The proposal was approved in October 2004, and a working group comprising volunteers from Capgemini, CLARS, EDS, HP and IBM developed IT architect certification (ITAC) requirements and policies during the next year. These were approved by The Open Group membership, and the program went public in July 2005. Following on the success of the ITAC program, the membership proposed and The Open Group implemented an analogous experience-based certification program for IT Specialists (ITSC).

A Competency Model for IT

In the development of ITAC, the first step involved the sharing of IT architect competency models across the ITAC group. While there were some minor differences in terminology, the skill models contributed were remarkably consistent. A similar process was later followed for the ensuing IT Specialists Certification (ITSC) program.

Good IT architects and specialists will have mastered skills specific to their disciplines, but truly successful professionals also will have skills borrowed from other disciplines — skills that allow them to work productively at a particular employer and in a certain client context, as well as within the context of a particular nation, region, enterprise and business unit.

The three most relevant areas with which IT architects and specialists share skills are project and program management, business and consulting.

Skills borrowed from project and program management include planning, sizing and estimation, risk management, leadership and team building. Business skills include basic competence in areas such as management, finance, legal and regulatory concerns, organizational structures and dynamics, governance and portfolio management. Finally, consulting skills include oral and written communication, conflict resolution, various kinds of assessments, political savvy and negotiating.

The diversity of knowledge and skills expected of successful IT architects and specialists confirmed that a certification based on experience rather than on individual skills was the correct strategy. The certification process seeks to validate successful application of the combined knowledge and skills necessary to achieve business results.

Implementing Certification

Board review of demonstrated skills and experience by certified peers was chosen as the evaluation method for this certification program. Because of the decision to use board review rather than a test, particular attention was paid to creating a demonstrably objective process. This was especially challenging because of the additional requirement of process scale to a high number of candidates.

Because many member companies already had large architectural practices and internal certification programs, an obvious strategy was to leverage these existing programs. This led to the idea of “indirect” certification by an Accredited Certification Program (ACP), by which a company could certify its own architects and specialist using an internal process that had been accredited to conform to The Open Group standard and that was periodically audited by The Open Group for continued conformance and quality control.

In addition, The Open Group would directly certify IT architects and specialists whose employers, for whatever reason, chose not to set up an ACP.

Getting Certified

Candidates for certification prepare a submission package consisting of a document of no more than 50 pages, based on a template provided by The Open Group and letters of reference. If the package is judged complete and the references are confirmed, it is passed on to a three-member review board, and a board interview with the candidate is scheduled.

The board members are themselves certified architects. The review board examines the package in detail to confirm that the evidence the candidate provided adequately demonstrates the skills and experience specified in the certification conformance requirements. The candidate then interviews with each of the three board members for one hour.

While the goal is for a board to reach a unanimous agreement to approve or reject a candidate, a 2:3 vote is required. Each board member’s conclusion is captured and preserved by an online candidate assessment tool. When a board member judges that a candidate does not satisfy some certification requirement, that board member must provide a specific explanation as to how the evidence fails to demonstrate the skill or experience required. This feedback is provided to the candidate. Candidates approved for certification also are provided with career development suggestions from board members.

Both the ITAC and ITSC have met all their goals and continue to grow rapidly in adoption. More information on both programs can be found at The Open Group Web site at http://www.opengroup.org.

Leonard Fehskens is vice president of skills and capabilities at The Open Group. He is responsible for all activities relating to enterprise architecture. He can be reached at editor (at) certmag (dot) com.