Credentialing organizations are expected to provide evidence of
“reliability and validity” for any tests or assessment used in the
credentialing process. Validity is considered of paramount
importance—but what is it?

The concept of validity has been a topic of hot debate among
psychometricians in terms of how it should be defined,
conceptualized, communicated, and used. These issues bring a
sparkle to the eyes of many psychometricians, but the eyes of others
tend to glaze over as the discussion ensues. Fortunately, one does
not need a psychometrician’s level of expertise in order to embrace
the basic concept of validity. Validity has to do with whether a
test serves its intended purpose effectively.

If
you want to test basic math computation skills, obviously you would
not use a reading comprehension test. The reading test would not be
a valid measure of computation skills. But would you use a math
test that included story problems or real-life situations? It
depends. Do you just want to know whether your examinees can add
two numbers? Or do you want to know whether they can determine, in
a real-life situation, what mathematical operation is needed, select
the two numbers, and add them correctly? A test that included story
problems would probably be less valid as a test of basic computation
than as a test of applied computation skills. Conversely, a test of
straightforward computation problems would be less valid for
measuring applied skills. Validity depends on the purpose of
testing as well as the characteristics of the test itself.

The first step in ensuring a valid credentialing exam, then, is to
clearly define the purpose of the exam. In both licensure and
certification, the purpose of an exam is typically defined as
protecting the public by ensuring that the credential is awarded
only to those who have demonstrated that they have attained a
certain level of knowledge and/or skill. The particular sets of
knowledge, skills, and/or abilities to be tested are specific to the
field or profession and are generally referred to as KSAs.

The next step is to design an exam that will serve the intended
purpose by measuring the right set of KSAs. In licensure and
certification, this is typically done by conducting a job analysis
or practice analysis. Experts in the field define the job across a
variety of settings in terms of the tasks performed and the KSAs
needed to perform those tasks. Ideally, input is sought from
practitioners at varying levels of experience. The outcome of the
analysis is a test blueprint: a list of KSAs to be tested and the
approximate proportion of the test that should address each KSA. A
test constructed according to such a blueprint will measure the KSAs
that are important for effective functioning in the field.

Designing a test on the basis of a solid job analysis or practice
analysis helps to ensure that we are testing the right set of KSAs.
Validity is further supported by constructing an exam according to
sound measurement principles, so that we test the right set of KSAs
effectively. This includes implementing good item writing
practices, evaluating the performance of items statistically,
ensuring that the test is long enough to provide reliable
measurement, etc.

Once a credentialing exam has been designed and constructed to test
the right set of KSAs effectively, it still may not serve its
purpose if the passing standard is not set at an appropriate level
of knowledge and/or skill. In credentialing, the score required to
pass plays a very important role in validity. If the passing score
is set too low, some people will be awarded a credential even though
they have not attained the level of knowledge or skill that is
expected of the licensed or certified practitioner. If the passing
score is set too high, some applicants who do possess the required
knowledge or skills will be shut out from the credential and its
benefits. In either case, the exam fails to serve its purpose, and
the meaning of the credential is changed.

There are many ways to establish additional evidence of validity.
For example, test scores might be correlated with subsequent job
performance evaluations. The steps listed above, however, are the
basic steps most commonly used for establishing the validity of
credentialing exams. If the purpose of the test has been clearly
defined, if the test was designed and constructed in such a way as
to measure the appropriate KSAs and measure them effectively, and if
the passing score has been set at an appropriate level, it is likely
that the test will serve its purpose effectively.

ERAC's Question and Answer Series is prepared by the CLEAR
Examination Resources and Advisory Committee (ERAC).