Random musings by people who think about educational psychology and the practice of instructional design

November 27, 2013

Reuse, Reduce, Recycle...Test Questions?

by Hana Kim, Pharm.D., PGY-1 Resident, Kaiser
Permanente of the Mid-Atlantic States

As an instructor, it is
difficult to create quality test questions, let alone make new ones every year. The question as to whether to recycle questions
is an issue any organization administering tests or examinations must address. When
a question is reused, it can increase the possibility that it will become more
public to test takers and can give an advantage to those who have access to the
question. There is also concerned about
repeat examinees, those who fail an examination and are required to retake it. Does reusing test questions appropriately challenge
students to learn the material?Does it
provide an unfair advantage to a select few students who have access to the
recycled questions?And, what can instructors
do to alleviate these concerns?

Timothy J. Wood studied the
role of reused test questions on repeat examinees who took the Medical Council
of Canada (MCC) Evaluation Examination (MCCEE).1 The MCCEE is a
basic medical knowledge test for International Medical Graduates that contains
324 multiple choice questions. The MCCEE is offered 3 times a year, 4 months
apart. Wood investigated whether prior exposure to test questions enhanced
performance among 130 repeat examinees. The examinees were presented 36 repeat
questions from the previous examination.The examinees had no knowledge that these questions would be reused. The investigators found that the examinees
scores on the reused and non-reused questions increased equally, a finding that
was consistently with previous studies. He noted that this might be due to
increased knowledge of the subject matter, hopefully due to an increased amount
of studying in preparation for the re-take exam. Wood concluded that prior
exposure to test questions had little impact on the performance. Although there
was no difference in examinee performance on repeat questions, the exam was
given 4 months apart and the examinees did not have access to the questions in
between test administration, so the results may not be representative of what
we’d see in most academic settings.

Similarly, Wagner-Menghin and colleagues conducted a study to evaluate
the effect of reusing written test questions. The authors specifically utilized
the Rasch model, which is a probabilistic psychometric framework measurement
model that estimates item difficulty and ability measures. The study noted four
conceptual factors that should be taken into consideration:

Reuse expectation:
passing items along to new test takers can be beneficial, especially when
reusing items is expected

Cheating attitude: many
studies on cheating have not focused on cheating with the reuse of test
questions

Exam’s consequences:
there are consequences if a student fails an examination; therefore, the pressure
to pass may promote cheating

This
study was designed to quantify the reuse of test questions based on an item’s
level of difficulty. Specifically, the authors introduced a new written
multiple-choice course exam to assess clinical skills in 671 medical students.
To assist the students, a “representative” set of multiple-choice questions was
included in the official study materials. Looking at item content, there was a
larger effect on those that tested application of knowledge versus recall
questions. There was a 50% increase in
student scores on application test questions that were reused compared to 20%
increase when recall-type test questions were re-used. Although the re-used material did not result
in increased overall scores, the authors postulate that the lack of benefit may
be due to a ‘deficit in study organization and time management for late test
takers.’ A test with 30-45% reused
questions (particularly with a large number of recall questions) is, therefore,
unlikely to substantially benefit students.2

When questions are pass down
from one student to another, it creates a potential unfair advantage as some
students have access to the items while others do not.What can professors do to mitigate the
problem of questions being passed down? One possible solution is to make all
previous examinations available to students so there is fair opportunity for
all students. This is exactly what a law professor at George Mason University
has been practicing in his classes. He develops new questions for every exam
and makes past exams and answers available to students.3This strategy certainly improves fairness but
increases faculty workload the workload as new questions need to be formulated
every year.

While some studies
demonstrate a statistically significant difference in scores when questions are
reused, one of the biggest concerns is that questions will get passed down from
year to year, increasing the possibility of “cheating.” These situations are inevitable but instructors
should consider safeguards to help mitigate this problem. Some options include creating
new test questions ever year or allowing a sufficient amount of time (2-3
years) between question re-use. Although there are several suggested solutions,
the question as to whether test items should be reused and recycled remains an
unanswered one.

Copyright Notice

Unauthorized use and/or duplication of the material published on the Educational Theory and Practice Blog without express and written permission from this blog’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to the blog essay's author(s) and the Educational Theory and Practice Blog with appropriate and specific direction to the original content.