Edwina Higgins and Laura Tatham
School of Law

Exploring the potential of Multiple-Choice Questions in Assessment

Abstract: This article explores the potential for using MCQs for
assessment in a subject discipline in which MCQ testing is not universally
accepted as valid. Some of the advantages and misconceptions associated
with MCQ tests are explored, and in particular we look at ways of testing
more than knowledge-acquisition. Two examples are drawn on: a paper-based
summative assessment on the Post-graduate Diploma in Law and web-based
formative assessment exercises in WebCT on a unit on the first year of
the undergraduate law degree. We conclude that the potential for web-based
formative assessment makes it worth investigating the use to which MCQs
can be put in testing intended learning outcomes, and make suggestions
for further reading.

Why MCQs?

We have to start by admitting that our interest in multiple-choice questions
was not sparked by an intellectual debate over the best way to measure
our intended learning outcomes but by an addiction to the TV Show Who
Wants to be a Millionaire? (we do). That said, there were two more
‘educational’ reasons why we started investigating the potential
for MCQs, and we were coming from slightly different angles. First, Laura
‘inherited’ subject leadership of the English Legal System
unit on the Postgraduate Diploma in Law in 1999 (this is the law ‘conversion’
course which is the first step for graduates of other disciplines who
want to qualify as legal professionals). Students are required to show
competence in this unit before being allowed to proceed with the rest
of the programme, so the marks for this unit must be produced almost instantaneously
and traditionally this has been achieved by a knowledge-based multiple-choice
assessment, which also addresses the problem that these students are new
to law and have not had time to develop their legal writing skills. Secondly,
Edwina has a particular interest in developing methods of web-based learning
using WebCT, and we are both involved in a project to provide greater
online support to the students studying the PGDL programme by distance
learning. In particular, Edwina has been working on (some might say obsessing
about) developing more effective means for students to test their understanding
and receive feedback online. MCQs are one obvious mechanism for this,
but we did not want to fall into the trap of using MCQs merely because
the technology facilitates it.

Therefore from two different perspectives both of us are interested in
multiple choice questioning and the questions we wanted to address were:

In summative assessment, could MCQs, adopted for ‘practical’
reasons because of a short marking turnaround time, really test whether
students have achieved the application elements of the intended learning
outcomes of a unit?

In formative assessment, to what extent do MCQs offer scope for effective
self-testing exercises and feedback in a web-based format?

So, we needed to know more about MCQs. This may sound odd to colleagues
in other disciplines who have been using MCQs for years. MCQs are in use
for a variety of tests ranging from driving tests to standard attainment
tests for schoolchildren (not to mention quiz programmes). Within legal
education, the validity of multiple-choice questioning as an assessment
is upheld by the America Bar Association, and all UK providers of the
Bar Vocational Course (bar one) use MCQ tests to assess all non-skills
subjects. Within undergraduate legal education however, poor old MCQs
still have something of a bad reputation. Yes, they are easy to mark,
but you can’t test more than surface learning of facts with multiple-choice
questions, we were told; an averagely lucky monkey could expect to get
reasonable marks in an MCQ test.

We wanted to explore whether this reputation was deserved. Could we use
MCQs to assess effectively the skills and understanding which we wanted
to test? If we could, then this would address Laura’s concerns about
the effectiveness of her summative assessment and open up the potential
for Edwina’s online tests to provide an effective means for formative
assessment. We discovered that some legal educators1 have argued
that MCQs can be used to test the same sort of skills as ‘traditional’
problem-based learning in law. What we also discovered, after speaking
on this topic at a law teachers conference, was that colleagues in other
institutions were keen to explore the potential for using MCQs if these
same concerns could be addressed. It seems the efficiency of MCQ tests
makes them an attractive proposition, whether because of increasing use
of multimedia learning tools or because of the practical pressures of
ever-increasing student numbers placing greater assessment burdens on
staff – as long as this isn’t a move towards ‘dumbing-down’
or encouraging rote learning.

Investigating the myths

Multiple-choice questions are credited with a number of advantages. They
are objective, so variations in marking due to subjective factors are
eliminated (although the questions themselves still have to be scrutinised
to ensure that they are not biased) and this also makes them easy to mark2
(and they do not necessarily require an experienced tutor to mark them).
This is likely to be a significant advantage for over-burdened staff.
They are efficient because questions take less time to complete, and therefore
it is possible to test a greater range of the syllabus through setting
compulsory questions (there seems to be an inherent assumption that MCQ
tests will feature compulsory rather than optional questions)3.
Nevertheless, there are a number of potential disadvantages associated
with MCQs, and these are considered here.

1 Does adopting MCQ assessment mean ‘dumbing down’?

This seems to be the major concern raised about MCQs. They have a reputation
as being “easy” and there is certainly a danger that they
may be – think about those £100 questions on Who Wants
to Be a Millionaire? – but any form of assessment can be designed
for a particular level of difficulty. With MCQs, the difficulty level
depends upon the nature of the questions asked and also how the multiple-choice
questions fit into the overall scheme of assessment. Careful attention
needs to be paid, as with any assessment mechanism, to the ability of
the questions to test the intended learning outcomes set for the students.
In a subject discipline where application and writing skills are required,
replacing the whole of an assessment strategy with multiple-choice tests
comprising simple knowledge-based questions would not allow an adequate
assessment of the required achievement, which would rightly be regarded
as dumbing-down – but the same questions might be extremely useful
for formative assessment throughout a unit, or as part of the summative
assessment. Have you ever despaired, like us, at the desire of students
to “question spot” by ‘learning’ only a limited
portion of the syllabus? A traditional end-of-year examination where students
answer a small number of questions out of a range covering the whole syllabus
(a very popular model in legal teaching, and one often regarded as the
“proper” way to assess student achievement in law) arguably
encourages them to do so. Having a compulsory MCQ section allows efficient
testing across a greater range of the syllabus – if that is what
you want to do – which is probably a more acceptable alternative
to setting a large number of compulsory essays or problems.

Additionally, multiple-choice questions need not be limited merely to
testing knowledge; tests can include more challenging comprehension/application-based
questions, and these might well allow more of the intended learning outcomes
for a unit to be assessed efficiently and thus be valid for summative
assessment. In other words, MCQs are versatile, and it is only if they
are inappropriately used or poorly designed that there is a risk of ‘dumbing-down’.
Some issues of question design are considered later4.

2 Are MCQ scores unrealistically high?

Also contributing to the reputation of MCQs as “easy” is
that the scores tend to be higher than those attained in non-objective
assessments in our experience. This exposes the discrepancy between assessments
which use the full marking range (0 – 100 per cent) and more traditionally-marked
assessments (in undergraduate law at least) where there may be a ‘glass
ceiling’ around the 80 per cent mark. This may be problematic where
an MCQ assessment is used within a diet of conventional assessments. Questions
must be carefully designed to ensure the same level of achievement is
required for a pass in an MCQ test as for other comparable assessments
(although it is really the subjective ‘glass ceiling’ effect
which is at fault, rather than the MCQs which are, after all, objective).

High scores pose less of a problem where the MCQs are used as part of
formative assessment, but students may have optimistic expectations of
their abilities if they have scored highly in MCQs throughout the unit
but are then faced with a final formal assessment asking them to display
different skills. The purposes of any formative assessment (and the summative
assessment, if different) need to be made explicit to students.

3 Can students guess their way to success in an MCQ test?

A second scoring issue is that a student has a theoretical chance of
“guessing” the correct answer in an MCQ – the ‘lucky
monkey’ problem. Arguably this is no worse than a student who adopts
the write-all-you-know approach to a question for which s/he can generally
expect to pick up marks for the correct points the marker has laboriously
identified within the largely irrelevant answer5. Nevertheless
this does present a problem to which there is no definitive solution,
although there are various possibilities which can be adopted6,
including:

use of negative marking to discourage students from equating multiple
choice with multiple guess;

adopting mathematical strategies to “normalise” marks
achieved;

ensuring there are sufficient options (answer choices) for each question
and/or raising the overall pass mark for the test to reduce the likelihood
of a student passing through chance.

Even if these solutions don’t sound satisfactory to make MCQs a
valid option for complete summative assessment, the problems are less
significant for formative assessment (although, as already mentioned,
care does need to be taken about false student expectations where the
strategies for formative assessment differ from the summative assessment).
However, the advantage of objectivity in MCQ summative assessment should
be stressed. Traditional forms of assessment, for example essays, raise
considerable problems in terms of subjective marking, even where assessment
criteria and moderation strategies are adopted, and the phrase “it’s
an art not a science” defensively employed. One of the lessons we
learned was that an assessment method shouldn’t be rejected just
because it is new or different in a particular subject discipline, any
more than another is automatically valid simply because it is the one
that has been used for years. In many cases carefully designed MCQs that
are challenging and require satisfaction of (all or part of) the intended
learning outcomes may be sufficient to reduce the effect of the scoring
problems outlined here.

4 Does using MCQ assessment encourage rote or surface learning?

In our experience some students will rote-learn whatever method of assessment
is adopted. Legal ‘problem’ questions allow examiners to distinguish
between surface learners and those who have shown understanding through
a higher level of application and analysis, and hence avoid rewarding
(and by implication encouraging) surface learning. We wanted to set questions
which would allow students with greater application and analytical skills
to shine. Appropriate MCQs can be used in the same way to distinguish
the surface learner from the deeper learner, where this is the purpose
of the assessment, by setting questions which require comprehension and
application skills, although testing requires a sufficient number of questions
to eliminate those lucky monkeys. We have to remember however that any
form of valid assessment is likely to require the student to display some
form of knowledge acquired by rote learning. Knowledge-based questions
are also a useful formative assessment tool to check whether students
have grasped the basics. We found that giving students a sample of the
type of questions they could expect was the best way of encouraging them
to adopt appropriate learning methods.

5 Can MCQs test oral and written skills?

This may well be a drawback for colleagues considering incorporating
multiple-choice questioning in their teaching. MCQs can test much more
than knowledge but they cannot test oral or written skills7.
They cannot test that students can form appropriate arguments and discuss
the subtleties of a topic. Where the intended learning outcomes of a unit
require oral or written skills a multiple-choice test cannot form the
entire assessment. However, MCQs offer ample scope to test much of the
knowledge and skills required by the legal curriculum and can therefore
be extremely useful for formative assessment purposes and, where appropriate,
as part of summative assessment. We decided to concentrate on what we
could test rather than on what we couldn’t!

Features of good question design

Many of the potential problems with MCQs identified above can be ‘designed
out’ with well-written questions. Poorly designed questions may:

Give away clues to the answer

Fail to test the skills required by the intended learning outcomes

Contain “implausible distracters” (obviously wrong answers)
which can be eliminated by students with only limited knowledge

Encourage rote learning

Confuse or frustrate students with sound understanding

All of the above problems could result in scores which do not reflect
student ability.

There are numerous guides to writing effective multiple-choice questions8,
and we don’t have scope to go into more than a few key points here.
A crucial factor will be the ‘distracters’ which are written,
and Who Wants to Be a Millionaire? again provides us with an illustration.
Recently the following question came up at the £100 stage: Which
US state was the third state to accede to the Union in 1787? To us
at least (!) this is a comparatively difficult question which certainly
had the contestant gulping, but the level of difficulty became laughable
when the options were revealed, which were (a) New Cardy (b) New Woolly
(c) New Jumper and what you have probably already realised is the correct
answer (d) New Jersey. We use this to emphasise that the art of providing
likely wrong answers (or plausible distracters to give them their
correct term) is a key factor in writing good MCQs9. The same
question would have been harder if the options had been, say, New Hampshire,
New Jersey, New Mexico or New York because all four are actually states.
Here the difficulty lies not with the question stem itself but with the
plausibility of the distracters.

It should be emphasised that tutor perception of the plausibility of
a distracter may differ widely from that of students taking the test.
We once set a question about the burden of evidential proof in a criminal
case which we intended to be a fairly simple knowledge-based ‘opener’
to a test. The scenario involved a man who had been arrested for an offence
involving a musical monkey, and we were dismayed at the number of students
who felt that the burden of proof in this case would be on the monkey…
(What this says about our teaching is quite possibly more depressing.)
One of the best ways of writing plausible distracters is to utilise common
student mistakes. This forms a “trap” for weaker students
but allows students with better understanding to demonstrate it. The use
of distracters based on common misconceptions in web-based formative assessment
situations is explored below.

A further trap which needs to be avoided is the temptation to make a
factual MCQ ‘harder’ by simply testing more obscure
knowledge (think about the top-money questions on Millionaire).
Factual questions need to be related to the intended learning outcomes
for the course. Best practice suggests that designing harder questions
should involve moving through Bloom’s Taxonomy from knowledge-based
questions to questions which demand comprehension or application10.
Application skills are a particularly important aspect of legal study,
so this transition was something we were keen to develop in our examination
for the PGDL English Legal System unit.

Developing the Multiple Choice Examination for English Legal System

This unit had adopted an MCQ test for its assessment because of the quick
marking-time. Students must show competence in this unit before being
permitted to continue with the remainder of the programme, so it is imperative
that the results are released quickly. With over 150 students on three
modes this is a heavy marking burden on the unit team. Student on the
Distance Learning mode, for example, take the test on the Friday afternoon
of the first of only four attendance weeks throughout the year and the
staff team release their results within an hour of the test’s end
so students know whether they can continue with the remainder of the programme
before leaving Manchester.

Given the ‘myths’ identified above, we needed to investigate
the validity of the MCQ assessment. We were not too concerned about the
scoring issues because in this assessment we are simply looking for competence;
we do not need marks to relate to a percentage or classification. We readily
admit that this allows us to side-step one of the key problems with MCQs
identifed by many authors. As far as the limitations for testing oral
and written skills are concerned, again this was not a worry as these
were never part of our intended learning outcomes. These skills are acquired
and assessed later on in the programme.

The issue which did concern us was how to ensure that the unit’s
intended learning outcomes – one of which is legal application skills
– were assessed. How could we set a test which would demand more
than just knowledge repetition? We devised a strategy which involved the
students having to work through a set of materials additional to the MCQ
test paper itself during the exam. The additional materials comprise a
fictitious Act of Parliament and a series of scenarios which have led
to prosecutions under the Act, including extracts from the (also fictitious)
legal judgments for each ‘case’. The students then answer
multiple-choice questions which test their knowledge of the legal system,
their comprehension skills in relation to the materials given and their
ability to apply their knowledge to the scenarios and draw conclusions
about the connections between the cases. This method of assessment allows
us to retain the fast turnaround time we need, while testing the students’
fulfillment of the ‘higher level’ intended learning outcomes
of the unit.

Developing formative assessment exercises using WebCT

Producing an exercise for English Legal System which allowed us to test
comprehension and application rather than just knowledge proved that MCQs
could provide a useful tool for assessing the skills law students must
acquire. This was an interesting prospect because it opened up the potential
for creating formative exercises for students to access online (of course
formative exercises don’t have to be online – it could be
an oral run-through of answers following a test or written feedback on
the questions, where students can identify for themselves where they went
wrong – but the great advantage is that tutors don’t have
to mark web-based questions). In contrast to essay or problem assessments
where feedback is laboriously dependent on individual answers, the closed
answer-range for MCQs means that tutors know all the possible errors that
can be made in advance of the test being completed, so MCQs offer an excellent
opportunity for the tutor to provide generic yet focussed feedback. In
particular, the format of MCQs is ideally suited for delivery via a web-based
learning environment, which means that students can complete assessments
and obtain marks and feedback immediately independent of tutor-contact.

Our first year undergraduates study a ‘frontloaded’ unit
called Introduction to the Study of Law for the first four weeks of the
programme, and this is supported by a WebCT site developed by Edwina in
the distant days of a university online fellowship. It contains revision
‘tutorials’ to consolidate the work done in the classroom,
provide further tips for completing their portfolio tasks and help students
revise for the end-of-unit exam. The students complete part of their assessment
by submitting a WebCT quiz, which is a knowledge-based test about the
legal system – the point of the exercise is not so much that they
know the stuff but that they have been able to find it within certain
web-based materials, to familiarise them with the amount of legal information
available on the internet (which is one of the intended learning outcomes).
What we particularly wanted to do, however, was to develop formative exercises
using the self-test facility in WebCT. This allows students to get instant
feedback, and – unlike the quiz facility – WebCT does not
record a ‘score’ for the student, so they can complete the
exercise without feeling self-conscious. Students can have as many goes
as they like at the self-test, either immediately or after further revision.

A particular advantage of using the self-test facility is that it helps
to make the revision tutorials on the WebCT site more interactive; instead
of just reading material which (except for the pretty pictures) could
basically have been given in a handout, the students can check their understanding
at regular points throughout the revision tutorials and have the opportunity
to engage with the material rather than just staring at the screen. There
is more to using the site than just pressing the ‘print’ button!
Sometimes, the self-test exercise may only be one question, as for example
with the self-test reproduced in figure 1, which accompanies a page in
which definitions of the legal term “ratio” were explored
with a practical example. The students have to be able to distinguish
ratio from the related concepts of ‘obiter’
and ‘decision’. None of the options given is a ‘standard’
definition of ratio, so this tests students’ comprehension of the
material.

Figure 1

Question: Which of the following comes closest to
your idea of what 'ratio' means?

Options

Feedback

The most crucial fact in the case

No - no fact alone, however important, can be ratio because
there must be some reasoning involved. Remember that you are
looking for the 'why' as well as the 'what'. Ask yourself: how do
you know it is the most important fact? Your answer will begin "Because
..." - that 'because' is the reasoning you are looking for. The
fact may form PART of the ratio, when combined with that reasoning.
Which definition looks like it involves reasoning?

The end result of the case

No - this sounds more like a definition of 'decision'. Remember
the decision is the outcome; the ratio is the reasoning which
led to that outcome, based on the material facts in the case. Which
definition looks like it involves reasoning?

The part of the case which is applied in later cases

This could be a definition of ratio. But it could also be
a definition of obiter. So if you were comparing ratio
with decision, this definition would help, but it wouldn't
help if you were comparing ratio with obiter. Obiter
can be applied in later cases too; the difference is that the ratio
might HAVE to be applied (depending on the court) the obiter
can only ever be persuasive – it would never HAVE to be applied.
There is more to help you understand obiter on the next page
of the tutorial. Can you see a definition which would only
fit ratio?

The judge’s reasoning in the case.

You are getting warm with this answer; but remember that obiter
can also involve the judge’s reasoning in the case. The difference
between the two is that the ratio is reasoning in relation
to MATERIAL facts, whereas obiter is reasoning in relation
to any other facts. Can you see a definition which narrows down to
only part of the reasoning?

The reason for the outcome
[correct]

Sorry if you were expecting to see a more traditional definition
of ratio but well done for figuring it out. Although it not
an ideal definition of ratio, it is probably the closest of
the choices offered to you. We know that ratio involves reasoning,
but we also know that the reasoning must be in relation to the material
facts to be ratio. Therefore linking the reasoning to the outcome
(although outcome on its own can clearly never be ratio) means
that it must be linked to the material facts of the case. Another
way of putting this is to think that ratio can be equated with
a statement: THIS...THEREFORE...THAT. The 'that' is only the outcome
and therefore isn't ratio; the 'this' on its own is not actually
linked to the case and therefore might only be obiter, but
put the whole thing together and you must have ratio. Trying
to reduce a case to a 'This...therefore that' statement may be a helpful
way of trying to find the ratio of the case.

Here, what we were trying to do was to maximise the effectiveness of the
feedback given so that students would be able to improve their understanding
to reach the right answer (although not all our self-test questions have
the quantity of feedback in the illustration). Effective feedback is the
key to formative assessment, so it is vital to devote time to writing
feedback from which students can learn and which encourages them to think
and evaluate their level of understanding: it does not simply tell students
where they went wrong but why. Well-written questions with plausible
distracters based on common student mistakes are particularly useful in
communicating to students the gaps in their knowledge or understanding
(all the distracters in Figure 1 are based on common errors about the
concept of ratio). The more plausible a distracter is, the easier
it is to write feedback for it; the better the feedback, the more the
students learn from it.

Some useful pointers for feedback include:

Giving feedback which is as detailed as possible – “No”
or “Wrong” does nothing to help the student identify the
reason for their mistake, although with some simple factual questions,
and with less plausible distracters, it can be difficult to do otherwise;

Writing feedback at the same time as the questions rather than trying
to do it later helps to focus attention on common student mistakes and
how they can be corrected;

If the question is web-based, directing students to the resources
required for them to correct their understanding and try again, rather
than simply giving them the correct answer;

Where appropriate acknowledging, with encouraging feedback, those
students who have answered questions well, or who have got something
‘nearly right’ by choosing a particularly plausible distracter.

Conclusions

This article probably seems of little value to those in subject areas
where MCQ tests are a standard part of the curriculum – we know
we are beginners – but the intention is to evaluate their potential
in areas where they have traditionally been rejected because of the myths
we identified earlier. MCQs are not, ultimately, a mechanism for reducing
the marking workload (if only!); there are skills they simply cannot test
and care needs to be taken with the scoring scheme. So why bother to consider
MCQs? Exploring the use of objective questions really opens up the potential
for web-based formative assessment so students can get instant feedback
and ‘individual’ (albeit automated) feedback on their understanding.
Having the staff resources to provide this individual support face-to-face
is an increasingly rare luxury as student numbers grow. Of course, feedback
from a PC screen isn’t the same as from a nice human tutor, but
our student evaluation forms show most of the students find it helpful.
Some even prefer it, because it has the advantage of flexibility; students
can interact at the times that suit them. Online exercises are a valuable
way to transform web resources from handout graveyards to something more
engaging.

Our enthusiam for both our MCQ experiments has led to some cross-fertilisation.
We now set the LL.B. Introduction to the Study of Law students an MCQ
application exam in a similar format to the PgDL English Legal System
one (though it is a bit easier) and the ELS students now get access to
a WebCT site with formative exercises similar to the ISL ones. The two
original ‘angles’ we set out to investigate have come together
in provision for both sets of students of an exercise on WebCT in which
they can work through a previous year’s MCQ test and additional
material, and obtain feedback on their progress via a WebCT self-test.
Our next project is to consider how a standard ‘problem’ question
in law could be broken down into MCQs to provide formative web-based exercises
in other legal subjects, to give greater support to our distance learning
students. Inevitably, the main stumbling-block is finding the time.

If you are thinking of developing MCQ exercises, the diagram in Figure
2 and the resources list below may be of help.

Log on to student motivation. The Times Higher Education Supplement
May 3 2002
Describes the use of formative web-based MCQs to complement face to face
teaching

Paxton M, (2000) A Linguistic Perspective on Multiple Choice Questioning
25 Assessment & Evaluation in Higher Education 2
A critique of the emphasis on multiple-choice testing as a response to
increasing student numbers in South Africa