Paul Riddy and Karen FillCentre for
Learning and Teaching, University of Southampton

Paper presented at the British Educational Research Association
Annual Conference, Heriot-Watt University, Edinburgh, 11-13 September
2003

Introduction

Faced with a bewildering array of eLearning
materials, modules, courses and programmes how can tutors or learners decide
which ones will best meet their needs and be of sufficiently high quality? Which
of a range of standards should eLearning developers adopt to understand and
address the requirements of teachers and learners and ensure the quality of
their offerings?

A European team from universities in England,
Spain and Germany, and training organisations in Italy and Greece, has worked
together over two years to produce a methodology for the evaluation of on-line
open and distance learning materials and Internet based programmes of study. The
team members were multi-disciplinary and represented both academic and
commercial backgrounds. Their expertise spanned course design, development,
production and delivery. Some members had skills in technological implementation
and course management. The aims of this EU funded project, Methodology for the
Analysis of Quality of Open and Distance Learning delivered via the Internet,
(MECA-ODL1) were to:

provide guidance for improving the
overall quality of e-Learning courseware

The MECA-ODL evaluation methodology essentially
looks at characteristics, which define the quality of a learning resource in all
stages of its development and implementation, in order to assess if it is
likelyto besustainable and deliverquality learning for
users.

The project developed through four main
stages:

Compilation of a compendium of existing ODL resources from
within each partner country, and non-partner countries, and review of the
scope of ODL materials

Development of an evaluation methodology, based on the
review results and existing evaluation schema

Development and testing of the on-line evaluation tool

External evaluation of the on-line tool.

The major outputs from the two-year project were a
methodological guide for the analysis of quality in eLearning and a software
tool that can be used on-line to evaluate materials, modules, courses and
programmes.

This paper presents an overview of some evaluation
schemes which existed at the project's inception, explains the MECA-ODL
methodology and contrasts it with these, discusses the results of external
evaluation, and makes recommendations for further development of the methodology
and tool.

During the project there have been significant
developments in the specification of Learning Objects (LO's), defined as "any
entity, digital or non-digital, which can be used, re-used or referenced during
technology supported learning" (IEEE, 2001). LO's are likely to play an
important role in the construction of eLearning resources, and readers are
referred to Littlejohn (2003) for an extensive discussion of this subject. It is
envisaged that evaluation methodologies will need to take account of this in the
future, but LO's are not explicitly referenced in the current MECA-ODL
evaluation methodology.

Existing Standards and Specifications

The main body involved with standards on a global
scale is the International Organisation for Standardisation (ISO). This
represents a hybrid of organisations from the public and private sector, with
members from 147 countries, and its standards are truly international:"International Standards provide a reference framework, or a common
technological language, between suppliers and their customers - which
facilitates trade and the transfer of technology." (ISO, 2003). The ISO
standards, which relate to this topic, are:

ISO 9000The ISO 9000
family of international quality management standards and guidelines has
earned a global reputation as the basis for establishing quality
management systems. (ISO 9000, 2000)

ISO 14915ISO
14915-1:2002 establishes design principles for multimedia user interfaces
and provides a framework for handling the different considerations involved
in their design. It addresses user interfaces for applications that
incorporate, integrate and synchronize different media. This includes static
media such as text, graphics or images, and dynamic media such as audio,
animation, video or media related to other sensory modalities. (ISO 14915-1,
2002)

Similar in concept, the Total Quality Management
(TQM) model is widely used, but is not actually a standard. This represents a
structured system for satisfying internal and external customers and suppliers
by integrating the business environment, continuous improvement, and
breakthroughs with development, improvement, and maintenance cycles while
changing organizational culture. (IQD, 2001)

These, and similar schemes, are concerned with
broad aspects of quality. They are complex, weighty and, for many educational
developers, probably unusable. They enshrine notions of quality that start with
development and continue beyond production, but are actually of little use for
education.

Other bodies, working in the area of Learning
Objects (LO's), have now produced tightly defined specifications and standards
which consider the technical implementation, interaction (interoperability), and
educational qualities of LO's. Two of the major international players in these
specifications and standards are the IMS Global Learning Consortium
(www.imsglobal.org) and the Institute of Electrical and Electronics Engineers
(www.ieee.org). Activities specific to the UK are coordinated by the Centre for
Educational and Interoperability Standards (CETIS, 2003). These developments are
already having an impact on the development of eLearning and future evaluation
tools.

Evaluation of eLearning Resources

To date a number of evaluation schemes for the
quality of eLearning resources have been developed. Unlike MECA-ODL, other
examples given below do not cover all aspects of resource design and
implementation. The differences are outlined in each case and summarised in
Table 1 below.

E-Learning Courseware Certification

Promoted by the American Society for Training & Development
(ASTD), E-Learning Courseware Certification (ECC) is

for asynchronous Web-based and multimedia
courses (and) recognizes courses that excel in usability and instructional
design. (...) The ASTD Certification Standards Committee, composed of
e-learning experts, academicians, instructional systems design
practitioners, and other learning leaders in the industry, has created these
standards (which) are supported by examples, clarifications, definitions,
scoring criteria, and other supporting information. (ASTD, 2002a)

The ASTD scheme evaluates compatibility,
interface, production quality and instructional design of e-learning courseware
and now offers an on-line pre-evaluation tool:

which allows the end-user to pre-screen an
asynchronous learning course against the 19 standards that are in place. The
browser-based tool allows the end-user to open a course in one window and
the tool in another so that the course can be reviewed. The tool keeps score
based on input from the end-user. (ASTD, 2002b)

Final certification is still done by external
evaluators. ASTD is planning to enable access to an online database of certified
courses.

Differences: The scope of the evaluation criteria
does not engage with the readiness of an organisation to deliver eLearning,
including the existing training culture. The criteria potentially offer a good
assessment of other aspects of a course, but it is difficult to tell from the
information available. ASTD offer a certification service for eLearning
courseware, for which they charge a fee, and presumably have experts well versed
in all aspects of course design and development to ensure comprehensive
evaluation.

Quality on the Line

The American Institute for Higher Education
Policy reported on '24 benchmarks considered essential to ensuring excellence in
Internet based distance learning' (IHEP, 2000). These were arrived at after a
review of the literature and carrying out research with institutions delivering
distance learning to determine which benchmarks were in use. The benchmarks are
divided into seven categories of quality measures, namely:

Institutional support

Course development

Teaching/learning process

Course structure

Student support

Faculty support

Evaluation and assessment.

Differences: This is a manageable set of criteria
by which to evaluate the quality of a course but, in our view, does not provide
a complete enough picture, especially in the areas of course design, content and
production.

Consumer Based Quality Guidelines for Learning
Technologies and Distance Education

This consumer guide is based on extensive research
into the literature related to technology assisted (distance) learning. The
quality of education and training is defined by what makes distance learning
modules effective and efficient from a consumer's perspective. The criteria are
presented as a serious of 15 questions, the scope of which is illustrated
below:

Completion takes the form of credit or credentials that
are:Recognised by professional accreditation
bodies and other educational institutions, equivalent whenever learnt
on-site or at a distance, transferable...

Differences: As consumer guidelines these provide
a good starting point for a potential learner to assess if a course is likely to
provide what they are looking for, and indirectly suggest evaluation of the
design, content and implementation. However, the consumer focus leads to the
guidelines being output based, and they do not offer sufficient detail for a
complete evaluation.

MECA-ODL

The basis for evolving the evaluation methodology
was the compendium of eLearning resources, the standards and evaluation
methodologies outlined above, and the experience of the team. The methodology
developed encompasses seven phases, spanning requirements ranging from
commercial to Higher Education (HE):

Conception - considers the readiness of the
organisation to engage with eLearning

Analysis - establishes eLearning needs and
methodologies

Design - pedagogical design and development process

Content - implementation of design including
selection of technologies

Production - building the application, content,
development team, sequencing

Each phase has a fully defined set of criteria
giving a total number of 140 criteria. An example from the Conception phase is
given in Figure 1 below.

Three categories of potential users of the guide
and tool were identified:

Developers of eLearning modules / courses

Users of eLearning modules / courses (learners)

Resellers of eLearning courses (trainers /
tutors).

After developing a range of criteria, the project
team considered each to minimise duplication and finalise their allocation to
each category. Each criterion was also weighted, on a scale of 1 to 5, to denote
its relative importance in an overall evaluation. These are recorded in the
paper guide (see Figure 1).

Phase: I. - Conception Criteria

I.1 Building e-learning strategy.

To build an e-learning strategy means to take into
consideration and change not only issues of technology but also issues of
learning effectiveness.

Code

D

U

R

Criteria

Weight

Comments

I.1.1

X

The learning culture in the organisation has been considered.

5

The impact of learning culture on effectiveness should be
considered.

I.1.2

X

The learning background and interests of the organisations' staff have
been considered.

4

The impact of staff factors should be
considered.

Figure 2: Example from the printed guide of the
Conception Phase

Note: the D/ U / R columns are used to
indicate which categories of user the criteria are applicable to: Developer,
User or Reseller.

Comparison of eLearning Evaluation schemes

A summary comparison of the schemes discussed
above with the MECA-ODL scheme is given in Table 1 below. The table indicates
the comprehensive scope of the MECA-ODL criteria compared to the other
evaluation schemes. The descriptions above also highlighted the differences in
terminology, and in the selection and organisation of evaluation criteria into
categories. For example criteria under QoL course development category
would fall into several of the MECA-ODL categories. The CBQG are all framed as
evaluative questions, but are equivalent to some of the MECA-ODL
criteria.

The other assessment schemes do not provide an
indication of the importance of the criteria (the weightings), or recommend
different sets of criteria for developers, users and re-sellers. These become
particularly useful in the on-line implementation of evaluation methodology, the
evaluation tool

The MECA-ODL online tool

The online tool is currently available in English,
German, Greek, Italian and Spanish at
http://wipaed-dbase.sowi.uni-bamberg.de/eLearning/WebObjects/mecaODLtool. At
registration a user selects the appropriate category - developer, reseller or
user - and is presented with the relevant phases and criteria when evaluating an
eLearning module or course. The evaluation process involves awarding a 'score',
again on a scale of 1 to 5, for the implementation of criteria (see Figure
2).

Figure 2: Data entry screen for application
evaluation

An evaluator may choose to ignore or score each
criterion presented. Evaluations can be partially completed and returned to at a
later date. The tool calculates an overall score for each phase, displaying this
as a number. Scores for all the criteria are also shown as numbers and
graphically (bar charts) and are automatically compared to the MECA-ODL 'ideal'
that has been derived from the weightings agreed by the project team. Examples
are shown in Figures 3 and 4

Figure 3: Numerical comparison of application with
the ideal

The tool was designed with flexibility for users
to input their own set of weights for each of the criteria, but it is important
that the same set is used if cross-comparisons between applications are
required. At administrator level, the base set of criteria and reference
weightings can be also be changed. The criteria, general descriptors, and
operational instructions are stored in a database, making it straightforward to
create a version in another language.

Figure 4: Graphical Comparison of Quality

As mentioned earlier, the tool was designed to
span the spectrum of eLearning evaluation ranging from commercial to Higher
Education (HE). In reality, within HE the full range of criteria will be rarely
used, as many of the criteria in the early phases are not considered, or in some
cases relevant, in the HE course selection process. This is brought out in the
example below.

Evaluation of PosFix

PosFix was developed at the University of
Southampton in 1993, as a small, standalone application for learning about
position fixing, with an emphasis on electromagnetic systems. It was designed
specifically for delivery in an on-line environment using a 'resource base'
approach (Hall, 1994), and was implemented within the Microcosm environment
(Davies, 1994). The application was still in use by undergraduate and MSc
students in 2000.

PosFix was designed for undergraduates,
postgraduates, and newcomers to Marine Science, with content suitable for
beginners to intermediate users. It replaced sections of taught courses at
undergraduate and masters level, aiming to provide basic knowledge that would be
used in hands-on practical sessions at sea. Development was funded by the
institutional TLTP project. The primary considerations were infrastructure
readiness for delivery, educational design including integration with the
existing courses, formative and summative evaluation. The application was
designed to be introduced via a tutorial, and its use was seen as contributing
to the transferable skills of the learners.

The evaluation using the MECA-ODL tool was
undertaken from the perspective of a developer, when the maximum number of
criteria can be scored. PosFix was used in a blended learning mode, and
developed for an academic teaching environment, so many of the criteria relating
to distance learning delivery and commercial training were inappropriate. Most
of these were in the Delivery,Analysis and Conception
phases. An example of the graphical output for the Delivery Phase is shown in
Figure 5. Most of the scores for criteria used were between 60% and 80%, with
the weaker areas being:

Conformance to usability guidelines - these were not well
established when PosFix was designed

User control of interactivity

User ability to record notes

Availability of user tracking information.

The last three were largely a function of the development
environment.

Figure 5: Graphical Comparison of Quality for the
Delivery Phase of PosFix

Note: missing columns illustrate the large number of
criteria that could not be scored for PosFix.

Overall PosFix scored highly. Principal weaknesses
were in the design and content which were not adaptable for different learning
styles, and non-conformance to metadata standards that are becoming increasingly
important in today's development climate.

The evaluation highlights the difference between
an academic and commercial environment. Within the University we were working
with a largely pre-defined audience. Some commercial training companies
specialise in tailoring training packaging to meet and the needs of a small
group of learners.

External assessment

The MECA-ODL guide and tool have been assessed by
several external evaluators, in HE and commercial training, in each of the
countries involved in the project. All agreed an evaluation tool was needed and
common views of the benefits of the MECA-ODL approach were deemed to be:

it provides a useful checklist for building an
application

it can be used as a guide for developing other
training/educational applications (e.g. digital training manuals)

the graphical comparison function is very useful for
comparing different applications

the concept of weighted criteria

the flexibility to adapt for own needs.

They also made many perceptive observations and
suggestions for further developments that the team hope to address if further
funding is made available. These include:

Distinctive sets of criteria for different purposes, e.g.
commercial, academic

On-line training course for evaluators

Wizards - e.g. to set up an evaluation (criteria
set)

Repository of example on-line evaluations

Link with learning objects metadata.

Although some of the original assessors commented
unfavourably on the large number of criteria, recent discussions with interested
parties who tried out the tool at the University of Southampton suggested there
are not too many. Users of the guide and/or the tool should retain the
flexibility to use them all or select their own sub-sets.

Planned developments of the Tool

Evaluation of eLearning resources requires a broad
skill set. Users without any background in eLearning development and
implementation would find it difficult to carry out an evaluation. Making a lot
more information available via the user guide and on-line help system would go
some way towards alleviating this problem. Ideally would-be-users would complete
a module on eLearning evaluation to support use of the tool.

The main developments in the methodology are in
removing some ambiguities in the criteria, and producing revised sets of
criteria for different user groups. In conjunction with this it would be useful
to have a simple way to set up the criteria and weightings for a particular user
perspective. It is anticipated this could be done through developing 'wizards'
which select the criteria set according to data input by the user. For example,
one such wizard could provide a much-reduced criteria set for the novice
eLearning evaluator.

Conclusion

This paper describes the development of an
evaluation methodology for eLearning resources, the implementation of this as an
online tool, its appraisal to date, and points out directions for future
development. The results suggest that such a tool will be an invaluable aid for
designers and developers of eLearning materials, who can use the criteria set as
a checklist. It could also be very useful for users and resellers of materials,
who want to compare the characteristics of different applications. However, our
use has shown that complete evaluation of an application requires access to
information which is frequently not available, or hard to find. For such
methodologies to become a useful 'standard' for evaluation, it is essential that
application developers are primed, and prepared, to make the information
available. As with any pedagogic evaluation, significant background in learning
and teaching methods is required to check if the pedagogic aspirations of an
application have been implemented effectively. This requires skilled
individuals, who are also comfortable with eLearning technology, to carry out
the more comprehensive evaluations. Finally, it would be important to ensure
consistency between evaluators if a meaningful database of evaluations was to be
constructed.

The MECA-ODL project was carried out with the support of the
European Community in the framework of the SOCRATES programme. The content of
this Project does not necessarily reflect the position of the European
Community, nor does it involve any responsibility on the part of the European
Community.

This document was added to the Education-line database on 28
October 2003