Abstract

Measuring the quality of a b-learning
environment is critical to determine the success of a b-learning course.
Several initiatives have been recently conducted on benchmarking and quality in
e-learning. Despite these efforts in defining and examining quality issues
concerning online courses, a defining instrument to evaluate quality is one of
the key challenges for blended learning, since it incorporates both traditional
and online instruction methods. For this paper, six frameworks for quality
assessment of technological enhanced learning were examined and compared
regarding similarities and differences. These frameworks aim at the same global
objective: the quality of e-learning environment/products. They present
different perspectives but also many common issues. Some of them are more
specific and related to the course and others are more global and related to
institutional aspects. In this work we collected and arrange all the quality
criteria identified in order to get a more complete framework and determine if
it fits our b-learning environment. We also included elements related to our
own b-learning research and experience, acquired during more than 10 years of
experience. As a result we have create a new quality reference with a set of
dimensions and criteria that should be taken into account when you are
analyzing, designing, developing, implementing and evaluating a b-learning
environment. Besides these perspectives on what to do when you are developing a
b-learning environment we have also included pedagogical issues in order to
give directions on how to do it to reach the success of the learning. The
information, concepts and procedures here presented give support to teachers
and instructors, which intend to validate the quality of their blended learning
courses.

Introduction

E-learning has become widely used in
every type of education (traditional and formal education, continuous education
and corporate training) because of its characteristics such as flexibility,
richness of materials, resource-sharing and cost-effectiveness.

In this work we paid more attention to
the blended-learning (b-learning) systems, which consider systems “combining
face-to-face instruction with computer-mediated instruction” (Graham, 2004).
E/b-learning has been largely used in the context of higher education. It
includes a wide range of learning formats including self-study and
instructor-led in both an asynchronous and synchronous mode. The e/b-learning
systems may represent as an alternative to traditional teaching/learning and
training and, therefore, has had to battle for recognition. As a consequence of
this need, procedures have been developed in order to demonstrate its quality.

Evaluating the quality of a b-learning
environment is not an easy task since this concept is not objective. It depends
on the students’ perceptions and there are also several multi-dimensional
variables factors with factors that we have to take into account. As the number
of b-learning courses is increasing, more and more, it is important to evaluate
the quality offered in order to help the potential users choosing the best
course.

Models to evaluate the quality in education context

There are several standards related to
quality in education context. The ISO/IEC 19796 series of standards, published
as ISO/IEC 19796: Information technology – Learning, education and training –
Quality management, assurance and metrics, provides a framework for
implementation and monitoring quality management systems in educational
organizations. Currently, two standards of this series are published – ISO/IEC
19796 Part 1 – General Approach and ISO/IEC 19796 Part 3 – Reference methods
and metrics – and other three are under preparation – ISO/IEC 19796 Part 2:
Harmonized quality model; ISO/IEC 19796 Part 4: Best practice and
implementation guide; and ISO/IEC 19796 Part 5: How to use ISO/IEC 19796-1.

The ISO/IEC 19796-1 (ISO/IEC 19796-1,
2005) is a general framework to develop and implement quality in educational
organizations. This standard contains the reference process model “Reference
Framework for the Description of Quality Approaches (RFDQ)”, that is divided
into seven process categories: needs analysis, framework analysis,
conception/design, development/production, implementation, learning process,
and evaluation/optimization. RFDQ model covers the whole lifecycle of learning,
education and training, including e-learning and b-learning.

Recently, a Portuguese standard that
specifies requirements for a vocational training management system, including
technology enhanced learning was published (NP-4512, 2012). This standard has
its focus on ensuring that the organization processes (such as those related to
the formative cycle: diagnosis of training need; design, development and
innovation of training products; planning, organization and realization of
training courses and assessment and/or certification of learning) are managed
with the aim of increasing customer satisfaction and conformance. The standard
NP 4512:2012 already considers the scenarios of e-learning, b-learning or
m-learning (mobile learning). Other countries, such as the United Kingdom,
Spain and Germany, also have developed standards for e-learning. The British
Standard 8426:2003 (BS-8426, 2003) makes recommendations for e-support in
e-learning systems independently of the pedagogical approach (e-support
provided by human tutors or automated; learners’ work done individually or in
groups; or the underlying pedagogy of a course involves learners in
constructing their own understanding or in committing course content to
memory). The Spanish standard UNE 66181:2012 (UNE-66181, 2012) specifies
guidelines to identify the characteristics of virtual courses in relation to
potential customers. The use of this standard is intended to increase
transparency and market confidence in e-learning. The Germany standard PAS
1032-1 (PAS 1032-1, 2004) provides a reference model for quality management and
quality assurance especially developed for the education and vocation training
sector, with a special focus in e-learning.

Still about this topic, several
initiatives have been recently conducted on benchmarking and quality in
e-learning products. For this paper, six frameworks for quality assessment of
technological enhanced learning were examined and compared regarding
similarities and differences. These frameworks were chosen as representative of
international initiatives, already used in the worldwide in higher education
institutions. A brief description of these frameworks follows.

Open ECBCheck Initiative (Ehlers, 2010)
from EFQUEL (European Foundation for Quality in e-Learning), aimed to offer a
quality label for e-learning in Capacity Building, is rooted in four best
practice labels: UNIQUe (EFQUEL, 2011), EFMD CEL (EFMD, 2010), D-ELAN DELZert
(Ehlers, 2010) as well as ISO/IEC 19796-1 (Pawlowski, 2006). Open ECBCheck
follows a certification process with three major steps: self-assessment, peer-review
of self-assessment report, peer-review report, including evaluation results,
and learning report as well as recommendations for certification. ECBCheck
covers seven main areas: Information about the program and respective
organization, target group and orientation, quality of the contents, programme
/course design, media design, technology, evaluation & review (Ehlers,
2010).

The SEEQUEL Core Quality Framework is an
outcome of the SEEQUEL (SEEQUEL, 2004) project, which was supported by the EU
e-learning initiative, originated from the collaboration between the e-learning
Industry Group (eLIG) with a number of European expert organizations and
associations, co-ordinated by the MENON Network. The SEEQUEL Core Quality
Framework is based on a matrix where a list of common quality criteria
applicable to the whole e-learning experience can be weighted by the several
users (people or organization), enabling any category of stakeholders to
position their perception of quality with respect to the perceptions of another
category of stakeholders. The SEEQUEL Core Quality Framework proposes a quality
model with three main quality characteristics: the learning resources, the
learning processes, and the learning context.

E-xcellence (EADTU, 2012) is a project
that started in 2005, with the support of the eLearning Programme of the
European Commission (DG Education and Culture), and in cooperation with 13
higher education e-learning and quality assessment and accreditation partners
in Europe. An outcome of this project was an assessment tool (programme and
institutional level) providing a set of benchmarks and quality criteria
covering six main areas: strategic management, curriculum design, course
design, course delivery, staff support and student support.

The report “Quality on the Line:
Benchmarks for Success in Internet Based Distance Education” (Merisotis &
Phipps, 2000) was commissioned by the National Educators Association and
Blackboard, Inc., and prepared by the Institute for Higher Education Policy (IHEP).
This report identifies 24 individual quality indicators, in seven main
categories: institutional support, course development, teaching/learning,
course structure, student support, faculty support, and evaluation and
assessment. The study called each indicator a benchmark, but they are, in
reality, attributes to indicate overall quality and so they are not measurable
against other institutional results.

The Quality Matters Program (QM, 2011) is
dedicated to quality assurance for online education, with a broad range of
subscribers, including K-12 schools and higher education institutions. QM is a
faculty-centered, peer review process that is designed to certify the quality
of online and blended courses. There are three main components in the QM
Program: The QM Rubric, the Peer Review Process and QM Professional
Development. The QM Rubric framework is a set of 8 general areas and 41
specific criteria used to evaluate the design of online and blended courses.
The general areas are: course overview and introduction, learning objectives,
assessment and measurement, instructional materials, learner interaction and
engagement, course technology, learner support, accessibility. The framework is
supported by a set of online tools to facilitate the evaluation by a team of
reviewers.

In the UK, two bodies established quality
assurance guidelines for their institutions: the Quality Assurance Agency for
Higher Education and the Open and Distance Learning Quality Council (ODLQC). In
the first one, e-learning guidelines are an extension of general learning
guidelines. The ODLQC defines six main areas for quality assurance criteria
(and possibly accreditation) as follows: outcomes, resources, support, selling,
providers, collaborative provision (ODLQC, 2005).

These frameworks aim at the same global
objective: the quality of e-learning environment/products. They present
different perspectives but also many common issues. Some of them are more
specific and related to the course and other are more global and related to
institutional aspects. The EFQUEL (EFQUEL, 2011) and E-xcellence (EADTU, 2012)
are the more detailed ones.

There are many researches done in the
e-learning quality. Nevertheless, it is difficult to choose one model to
support the development of a b-learning environment, since we needed to
consider different perspectives and different levels of detail. At same time,
it is also important to consider pedagogical issues that influence the success
of any e-learning environment. As so, the main idea was to create a new reference
that summarize the main dimensions and criteria referred in the existent
quality frameworks and adds new pedagogical elements in the context of the
quality in a b-learning environment.

In this work we collected and arrange all
the quality criteria identified in the models referred above in order to get a
more complete framework and determine if it fits our b-learning environment
(blended learning in Portuguese higher education). We also included elements
and pedagogical issues related to our own b-learning research and experience in
Portuguese Higher Education, acquired during more than 10 years of experience.

A framework for measure the quality of a b-learning
environment

There is not a consensus grouping the
e-learning elements. The six models referred above present many ideas of
quality grouped in different categories. Some of them referred the same element
using a different label. Others grouped one or more elements in the same
categories. In one model we can find elements in one category that are divided
in different categories in another. Based on our own 10 years of experience and
on the most common groups, in this work we considered the following categories:
Institutional Aspects, Program and Course Design, Media Design, Technology and
Evaluation & Review. These categories included the elements identified in
each models studied as shown in the table below:

As a result of the analysis of the
selected frameworks we identified the main quality areas, each one with a set
of criteria, aiming the self-assessment of the b-learning courses as described
below.

Institutional Aspects

The institutional aspects are related to
the cultural organization and global elements that should be taken into account
when we are preparing a b-learning environment/product. In different ways, all
models studied refer institutional aspects. In general, present technologies
should use to be innovate the learning process and to face nowadays demand.

Education and Technology Research

E-learning strategy should be a part of
general educational strategy, should be embedded within teaching/learning
strategy of the institution and widely understood and integrated into the
overall strategies (EADTU, 2012). Faculty should provide incentives to
innovative practices, to encourage development of distance, including rewards
for effective teaching in distance learning (Merisotis & Phipps, 2000).
Internal and external publication on teaching and learning issues related to
e-learning should be encouraged and rewarded (EADTU, 2012). The institution
should encourage and support participation in inter-institutional collaboration
and exchange programmes related to teaching and learning development (EADTU,
2012). The institution should have an identified group of key staff responsible
for formulating, evaluating and developing institutional e-learning policies.
That policy should include the weights of blends and on the use of external
environments and resources such as social networks. The institution should
invest in the development of online assessment tools and techniques. It should
have evidence of research and development of online assessment and the
dissemination of these across the institution (EADTU, 2012). It is expected
that learning design choices will vary with the subject and level of the
course. Policies on research and scholarship in innovation in e-learning need
to cover both technical and educational aspects. It should include policy for
scheduling curriculum (face-to-face sessions, deadlines for assessments). It
should consider the needs of the target audience. Learners should know why the
physical attendance sometimes is required, for instructional reasons or
identity reasons. Institution should have an effective mechanism to share
knowledge and experience in the design of course content and consequent impact
on students’ learning. The policy in infrastructures should include: financial,
physical and technical resources; staffing and staff development; management,
responsibilities and accountability (EADTU, 2012). Institution should
investigate and explore emergent technology in the field of e-learning, should
provide a framework of technical accessibility and presentational standards that apply to e-learning materials and systems. It should include the
evaluation of the requirements needed, such as equipment purchase, software
implementation, recruitment of staff, training and research needs, staff
workload and technology developments. There should be an institutional plan for
the provision of training in the technical aspects of e-learning. Staff
development programs in online assessment should be provided. Information about
how to use the institution’s e-learning system and services should be provided
to all users in a logical, consistent and reliable way (EADTU, 2012). Process
which has been considered successful and effective should be shared among staff
and used to train new staff (SEEQUEL, 2004). Institution should provide
training and support to staff as well as example materials, good practices and
netiquette. Even on technical aspects in the course development (EADTU, 2012).
Faculty members should be assisted in the transition from classroom teaching to
distance instruction and be assessed in the process, including having training
sessions before and during the online classes. Guidelines regarding minimum
standards for course development, design and delivery should be provided. An
analysis of training needs of the staff should be conducted in a regular basis
(SEEQUEL, 2004). Documented technology plan should be in place to ensure
quality standards. Faculty members should have access to written resource to
deal with issues arising from students’ use of electronically-accessed data
(Merisotis & Phipps, 2000). Institution should offer a e-portfolio service
to assist students in recording evidence of their knowledge and skills
development (EADTU, 2012) and should also provide a e-repository (repository of
digital contents) (Peres & Pimenta, 2011). The institution should have a
process for indexing and archiving its e-learning materials for evaluation and
potential re-use (EADTU, 2012).

External providers

Medium and long term partnerships with
learning and training providers should be established (SEEQUEL, 2004). Any
provision delivered by two or more organizations should be covered by a written
agreement which clearly specifies the respective rights and division of responsibilities.
One of the organizations should be the leader and learners should know it
(ODLQC, 2005; EADTU, 2012; SEEQUEL, 2004). Adherence to all relevant legal
requirements, national or foreign should be ensured (SEEQUEL, 2004).
Institutions should be aware of the national policies regarding recognition of
qualifications in the country partners. Institutions should work closely with
professional bodies in the development of online professional communities
(EADTU, 2012).

Teams with peer review

The team must comprise content experts,
instructional designers and technical experts (Merisotis & Phipps, 2000).
e-Learning debates should be developed in order to collect and confront
different points of view (SEEQUEL, 2004). Key stockholders should be involved
in the program design (EFQUEL, 2011) and be consulted on a regular basis
(SEEQUEL, 2004). The course should be developed by a faculty team with a peer
review. The course must be approved through a broad peer review process
(academic and technical aspects) (Merisotis & Phipps, 2000; EADTU, 2012).

Course design, development and evaluation
should involve individual or team with expertise in both academic and technical
aspects (EADTU, 2012). People responsible for the analysis, design,
development, implementation and evaluation should be qualified (ODLQC, 2005)
(EFQUEL, 2011). Responsibilities of different staff groups (teachers, tutors,
etc.) involved should be specified and clear to learners. The roles of
individuals with the project team should be well defined (EADTU, 2012). All
tutors should have specific knowledge and competences to facilitate online
courses/programmes, to attest this, a set of predefined tutoring skills should
be used as a standard within the program (EFQUEL, 2011). A document of
responsibilities of each intervenient should be produced (expertise on the
subject, e-tutor, technical support, quality assurance manager, etc.) (EFQUEL,
2011; SEEQUEL, 2004). It should also include legal and ethical responsibilities
(EADTU, 2012) and the information on how people and services can help students’
succeed (EFQUEL, 2011). Procedures for staff profile should be established and
selection should be based on those competences. A system for recognizing the
staff’s competencies should be in place. It is important to have exchange of
agreements with other educational institutions for students’ virtual mobility
providing e-learning programs and operability (EADTU, 2012).

Learning outcomes

The learning outcomes should be agreed
between staff and learners (EFQUEL, 2011) (SEEQUEL, 2004) and written in the
students’ perspective, using an action verb. They should reflect the level of
performance students will achieve in a measurable form and should be related to
the course program (EFQUEL, 2011). Learning outcomes should reflect both
knowledge and skills to be developed (EADTU, 2012). The level of ability
inherent in the outcomes should be matched to a national level of qualification
(ODLQC, 2005; EADTU, 2012) and be placed in a wider educational, vocational
& professional context (ODLQC, 2005). The institution should have a clear
policy regarding the acquisition and assessment of core transferable skills,
including e-skills. Courses, including their intended learning outcomes, should
be regularly reviewed, updated and improved using feedback from stakeholders as
appropriate (SEEQUEL, 2004; Merisotis & Phipps, 2000; EADTU, 2012). Beyond
global learning outcomes it is also important consider the soft skills that
refer to a transversal objectives such as:

Promotional and administrative activities

E-learning provider should maintain and
demonstrate a strong commitment to educational value. It should adopt widely
accepted norms of good ethical practices (ODLQC, 2005).

The provider should conduct all promotion
activities in a fair and ethical manner, following the best practices and
legislation. All promotional materials should give clear and accurate
information. All enquiries from potential applicants should be handled promptly
and appropriately, avoiding mis-selling (ODLQC, 2005).

Institutional policies, services and
resources should be clearly stated. Students should be provided on how to
access them (QM, 2011). The institution should have a credit transfer policy
aligned with national system credit (EADTU, 2012). A system to recognize and
accredit the learners’ prior competences and knowledge should be in place. The
evaluation system should be able to measure to what extent informal learning
meets the expected objectives and outcomes. A system for allowing credit
transfer from informal to the formal settings should be foreseen. A process
which helps learners reflect on the transferability of their acquired
competences and skills to their specific context of use should be in place. The
vision of the quality of learning and value system should be shared and agreed
within the organization (SEEQUEL, 2004). Student interaction with faculty
should be facilitated through a variety of ways (Merisotis & Phipps, 2000)
even to deal with difficult situations (ODLQC, 2005). A structured system
should be in place to address students’ complaints (Merisotis & Phipps,
2000). It is important to have a documented process to manage complaints. A
person responsible for dealing with learners' complaints related to the
programme, assessment, tutoring should be identified (EFQUEL, 2011). Enrolment
when completed should be confirmed to the learner. A documented confirmation of
outcomes should be available where students finish the course (ODLQC, 2005).
The administrative impact of e-learning and b-learning systems on the workloads
of all staff groups should be assessed and adjustments made as required (EADTU,
2012).

Information available

The potential students should have all
information available (online, brochure, flyer, etc.) in order to orient their
decision (EFQUEL, 2011; ODLQC, 2005). They should be introduced to the purpose and structure of the
course (QM, 2011; ODLQC, 2005; EFQUEL, 2011). Students should have a clear understanding of all components and
structure of the course (SEEQUEL, 2004). Students should be provided with clear course objectives, learning
outcomes, concepts and ideas. Written information should be supplied to the
student about the program (Merisotis & Phipps,
2000; EFQUEL, 2011). Students should be provided with a
clear picture of what will be involved in using e-learning resources and the
expectations that will be placed on them (ODLQC, 2005; EADTU, 2012). Beyond this, it is also important to give
information related to technical requirements (minimum basic computer
configuration, operating system, internet access, plug-ins, particular
software, firewall access, etc.) (ODLQC, 2005; EFQUEL,
2011), pre-requisites of knowledge and competences
(including technical skills) (EFQUEL, 2011; EADTU,
2012; QM, 2011; ODLQC, 2005), target group, variety of
the methods, requirements of evaluation, timetable, workload, expectation about
students participation in community of practice related to social and academic
dimension. A document describing the characteristics of the target group should
be produced (EFQUEL, 2011).
Specific expectations about a minimum amount of time per week for study and
homework should be in place (Merisotis & Phipps,
2000). All conditions relevant for the course should be
explained to learners (ODLQC, 2005). It should be explained how the methodology approach (including
technology and blended approach) leads to the achievement of the learning
objectives (EADTU, 2012; EFQUEL, 2011). The contribution of e-learning components to the development of
educational objectives needs should be clear (EADTU,
2012). The information of the contact of the person
which responsible for the pedagogical and methodological issues, technical
issues, for the tutoring and for the complaints should be given (EFQUEL, 2011). Learners should be able to
discuss the suitability of the course with the responsible and know who will be
responsible for the evaluation (internal or external agent) (ODLQC, 2005). If applicable, it should also
refer the selection process and fees (EFQUEL, 2011), it should include the possible extensions to finish the course (ODLQC, 2005; EFQUEL, 2011). All information
should be consistent and reliable. It should be also easy to access, update,
coherent, consistent, etc. In addition a document to students with value system
of the institution and students’ role, rights and responsibilities should be
created.

Program and course design

Learning methods

Program and methodology should take into
account professional context, previous experience, prior learning and allow
self-directed learning. The course should allow the personalization of the
learning path (EFQUEL, 2011).

The blending of learning methods (online,
face-to-face, self-learning, tutor-facilitated, asynchronous, synchronous)
should be appropriate and meet the needs and characteristics of learners
(EADTU, 2012; EFQUEL, 2011). Should have a reasoned coherence between learning outcomes,
the strategy for use e-learning, the scope of the learning material and the
assessment method used (EADTU, 2012). A document which sets out the
relationship between learning outcomes, learning activities and assessment
should be produced. In a b-learning context there should be an explicit
rationale for the use of each component in the blend (EADTU, 2012). The
expectations on students regarding their participation in the on-line community
of learners and for a minimum amount of time per week to study and do homework
assignments should be clear both in general terms (for all applicants) and in
relation to specific parts of their course or programme (EADTU, 2012; Merisotis
& Phipps, 2000).

Learning objectives

Course’ learning outcomes should be
clearly defined and aligned with learning objectives (QM, 2011; California
State University (CSU) Chico, 2003). Each course should include a clear
statement of learning outcomes in respect of both knowledge and skills. It is
also important to ensure that all learning objectives are measurable and
written from the students’ perspective. The learning objectives should be
appropriately designed for the level of the course and students should have
instruction on how to meet them (QM, 2011) the results and outcomes of the
learning experience should be agreed between the staff and learners (SEEQUEL,
2004). The learning objectives specification process conducts the development
of an important guide to be used both by teacher and students. The use of a
taxonomy may facilitate the process such as Bloom Taxonomy (Peres &
Pimenta, 2009). Bloom (Bloom & Krathwohl, 1956) suggests a taxonomy of
learning objectives sorted out in six levels: Knowledge, Comprehension,
Application, Analysis, Synthesis and Evaluation. The intention is to display
the behaviours starting from the simplest to the more complex one.

Assignments should be clearly formulated
and adequately explained to learners. Learners should have a clear
understanding of what they are expected to perform and how their performance
will be measured.

It is also important to store and
organize evidence and records of the results achieved within the system (ODLQC,
2005; SEEQUEL, 2004). The confidentiality of their records should be respected
(ODLQC, 2005). The evaluation process should be used to improve the
teaching/learning process (Merisotis & Phipps, 2000) and the evaluation
system should be able to measure to what extent informal learning meets the
expected objectives and outcomes (SEEQUEL, 2004).

Curriculum

Another concern should be on the
curriculum design. The objectives of each module/unit should describe outcomes
that should be measurable and consistent with the course-level objectives (EADTU,
2012; QM, 2011). They should be designed in order to include components that
contribute to the development of outcomes. Students should be able to relate
course contents to skills and learning outcomes defined to the course (EADTU,
2012). According to Peres and Pimenta (2009) after defining objectives and
designing learning objectives assessment, we should establish the sequence of
contents. This organization avoids the specification of learning objectives
based on the contents. This scenario usually results in sentences such as
“understand the content A” and in a lowest level of knowledge (first or second
Bloom taxonomy level). Despite the importance of these levels, if the learning
objectives consist in achieving a higher critical thinking level, it is
important to explicit it on the objectives definition associated with analysis,
synthesis and evaluation.

The content of the program should be
organized in a logical sequence from the simpler to the most complex concepts
(ODLQC, 2005; EFQUEL, 2011). The modules/lessons/units should be built
progressively on each other (EFQUEL, 2011). The modules/segments should have a
varying lengths determined by the complexity of the learning outcomes
(Merisotis & Phipps, 2000). A written information about the program should
be supplied to the student (Merisotis & Phipps, 2000). The modules should
be introduced with their introductory elements such as: brief description,
objectives, estimated amount of time required, eventual assessment (EFQUEL,
2011). The self-contained modules should be used to assess student mastery
before moving forward in the course (Merisotis & Phipps, 2000). The
curriculum should include research modules taking into account the skills and
independence that will be demanded of students in conducting research remotely
(EADTU, 2012).

The curricula should be designed in such
a way that allows personalization for individual learning styles and needs and
a flexible path for the learner (SEEQUEL, 2004; Merisotis & Phipps, 2000).

Learning influence factors (motivation)

Before starting the program, students
should be advised about the program to determine if they have the
self-motivation and commitment to learn at a distance. Assessment instrument
should be used in order to determine the students’ learning styles (Merisotis
& Phipps, 2000) and motivation. The learning methodologies should motivate
learners to actively participate in the learning process (EFQUEL, 2011; SEEQUEL, 2004). Should valorize the learner’s self-esteem and competences among
the learning community (SEEQUEL, 2004). The learning methods should take into
account the balance between time to develop activities and complexity (Peres
& Pimenta, 2011). One way to motivate students is to give class voice-mail
and/or e-mail to encourage students to work with each other and their
instructor(s) (Merisotis & Phipps, 2000).

Preparatory classes, additional learning
materials, recommended reading as well as pedagogical guidance and other forms
of support should be available during the course in order to bridge learning
deficits (EFQUEL, 2011). In general the following main learning influence
features could be identified: contextual analysis results; time definition;
e-learning strategy complexity; nature of subject in study; personality, ways
and individual learning styles; previous experiences, knowledge and culture
(Peres & Pimenta, 2009).

Learning activities

The workload demanded by the course
should be realistic regarding the objectives, curriculum and according to the
characteristics of target group, including full time job occupation (EFQUEL,
2011). The design of an instructional strategy should conduct to the learning
success, individual or in group according to pedagogical models. Many
pedagogical views may be used to support the instruction planning. The
selection doesn’t have to be exclusive, it is possible to use more than one
pedagogical approach. The choice of the pedagogical model should consider the
moment of learning. At the beginning of the subject study, it is important to
make sure that students are getting the basic knowledge (behaviourist and
cognitivist theory). Then, it is important to consolidate it and promote the
self-learning based on previous experiences (constructivist theory). At the
end, it is important to promote a deep learning by social interaction (social
constructivist theory). This learning path should be aligned with learning
objectives (Peres & Pimenta, 2009).

At beginning of learning activities, it
is important to provide the self-introduction by the instructor and students
(QM, 2011). A learner centred learning design should facilitate the development
of the desired skills and expected competencies described in the learning
objectives (EFQUEL, 2011; California State University (CSU) Chico, 2003; EADTU,
2012). Students should have clear instructions on how to start and where to
find various information of course components (QM, 2011). Opportunities for
online publications and peer review should be provided (EADTU, 2012).

Learning activities should also provide
opportunities for interaction in order to support active learning (QM, 2011).
Social and collaborative activities should be included in the program
methodologies and contribute to the achievement of the learning objectives, it
includes, for example, peer review, group work, discussion board, (EFQUEL,
2011) or problem-solving group activities (Merisotis & Phipps, 2000). At
any time, student interaction with other students should be facilitated through
a variety of ways (Merisotis & Phipps, 2000). Learning activities should
enable participation in academic community and contact with external
professionals. The institution should provide mechanisms for students to
participate in active communities of professional practices in order to
stimulate a critical attitude (EADTU, 2012). The requirement for student
interaction should be clearly articulated. Institutions should provide an
online community for student-student and student-teacher interaction and make
their policies available (EADTU, 2012). Etiquette expectation for online
discussion, email and other forms of communication should be clearly stated
(QM, 2011). To support communities of learners the activities should be
designed in order to offer an appropriate use of asynchronous tools (e.g.
discussion forums, wikis, blogs, social networking sites) and synchronous tools
(e.g. video-conferencing, real-time chat) (EADTU, 2012). In spite of the
objectives defined, higher education courses should provide multiple activities
that help students to develop critical thinking, problem-solving skills
(California State University (CSU) Chico, 2003) analysis, synthesis, and
evaluation (Merisotis & Phipps, 2000). The most important is to develop
learning activities in order to promote the achievement of the stated learning
objectives (QM, 2011; Peres & Pimenta, 2011) and to be relevant to
professional practice, including case studies, practical examples, good
practices and real-life examples (EFQUEL, 2011).

Courses, and learning activities, should
be designed with a consistent structure, easily understandable for students
from various learning styles (Merisotis & Phipps, 2000) and offer multiple
visual, textual, kinaesthetic and/or auditory activities to enhance student
learning and accessibility (California State University (CSU) Chico, 2003). The
description of learning activity should include the objectives, pedagogical
models, subject/community, title and general description, tools, e-contents,
activity phases, division of labour, rules and results (Peres & Pimenta,
2011).

The learning experience should be built
in a flexible manner so as to ensure its contextualization and relevance to the
learner’s context (EADTU, 2012; SEEQUEL, 2004). The flexibility should be also
in terms of time, place and pace (EADTU, 2012). The instruction path should be
students’ centred, trying to promote an inclusive environment that explores the
student’s differences and the openness of the present digital network.

The course and learning activities
instructions should be linked to a description of the technical support and
institutional accessibility polices and services (QM, 2011).

Learning process and eTutoring

The course should offer ample
opportunities for interaction and communication: student to student, student to
instructor and student to content (California State University (CSU) Chico,
2003). The tutor should maintain and demonstrate a clear commitment to help
learners achieve their educational goals. Nevertheless, learners should be
responsible for their own learning and be informed of the tutor support.
Students should be encouraged to complete their courses (ODLQC, 2005). Access
to tutors should be provided on a regular and sufficient basis, known to both
tutors and students. Tutor should be able to use a variety means to interact
with learners (e-mail, forum, VLE tools, etc.) (EADTU, 2012). A plan to support
the interaction and collaboration within the learners community (SEEQUEL, 2004)
and with other students (Merisotis & Phipps, 2000) should be in place.

During the learning process, the tutor should
track and monitor the behaviour of the users (SEEQUEL, 2004) and provide
guidance and accompany to the learners. Tutoring should offer opportunities for
learners to determine their own learning pace. Despite the deadlines that must
be established, learners should be able to control their own path throughout
the program. Tutors should provide learners with timely expert advice on course
issues or materials and feedback on assignments (EADTU, 2012). Learners should
be also informed of the timeframe (EFQUEL, 2011), the certain period of time to
grade and return all assignments (Merisotis & Phipps, 2000; QM, 2011).
During the course development the tutor should provide timely, accurate,
helpful feedback to learners on tasks/activities (Merisotis & Phipps, 2000; EFQUEL, 2011), non-threatening and appropriated to the level of the course
(Merisotis & Phipps, 2000; ODLQC, 2005), using measures for monitoring
(SEEQUEL, 2004) and the various learning styles of students should be
considered (Merisotis & Phipps, 2000), whenever possible in a personal
basis (ODLQC, 2005).

Through the learning activities learners
should be encouraged to consider and use higher thinking skills and to view
issues from different perspectives (EFQUEL, 2011).

During the learning process, students
should be instructed about the proper methods of effective research, including
assessment of resource validity and to help them use electronically accessed
data (Merisotis & Phipps, 2000).

When there are one or more tutors, steps
should be taken to ensure that tutor support is consistent (ODLQC, 2005).
Directions should be provided as to how students can participate in a broader
academic community (EADTU, 2012). Students should have access to support
services including technical help desk, administrative support and course
choice advice (EADTU, 2012).

Learning materials/resources

Regarding to materials, it is important
to guarantee that each learning unit is supported on the elements needed to
guide learners in achieving the learning objectives (EADTU, 2012; QM, 2011; EFQUEL, 2011). The purpose of instructional materials and how they should be
used for learning activities should be clearly explained (QM, 2011). They
should be structured to facilitate individual study and the development of
study skills (ODLQC, 2005). They should be current, present a variety of
perspectives on the course contents (QM, 2011), appropriated to the learners’
need, knowledge, and experience (ODLQC, 2005; EFQUEL, 2011). Contents should be
relevant and clearly presented, build on and reinforce prerequisites concepts
and skills. Introduce, assess and reinforce new concepts and skills, should be
logically structured and sequenced (EADTU, 2012). The contents should be
sufficient and not in excess. They should be challenged, centred on the
student, be relevant for students’ life, allow the interaction, tell a story
with emotion. Little pieces of content are better than a more extensive one.
The language should be simple and include visual elements. Any noise should be
deleted. It is important to capture students’ attention in the beginning,
trying to evolve with contents.

Resources should be adequately balanced
regarding cognitive load and presented in a sub-divided form, in a logical
sequence, without extra information. The course material should promote
collaboration among students (Merisotis & Phipps, 2000; EADTU, 2012) and
students-to-content. Independent learning materials should provide learners
with regular feedback through self-assessment activities or tests. The availability,
function and purpose of independent learning materials should be clearly
defined and communicated to students. Self-paced materials should incorporate
extensive embedded testing of learning outcomes (EADTU, 2012).

It is also important to ensure that contents
are provided in a flexible manner, allowing different learning paths (EFQUEL,
2011), customization and personalization to individual learning styles
(SEEQUEL, 2004). Media rich contents should be utilized with specific purpose
(EFQUEL, 2011). This can include videos, tutorials, interviews with
specialists, scenario based learning, games, etc.

Sufficient library resources should be
available to the students (Merisotis & Phipps, 2000). At the same time,
students should be provided with hands-on training and information to help them
in seeking material through electronic databases, interlibrary loans,
government archives, news services, etc.

Concerning the modules/lessons/units it
is important to provide a glossary of terms associated to the learning materials
and available from any part of the course (Merisotis & Phipps, 2000). A
distinction between compulsory and recommended study/reading materials should
be made (QM, 2011). The bibliography should be commented.

The materials should be effective and not
contain significant errors of facts, misleading or out-of-date information,
concepts and approaches (ODLQC, 2005). Contents should be inclusive, respect
cultural diversity and gender sensitive (EFQUEL, 2011). It is also important to
produce a lesson plan, including the learning objectives, reference to the
contents and learning activities (face-to-face or online) (Peres & Pimenta,
2011).

Media Design

Accessibility

Accessibility standards should be
considered in the design of the course. The course should contain equivalent
alternatives to auditory and visual contents. Course should employ accessible
technologies and provide guidance on how to obtain accommodation (QM, 2011)
(EFQUEL, 2011). The course design should accommodate the use of assistive technology
that can help students with disability (QM, 2011; EADTU, 2012) e.g. with
respect to keyboards, touch screens, screen-readers (and “talking books”) or
speech recognition. Learning material should be accessible and usable via a
variety of devices including mobile devices.

Usability

The course design should facilitate
readability and minimize distractions (QM, 2011). The layout of the course
should be clear and free of unnecessary elements. The size and type of font
utilized should be comfortable for reading. The images, illustrations, tables
and other visual elements should be easy to read (EFQUEL, 2011). Course should
be designed with a consistent structure easily discernible to students from
various learning styles (Merisotis & Phipps, 2000). Interfaces used in the
technical design of course should conform to up-to-date usability and
accessibility standards. Font, text, placement and presentations should be
consistent (EADTU, 2012). All interfaces should be functional and attracted to
students.

Navigation

The course should be well-organized and
easy to navigate, the aesthetic design should present and communicate the
course information clearly throughout the course. All web pages should be
visually and functionally consistent throughout the course (SEEQUEL, 2004).

The navigation through the mandatory
learning materials should allow learners to know about their progress and
position in relation to the overall content, identifying the unit, module,
lesson, part of a unit, etc. (EFQUEL, 2011). Navigating throughout the online
course should be intuitive, consistent and easy to navigate.

Printable

All screens, tables of contents and
learning materials, including additional sources should have a printable
version (EFQUEL, 2011).

Cultural diversity

Materials should be neutral as to sex,
ethnicity, age and related issues (EFQUEL, 2011).

Copyright

All images, graphics, illustrations
should be copyright free (EFQUEL, 2011). All resources and materials used in
the course should be appropriately cited (SEEQUEL, 2004; QM, 2011). Learning
materials should comply with legal requirements, copyright issues should be
identified and documented (EADTU, 2012).

Download

The materials available for download
should take into account reasonable standards of time for download, regular formats
and forms of compression (EADTU, 2012; EFQUEL, 2011).

Technology

Server and Applications

The technical infrastructure that
maintains the e-learning system should fit the purpose and support both
academic and administrative functions. Technical infrastructure should be well
defined and support institutional e-learning objectives. Institution should set
standards for the operation of its technical infrastructure that are
benchmarked against other major online customer service providers (EADTU, 2012).

The tools (learning management system and
other tools) used in learning strategies and collaborative learning should be
in accordance with the information technology infrastructure available, with
the target group equipment and connectivity, learning skills and needs, staff
teaching skills, learning objectives, assignments and other activities (EFQUEL,
2011). The selection of the tools should be based on the learning outcomes and
objectives (QM, 2011) (Merisotis & Phipps, 2000). A system for learners profiling
should be in place. The diversity and identity of each individual learner
should be guaranteed (SEEQUEL, 2004).

The course technology should be current
and students should readily access the technologies required in the course (QM,
2011). Search functions should be available for forums discussions, blogs, etc.
whenever such tools may be utilized (EFQUEL, 2011).

Security and performance

Appropriated operating and security
standards for all aspects of provision of online services should be defined.
Measures should be in place for system recovery in the event of failure or
breakdown (EADTU, 2012).

Electronic security measures, such as
backup procedures, should be in place to ensure the integrity and validity of
information (Merisotis & Phipps, 2000; SEEQUEL, 2004). The virtual learning
environment should runs on an adequate server, which guarantees its stability.
The course should be tested on various browsers and operating systems before
launching (EFQUEL, 2011). Learning management system should be integrated with
the management information and administrative system (EADTU, 2012).

Information such as system recovery, key
performance indicator of system availability, download time, queuing time for
access, etc. should be provided. Monitoring the patterns of the use of the
system by students and staff should be a source of information for improvement
in pedagogical as well as technical issues (EFQUEL, 2011).

Support

Support for building and maintaining the
distance education infrastructure should be addressed by a centralized system.
Easily accessible technical assistance should be available to all students.
Technical assistance in course development should be available (Merisotis &
Phipps, 2000). Students should be confidante while using virtual learning environment,
even people with disability (EFQUEL, 2011).

Evaluation & Review

Periodically review

A feedback procedure for assessing
effectiveness should be implemented (SEEQUEL, 2004). The provider should be
committed to the continuous improvement. A process for integrating the
recommendations for improvement should be foreseen as a part of the programme
(EFQUEL, 2011). Procedures to ensure the quality, effectiveness and relevance
of the material should be developed on a regular basis. It is also important to
evaluate the accessibility and effectiveness of the resources’ usage.
Procedures in order to assess the effectiveness and relevance of the learning
materials should be developed (SEEQUEL, 2004). Course materials, including the
intended learning outcomes should be regularly reviewed, updated and improved
using feedbacks from stakeholders as appropriate (Merisotis & Phipps, 2000; EADTU, 2012) to ensure clarity, utility and appropriateness (Merisotis &
Phipps, 2000).

The program’s educational effectiveness
should be measured using several methods. The results should be used to improve
the teaching/learning process. Specific standards should be in place to compare
and improve learning outcomes. Data on enrolment, cost, and
successful/innovative uses of technologies should be used to evaluate the
program effectiveness (Merisotis & Phipps, 2000).

The performance of the e-learning systems
should be monitored and opportunities for performance improvement identified.
Performance of mentors, tutors and moderators should be monitored regularly.
Problems and issues should be acted upon promptly. Longer term improvements
should be identified (EADTU, 2012). On the formative evaluation we should
answer questions such as “do the activities cover all learning objectives?”; “are
there activities that are not covering any objectives?”; “do students have all
the necessary information?” etc. (Peres & Pimenta, 2011).

Collected data

A questionnaire developed specifically
for the program should be used in order to assess overall quality and
appropriateness of:

Course media and technical support (registration
process, access and user friendliness of learning in the platform, download
time) (EFQUEL, 2011).

Data on pattern of students use on
e-learning environment may be gathered and analyzed, ‘learning analytics’, in
addition to evaluations information’s from survey activity (EADTU, 2012).

Final Report

To evaluate the quality and overall
coherence of the course in order to develop further improvement, it is
important to get learners’ feedback that could be collected through
questionnaires or other means. An evaluation report should be prepared, in
which feedback is analyzed and converted into clear recommendation for
improvements in future programme design. The final report should include the
learners’ course evaluation. The report should include the following topics:

Evidences and results achieved should be
organized and stored within the system (SEEQUEL, 2004). An evaluation of
cost-benefits should be made.

Conclusions and Future Work

More and more institutions are offering
courses at distance. This might be a good solution to overcome some
difficulties (e.g. time and distance) but it is necessary to promote a
reflection about the quality of these practices as well as of the b-learning
product offered. Moreover, the success of a course also depends on the
expectations of students and in the ability of the educational institution to
meet them. The information, concepts and procedures here presented give support
to teachers and instructors, which intend to validate the quality of their blended
learning courses. The framework developed helped to identify the areas to be
analyzed and reflected upon. As a result of this research, the following
elements arose in the analysis of the quality of a blended learning
environment: Institutional Aspects (education and technology research, external
providers, teams with peer review, learning outcomes, promotional and
administrative activities, information available), Program and Course Design
(learning methods, learning objectives, assessment & test, curriculum,
learning influence factor, learning activities, learning process and
e-tutoring, learning materials/resources), Media Design (accessibility,
usability, navigation, printable, cultural diversity, copyright, download),
Technology (server and applications, security and performance, support),
Evaluation & Review (periodically review, collected data, final report).

In the future, in order to get more
accurate results, we will propose an evaluation methodology based on a relative
scale to weigh main areas and associated criteria, a [0..1] scale, where 0
means that area is not important for the research and 1 means of maximum
importance. The total weight of all main areas must be 1. Similarly, the total
weight of all the criteria of a main area must be 1. Each criterion has its own
score for evaluating the quality level also in a [0..1] scale. The evaluation
is organized as a tree-like structure of quality characteristics and the
weights are determined using pairwise comparisons, adapted from Analytic Hierarchy
Process (AHP) approach (Saaty, 2008). Unlike the way of assigning a number from
a fixed scale with an arbitrary unit to the weights of each area/criterion, the
measurements are not fixed but depend on each other and on the context of the
course and its objectives. We think this methodology will facilitate future
benchmarking within others courses. The final quality index Q is obtained by
the bottom-up iterative aggregation of the scores as in AHP. First the score
for each main area Ai is calculated as the sum of the products (Si * Wci),
where Si is the score for criterion i and Wci is the corresponding weight. The
final quality index Q is calculated as the sum of (Ai * Wai), where Wai is the
weight for main area Ai.