A Framework for Evaluation: Including the Student who Discontinues

Introduction

With the burgeoning use of the Internet to facilitate on-line learning, it
is timely to examine past and current theory and practice in evaluation of on-line
and distance education (Calvert, et al., 1991; Robson, 1997). In doing
so, the role of the significant number of students who discontinue emerges as
a factor (Peters, 1992). Discontinuation has long been noted in face-to-face
education and is even more noticeable in distance education - between 31% and
76% in some cases (Malley, 1976, 171: Castles, 1992, 39: Commonwealth of Australia,
2001). The factors identified as related to this (Long, 1994) give little reason
to suspect that on-line learning is different.

The problem

In the past there has been much theoretical and practical work done on the
classification of such students and the reasons for their discontinuation (Calvert,
1995; Long, 1994). Robson's study (1996) of students learning mathematics through
teleconferencing, raised the question of the effect of those students who left
on the results which are used to judge the effectiveness of the course, and
the effect of discontinuing students on course evaluations in general, highlighting
the need to extend the current evaluation theories to embrace this aspect. Discontinuation
is an action which results from evaluation of the situation by a central participant,
but most evaluation models for distance education do not specifically involve
these students. This may be because of difficulties in data collection or because
of a philosophical stance which discounts the relevance of their perceptions.

In considering students who discontinue, Durkheim's suicide theory (1952) and
Spady's model of influences on discontinuation (1971), determine the factors
influencing such student evaluation and subsequent action. Durkheim's strong
emphasis on the importance of integration of the individual with society confirms
the need to examine the educational environment provided by an institution,
the types of students in the institution and the integration between these,
together with the integration between the educational environment and the course
goals. The timing of the student input also has relevance here so that all students
can make a contribution to the discourse (Owen & Rogers, 1999, 96; Cheung,
1998, 26).

The problem then is to use a framework for evaluation of a contemporary on-line
course which acknowledges the past and present practices and theories, but which
addresses the current issue of discontinuing students. Immanent critique reveals
that discontinuing students are not considered in most evaluations purporting
to include and represent the student body, while ideology critique reveals the
distortions caused by ignoring these students.

A framework accommodating discontinuation

Theorists, and the schools of thought they represent, have much to offer to
the problem of evaluating a distance education course and considering these
discontinuing students. Their impact must be considered in relation to the integration
of students in a course, with the length and timing included as factors. An
evaluation needs to consider these issues in relation to the dominant paradigms
to be relevant in the twenty-first century on-line education scenario.

Theory and the development of a framework

In developing a framework for resource-based learning, it must articulate a
shared point of view to support and inform planning and action (NBEET, 1997),
'to provide the necessary structure to cope with the inherent complexity of
human behavior' (Garrison, 1988, 209). A theoretical framework cannot tell practitioners
what to do in a particular circumstance, but can be modified to suit the circumstances
with the criteria for deciding relevance being a matter of deliberation, interpretation
and interaction between the parties (Posavac, 1997, 125). House argues that
the evaluator should act like a human being, 'as a critical intellect, less
on technique' and 'the evaluator must get close to the course, experience it
and convey that experience in its totality' (1973, 23). This is leaning towards
the Interpretive or Social Critical paradigms with just a touch of Post-Structuralism.
Generally educators have moved from the rigidity and sometimes inappropriateness
of a Scientific methodology towards the Interpretive approach to evaluation,
then to a Social Critical approach but all of these approaches are used still.
Evidence of evaluators' philosophies are plentiful in the literature. For example,
Rossi, Freeman and Wright's definition of course evaluation (Rossi, Freeman
& Wright, 1979, 2) reveals a non-Scientific approach with the charter to
improve social conditions, so leaning towards the Social Critical Theory. An
evaluation framework must accommodate this flexibility.

The framework for evaluation of an on-line or distance education project developed
by Robson (1999) takes into consideration the Spady (1971) and Tinto models
(1975) of factors influencing discontinuation and is therefore pertinent in
this case. Considering the importance of this group in an evaluation, the framework
uses and extends their models to cater for distance education. Rather than being
prescriptive, the framework highlights areas that should be explored in order
to better understand a situation. It shows evaluation generally, so that it
has the flexibility to be individualised to a specific situation, yet illustrate
the permanent inter-relationships between the various factors. It is in harmony
with the dominant evaluation methodologies and styles currently adopted in on-line
and distance education. It is designed to facilitate input from discontinuing
students and to be undertaken in a time frame which is realistic when rapidly
changing technologies form part of the program being investigated.

Consideration of the works of Durkheim (1952) on suicide, Habermas (1971) on
evaluation paradigm choice, Foucault (1980) and Lyotard (1994) on individuality
of contexts, and to a lesser extent Spady (1971) on discontinuation, then leads
to the sub-sections of a framework for evaluation. In analysing a situation
using Post-Structuralism, the course is dissolved into parts in order to understand
it. Thus, a framework gives an indication of important sub-sections that could
aid de-construction, so the extent of individualisation of an evaluation in
the framework becomes of interest in a Post-Structuralist approach, even though
this approach would normally reject any prescribed structure.

It is argued that the students who discontinue are such a vital element in
any course that the areas of concern in their case, are causes for concern to
the course as a whole. Therefore the Scientific Durkheim and Spady theories
become an important part of an evaluation in that the degree of integration
of the student goals and the educational environment of an institution is an
indication of the nature of a course. So, too, is the harmony between an institution's
goals for a course and the environment in which it operates. Technology and
the human-computer interface, which are essential to contemporary distance education
(Inglis, 1999), are factors in both social and academic integration. These aspects
of Durkheim's and Spady's theories provide a theoretical platform for the inclusion
of both continuing and discontinuing students in determining the extent of the
integration. Consequently, an evaluation incorporating consideration of the
integration of student and course goals with the educational environment, better
reveals the situation.

Relevance of a framework

Although a framework, as an heuristic device, cannot capture all the complexities
of practice, it can act as an organiser of options. The theoretical framework
for evaluation of on-line education within higher education used here, incorporates
many of the aspects of evaluations previously developed and implemented. It
encompasses fundamental theory on evaluation, individualisation and change.
But, rather than being prescriptive, it highlights areas to be considered or
explored in order to better understand and reveal a situation. It shows evaluation
generally, so that it will have the flexibility to be individualised to a specific
situation, yet illustrate the permanent interrelationships between the various
components of educational evaluation where, 'the field of distance education
encourages multiple perspectives, different epistemologies and thus different
ways of knowing' (Jakupec & Kirkpatrick, 1997, 199). It is in harmony with
the dominant evaluation methodologies currently used, and incorporates most
of the attributes of the models and styles currently employed. Importantly,
it caters for consideration of the discontinuing students.

Figure 1: A framework for evaluation of a distance education
unit or project.

Figure 1 illustrates how the evaluation rationale will determine the methodology
used in an evaluation and that this methodology, together with the course or
project goals, the educational environment, the student goals and the matches
between these will affect the judgement or understandings resulting from the
evaluation. The resultant action will inform the course goals, educational environment
and student goals making it an on-going, cyclical phenomenon.

The question of the discontinuing students affects how much is to be evaluated.
If their input is seen as an essential part of the evaluation, then as much
information as possible must be gleaned from them before they 'disappear'. This
is especially true when it can be seen from the literature that predicting whether
a student persists or not, is a complex task. This implies that the evaluation
must be thought through before students enrol, so that all students have the
opportunity for appropriate input before discontinuation starts. Obviously,
there will still be more opportunity for input from the students who continue.
This means that the timing of the evaluation should include the time interval
from the enrolment of students to the withdrawal of the first student, or even
the completion of the first assessment task. It is important to decide on the
evaluation methods early on in the evaluation, so that the evaluators can maximise
the use of their time during that interval. Therefore, in deciding how much
to evaluate, the evaluator sets the boundaries for the exercise. Habermas (1975,
57), Herman and Mandell (1999, 31) hold that everyone with an interest to participate
in a communicative discourse should do so, and consideration is normally given
to this when developing an evaluation rationale.

Course goals

There is value in an evaluation in examining the rationale for developing and
implementing the course by the institution or others in the first place, and
its guiding educational philosophy. Whether a course aims for massification,
individualisation, or globalisation is of interest here. So too, are any policies
on incorporating open learning or flexible learning into the course. The target
audience should be identified and the acceptable discontinuation rate determined
at this point. Even if the course goals cannot be determined precisely (Posavac,
1997, 131), there is value in considering them, because a course does not exist
in a vacuum, but is a response to a variety of influences, including that of
need, in the given context. Evaluation may already be part of the course plan
and an instrument in achieving the goals (Owen & Rogers, 1999, 29). Generally,
evaluators would assume that improving student learning would be part of the
course goals (Lewis, 1989, 9; Morgan, 1993, 86), and these espoused aims can
be compared with reality. For instance, the evaluators can investigate the course
rationale to determine whether, as part of a university, its role is to transform
students to be instruments of social change, an extension of knowledge, or tools
of economic growth (Paul, 1999, 31) because curriculum assumptions are usually
based on some educational philosophy, sociology and psychology (Jakupec, 1998,
111). For example, '[w]hen students become 'learners' changes follow in both
the control and context of curricula' (Posavac & Carey, 1997, 26).

The learning theory, on which a course is based, and its enactment, will influence
these course goals. As well, the nature of any technology used can be relevant.
Also, when the course goals are considered in the light of the institutional
goals, the relationship between the institution and the outside world becomes
an issue. However, as technology in the workplace where the students aim to
go is changing, the skills they need to learn at university are changing too,
so the new 'entrepreneurial' form of higher education impacts on the course
goals and community context (Garrick, 2000, 244). Failure of an institution
to adjust to this, is like Durkheim's anomic suicide.

Community context

Most institutions are set physically within a community, which may or may not
include the on-line education students. The hopes, goals and expectations of
the community will influence the academic system of the institution and the
educational environment that students will encounter. The community itself lies
in contemporary society with its changing interests in globalisation, justice
and technology, but cost is a dominant factor too, in distance education today.
Power is closely aligned with this and so resources available in the development
and presentation of a course exposes underlying power structures (Foucault,
1980, 89) especially where technology is involved. Financial links between the
community and the institution in the form of government support or control,
also influences the strength of the community commitment, so cost becomes an
important factor and a realistic control on the implementation of theory in
a community. It addresses human, technological and institutional constraints
and possibilities. Therefore, the community context within which the evaluation
takes place must be acknowledged if the findings are to be valid and generalisable
(Jakupec, 1998).

Academic system

Institutions within contemporary higher education have profiles that emphasise
the changing academic, administrative and technological cultures. This background
affects the relationship between the learning theories espoused by an institution,
and the teaching strategies used to educate the student, within the academic
system. The current merging of distance education, open learning and flexible
learning (Tait & Mills, 1999) and the accent on life-long learning (Candy,
1991) can also influence the model used to construct and guide the on-line education
in an institution. Therefore, content, materials, support and design, become
important elements of the academic system of an institution, influencing the
administrative structures and presentation standards (Cheung, 1998, 28). Gibson
and Hatherell (1997, 133) talk of this as the 'tone' of a university. But, technology
changes rapidly. Therefore, as on-line education is strongly dependent on the
technologies used, and as student satisfaction is also dependent on the technologies,
then the maximum time for an evaluation may be limited by the lifetime of the
particular technologies being used. Consideration of the large amount of work
on human-computer interaction (HCI) and the emerging work on teaching and learning
with technology, is ensured by incorporating these issues into the examination
of the academic system and educational environment supplied by the institution.

These institutional policies and procedures, together with student and teacher
attitudes towards on-line courses, play a part in supporting the learning environment.
Therefore, policies on aspects, such as interaction, assessment procedures and
individualisation of teaching have a bearing on the learning process. The strength
of negotiation is another facet of the system. This can be seen where students
and the teacher explain and justify their ideas to others, reflecting critically
on other's ideas, together with some flexibility of time, place and pace and
choice of curriculum. Identification of the power structures within the academic
system is of interest to evaluators here.

The generalised framework in Figure 1 aids in the consideration of HCI and
technology which incorporates the psychosocial characteristics of learning environments
together with the effect of the computer interface on the students' learning,
influences the academic system and consequently the educational environment.
Because technology changes rapidly, hasty turnover can affect student satisfaction.
This will in turn influence the student's goals and how the student reacts to
the course and the technology. Therefore, there can be value in an evaluator
identifying both changes in technology and the forces behind these changes to
reveal the basis of power in the situation. In summary the academic system or
'learning milieu' (Morgan, 1993, 95) influences the overall educational environment
of a course. Inclusion of the academic system in the framework ensures consideration
of this factor during an evaluation.

Social system

Durkheim demonstrated the influences of the society within which the students
and course operate (Owen & Rogers, 1999, 23). The social system (social
context, roles and relationships, human performance and well-being, equipment,
and the physical or virtual social environment), which an institution facilitates,
influences the students' persistence as well as their learning, and in turn
generally effects the resultant educational environment (Spady, 1970; Freitag
& Sullivan, 1995; Thorpe, 1995, 175). As shown by the investigations of
discontinuing students (Robson, 1997), informal and formal peer group interactions
(Lyall & McNamara, 2000, 118), informal and formal staff-student interactions
and external interactions all have a place in developing the social system of
a course.

Higher education social systems are shaped by the changing times (Rossi, et
al., 1999, 15) and there will be social systems which are independent of the
institution which have a bearing on the students. These various social areas
can differ in nature and goals from that being evaluated (Rossi, et al., 1999,
5) but generally the institutional social system will have the dominant effect
on the resultant educational environment, but this is not necessarily always
so.

Students, especially younger students, need socialisation (Wilson, 1997, 281)
and technology may or may not be used to facilitate this social interaction
between students and between the teacher and students (Nunan, 1990, 3). Papert
noted that the 'context for human development is always a culture'. The social
system a vital part of an educational environment and consequently of its evaluation.

Educational environment

The academic and social systems together form the educational environment (Chambers,
1995, 349) or 'organisational milieu' set within a community context in which
the teaching, learning and interaction take place (Morgan, 1993, 95). The teaching
strategies used to educate the student via various technologies, arise out of
the academic system to form the educational environment. Results from the examination
of this influences any judgements and understanding resulting from an evaluation.

Institution/community commitment

Following Durkheim, the degree of integration between the course goals, the
academic system, the social system, community context and the educational environment
affects the institutional and the community commitment to an on-line course.
Habermas argues that social integration is possible through understanding life-worlds
but that as system integration does not involve life-worlds, integration becomes
impossible (Stevens, 1995, 38). Durkheim however, has argued strongly that it
is not only possible, but necessary and in following Durkheim's arguments, the
importance of integration is shown by the serrated edges of the areas in Figure
1. The resultant commitment to a course influences the judgement and understanding
of the course during an evaluation. An arrow connecting the serrated edges to
the judgement and understanding illustrates this.

Student background

Durkheim and Spady have demonstrated that a student's background will affect
that student's attitudes, goals or needs. A basic pedagogic principle is to
find out about the students' social and educational backgrounds, that is, socio-cultural
information (Chambers, 1995, 347) in order to move forward. Prior mental contents
and external inputs such as teaching, form this background (Young & Marks-Maran,
1999, 178). So, investigation of previous experiences informs an evaluation
(Powell, et al., 1999, 98). Assessment procedures also play a part, as
a student's perceptions of assessment are influenced by that student's previous
experiences. In fact, most evaluation approaches see students influenced by
their societal context. So, the attitudes towards study and technology are formed
in the student's background and in turn influence how the student reacts to
the course and the technology and how the student forms goals. An arrow in Figure
1 connecting the student background and goals illustrates this. So students'
experiences, fears, prejudices and perceptions of using technologies in the
course, form an insightful part of the evaluation of attitudes towards study
and technology (Chambers, 1995, 349). For instance, technophobia detection is
important, both in the conduct of the course and in the evaluation. It is useful
to ascertain such student background variables early as they become important
data if a student discontinues. Because this act flags student evaluation of
the course, factors leading to that decision and subsequent action, have a central
place in the analysis.

Evans (1994, 20; 1999, 222) notes the importance of researching the diversity
of student backgrounds in order to accommodate them in appropriate educational
courses as knowledge of a student's intellectual background is the usual starting
point for helping students learn. Certain variables have been shown to affect
the discontinuation rate and as part of the participant cohort, then a study
of these variables would normally be of interest in an evaluation. However,
all the variables influencing students to discontinue cannot be addressed when
considering discontinuation rates in the evaluation process, but an appropriate
selection can illuminate the situation.

Student goals/needs

Generally students want to be informed about the field of study they have chosen
and to reason about the area, as part of the course they are undertaking. Nevertheless,
there has been a shift in values and self-interest of both students and community
members, over the past few decades, influencing students, courses and their
evaluations (Rossi et al., 1999, 19). However, improving their learning would
normally be among the student goals (Morgan, 1993, 86). A learner-centred approach
sees determination of the student goals and needs as pivotal. But, if a course
or evaluation is individualised, then there will be multiple needs and goals
to be answered. The task is simpler when a common set of student goals is determined
by the institution, the students or the evaluator. Technology may well be the
tool which enables these goals to be achieved, but if an educational environment
is not tailored to these individual goals, then any mismatch with the culture,
say through globalisation, will be detected when compared to the student goals.
For example, Gooler (1979, 191) uses retention of students as an indicator of
how well personal goals are met. Therefore, when a student discontinues, having
pre-determined the student's goals sheds light on the degree of success of the
course in meeting those needs.

Student commitment

This is normative congruence identified by Spady (1970), academic integration
as seen by Tinto (1975) or criteria of appropriateness of Chambers (1995, 348).
Student persistence is an indication of commitment and depends on the integration
between the student goals or needs and the educational environment provided
by the institution. The strength of this commitment is evidence of the integration
and informs the judgement or understanding of the course (Chambers, 1995, 344).
Consequently, input should come from students with low commitment who discontinue,
as well as students who have strong commitment and persist. Among that input
should be evidence concerning goals that have changed and goals achieved along
the way. Financial aspects can also be considered here as student financial
costs and support schemes have significant influence on commitment.

Wadsworth (1991) argues that where users can go elsewhere for education then
attendance is an indicator of commitment. But, where education is a monopoly
then indicators may range from discontinuation, conflict, resistance, non-attendance,
disinterest and passive compliance to more favourable comment. But, all the
discontinuing students have one factor in common - the fact that the disharmony
that they felt between the educational environment and their needs, resulted
in discontinuation. Because of this, any evaluation of a course must take their
input into account or it risks producing spurious results. Furthermore, it should
not just take the discontinuation or their special characteristics into account,
but the factors that made them discontinue.

Student development is taken up in an evaluation by considering both student
and course goals (Cheung, 1998, 28). As well, the new emphasis on flexibility
in education impacts on both the 'student goals - academic environment' match
and the 'course goals - academic environment' match. The importance of these
is shown by the serrated edges of the areas in Figure 1. An arrow connecting
the serrated edges to the judgement and understanding illustrates this commitment.

Timescale

If a technology changes during an evaluation then it can be considered as evaluation
of different situations, or as separate evaluations with different uses of technology
creating different scenarios. Alternatively, this could be viewed as an evaluation
of a changing scenario as in formative evaluation. Either way the point must
be addressed. In a contemporary on-line course the timescale becomes wedged
between maximising the opportunity for students to express their ideas about
the course, and minimising the effects of changing technologies. The Post-Structuralist
and Post-Modernist influences reach into the other methodologies when they hold
that what is viewed can only be considered valid in a specific context at a
specific point in time. They show that timing the evaluation to include the
discontinuing students and also conducting it over a generally shortened time
frame within the lifetime of a technology, should form part of an evaluation
framework. This means that positioning the evaluation to include discontinuing
students, is in accordance with the Post-Structuralist view that evaluation
considers a 'frozen instance' (Evans, 1994, 17). This is reflected in the inclusion
of a timescale as a component as shown in Figure 1. This introduction of the
elements of time and timing, provides a theoretical account of 'our present
moment of time in history' (Jameson, 1984, 85). The thicker arrows on the timescale
pointing inwards indicate the need to consider a shortened timescale for an
evaluation.

However, a number of contradictions arise in evaluation involving discontinuing
students. One is the espoused ideology of access and equity for students seeking
to undertake on-line education. Most distance education policies base their
raison d'être on this foundation. However, in the majority of cases in
the higher education sector, the discontinuing students are not included among
the evaluation participants. The curtailing of their access to education and
the strength of the equity of their education within the course is not being
questioned. Their voices are not heard. The framework used here addresses this
inconsistency by ensuring that they are at least considered.

A second contradiction concerns the generalisations arising from an evaluation
of a changing situation. Most evaluations result in judgements or better understandings
of situations. These and recommendations for action, are usually written up
in a report for wider dissemination to support any responsive action in this
situation and others. If an evaluation is tailored to a specific situation then
great care must be taken in forming generalisations, which purportedly apply
to the specific scenario even after it has been changed, and even more care
must be taken in applying these generalisations outside this situation. The
virtues of an evaluation of a stable environment should guide evaluators to
consider this issue.

In addressing these contradictions, evaluation is grounded in the fundamental
philosophy that education is designed for self-fulfillment of the student and
for the betterment of society. Even though the third emerging aspect in education,
of cost-effectiveness and efficiency, is put to one side in this case, discontinuation
of studies by students, is at some cost to the student, the educational system
and society. If an evaluator, by listening to these voices, can enable courses
to better cater for these students, then resources, previously wasted through
discontinuation, can be used more fruitfully.

Conclusion

This framework used to cater for students who discontinue has been formed by
merging different theoretical perspectives. Of special interest is the Durkheim/Spady
theory demonstrating the influence of the aspirations of both an institution
and its students and the educational environment that the institution provides,
on a judgement and/or understanding of a situation and consequent action, or
inaction. Durkheim's basic paradigm is used to justify a structure within the
framework to facilitate input from all students - both continuing and discontinuing.

The perspective of time incorporates the work of Foucault and Lyotard and the
idea of evaluation being 'a Post-Modern moment' (Usher & Edwards, 1994,
174). This leads to the stance that the 'moment' generally contains both a stable
technology and the perceptions of the students who subsequently discontinue
in order to be truly discerning. This is done by including consideration of
the timing and timescale in the framework.

Ideology critique has been used to establish a central place in the framework
for the educational environment of the course. Opportunities also exist within
the framework for consideration of the aims of the course and actual achievements.
As well, immanent critique reveals the virtue of considering both the course
and student goals and course implementation to expose the espoused aims. The
changed, and changing, nature of the students in an on-line course is catered
for by consideration of the student background. The emphasis on determining
the student goals and needs is helpful if a student discontinues; but is helpful
too, in matching student needs with the educational environment provided by
the institution.

Similarly, consideration of the social system ensures that the students, as
part of the critical reference group, remain central in an evaluation. This
generally espoused, but sometimes neglected, aspect is highlighted. As well,
the early timing of the evaluation caters for the discontinuing students, as
does the emphasis on integration of the students' goals with the educational
environment. By moving the timing of the evaluation to an earlier position,
evaluation caters for the sometimes very large student discontinuation rate
and by planning an evaluation before the students start the course, valuable
data can be gleaned from these students before they leave. Also, the short length
of time for the evaluation to accommodate a stable technological context has
been incorporated. This way, data can be collected before the technology changes
and if necessary after it changes.

Thus, evaluation is set within a social context and the considerable changes
in higher education currently affect this context in which the on-line course
is conducted. But, by siting the elements of the evaluation within the community
context, consideration of this changing scenario is ensured. Associated with
this is the blurring of distinctions between distance education, open learning
and flexible learning, so although the focus is on on-line education, it has
the potential to inform practice in distance education, open learning and flexible
learning as well. Again these influences can be considered in the community
context and the educational environment. An evaluation then enables evaluators
to look to the future and includes a cyclical process of action and betterment.

The main attribute of this form of evaluation is flexibility to cater for changes
in society, the higher education environment, distance education and on-line
practices. Yet, at the same time, as it encourages flexibility, it ensures reflection
on the fundamental components of evaluation and relationships between them within
an educational evaluation. This reflects the diversity of scenarios to be examined,
so a balance must be found between flexibility to cater for different situations
and consistency to enable generalisations to be made. The use of a framework
accommodating this ensures that an evaluation is comprehensive and efficient.
It is especially appropriate in this era of expensive technologies and burgeoning
on-line courses and students. Its use should point the way to better quality
evaluation and thus better quality teaching and learning.

References

Calvert, J. (1995). A study of the academic results on on-campus and off-campus
students. Toowoomba: NCODE.

Jameson, F. (1984). Postmodernism, or, the cultural logic of late capitalism.
New Left Review, 146(July-August), 52-92.

Lewis, R. (1989). What is "quality" in corporate open learning and
how do we measure it? Open Learning, 4(3), 9-13.

Long, M. (1994). A study of the academic results of on-campus and off-campus
students; comparative performance within four Australian tertiary institutions
(Commissioned Report 34). Canberra: National Board of Employment, Education
and Training.

Lyall, R. & McNamara, S. (2000). Influences on the orientations to learning
of distance education students in Australia. Open Learning, 15(2), 107-122.

Nunan, T. (1990, September 17-19). Evaluation of distance teaching - a review
of techniques and standards. Paper presented at the Improved Distance Education
on Agriculture, (pp. 1-9). Glenormiston: Victorian College of Agriculture and
Horticulture.