What are the current levels of learning technology usage in the five institutions?

All

Y

Y

Y

0

What is the need for basic and high level IT competence in staff and students, how does this vary between institutions and disciplines, and how is this acquired in each institution/department?

Staff and student IT training needs

S1: A, B, C, D

S2:

S3: A

Y

Y

Y

What technological approaches might enhance such research capabilities and how do these reflect any differences in discipline approaches?

Technological approaches

All

Y

Y

0

What courseware materials might be of value in each subject discipline and what are their characteristics?

Identify applicability of courseware

S1:

S2: C, D

Y

Y

0

What are the criteria for evaluating TLTP and other existing materials and technological tools?

S1:

S2: C, D

Y

Y

0

What institutional provision, support and organisation structures assist effective implementation?

infrastructure & support issues

All

Y

Y

Y

C

How comparable are the 5 sites in terms of IT infrastructure, support and teaching & learning strategies?

S1: CS2: C,D

Y

Y

Y

What is the applicability of the project outcomes and phase 2 institutions, i.e. what criteria/prerequisites will enable HEIs to map onto the TELRI framework

Applicability across the sector

Effective dissemination

All

Y

Y

Y

0

What project approaches have worked most effectively and to what extent does this vary between pahse 1 and 2 institutions and departments

Project team approaches, communication, dissemination

Projec teamS1:BS2: C,D

In what form are the project outcomes most effectively disseminated

Dissemination engagement

Phases

1a

Initial depts at Warwick & Oxford

1b

Second round depts at Warwick & Oxford

2

Depts at Birmingham, Durham & Southampton

3

Consultancy & workshops, feedback forms

Evaluation methods

ImplementationSpecific evaluation of the implementation of new methods in the courses selected will be conducted using a variety of standard evaluation methods. The approaches to implementation, and evaluation thereof, will be strongly informed by continuous communication within the project management structure.

Project approachesThose associated with the project will be evaluating their activities continually, both formally and informally, integrating their findings into practice to ensure continual improvement. To this end, each project team meeting will include evaluation of activities, reported on formally where appropriate. Quarterly reports presented by the project manager to the steering group, along with the bimonthly reports and the annual reports submitted to TLTP (and Tavistock/OU IET), will also have evaluative components. Project team members will, in their interactions with departments and individuals, be sensitive to feedback, both formal and otherwise, concerning the appropriateness and effectiveness of their work. Each intervention will be evaluated, in a process that will be devised by the project team, and that will take full account of the views of staff and students involved.

Evaluation will be made possible through information collected from:

Research Phase

Evaluation Area

Evaluation method(s)

Current stages of departmental IT-based teaching innovation; preparation for QAA Subject Review; comparisons between departments at Warwick and Oxford;

Semi-structured interviews

Research methodologies and capability components in curricula;

Literature reviewsDiscussion of TELRI model

Overall objectives of departments in TELRI project; current teaching methods; discipline comparison (subject benchmarking analysis); attitudes and priorities for areas to develop of academic staff in departments; student motivation, achievement and problem areas; involvement in other collaborative projects; number of staff to be involved;

Semi-structured interviews

IT infrastructure capacity; current limitations; planned developments; support structures;

Individual staff from similar disciplines at Warwick and Oxford; common issues identified and discussed, conclusions drawn; mapping educational model for research capability development; feasibility of technological approaches.

Student groups to look at motivation, understanding of course objectives and assessment criteria;

Project team meetings and progress reporting Reporting from project team officers as research & implementation work proceeds; summaries of important issues; outcomes and conclusions documented; effectiveness of communication and reporting mechanisms of the team; issues to be discussed by steering group.

The project will make it a priority to include a qualitative cost-benefit analysis in a number of the interventions. An approach is indicated below.

Educational effectiveness must first be measured, comparing the effectiveness of the new teaching and learning strategies with the previous approaches, including the role played by the technological intervention. Immediate and long-term benefits may occur in:

explicity in curricula about learning outcomes & assessment criteria;

enhanced student achievement in the skills that contribute to research capability;

improved access to learning opportunities;

cohesion of teaching and learning processes into a coherent strategy;

increased ratio of motivators to barriers (leading to increased uptake and success);

maintenance of quality in courses involving larger numbers of students;

reducing teaching staff work loads in preparation, delivery, support and marking;

increasing the quality time that staff can spend with students;

reducing the overall cost of methods and materials in the long term;

more effective use of IT infrastructure;

potential for distance learning development.

In terms of the benefits in improving student motivation and learning, one would ideally measure learning outcomes of groups making use of IT against those who do not. This might be achieved by evaluating outcomes with the previous years' equivalent groups. It is unlikely, given the relatively small scale of the project, that one would be able to identify major learning shifts. However, it should be possible for each intervention to derive qualitative data on changes in student capability, and this will be obtained through interviews with staff and students.

The cost of each intervention might also be measured, in terms of: improved or increased use of existing equipment; provision of new equipment; staff training and learning time; and production of new tools and materials. The intervention cost must be compared with the cost of current or where possible alternative methods. Cost in pilot courses will be projected to longer-term benefits of the intervention in terms of maintaining quality and values, staff development, student motivation and achievement, and other factors affecting sustainability (validity and viability).

Collaboration with the experts in the area of cost-effectiveness will inform our approaches and assist with the practical methodology, based on their experience at institutional levels. Gordon Doughty at University of Glasgow (previously TLTSN Centre) has already been approached for external consultancy.

Statements about cost-effectiveness may be derived by considering the overall benefits against the net cost of implementation. This can be guided by change management considerations that relate the likelihood of sustainable change in terms of the balance between desirability of change, net costs and extent of disruption required.

External evaluation

An external evaluation has been commissioned from the Centre for Educational Development, Appraisal and Research (CEDAR) at the University of Warwick. Dr Andrew Parker will carry out the external evaluation and consultation is taking place with the Project Director and Project Manager.

An interim evaluation will review the first eighteen months of the project, and will focus on the extent to which the project team has met its planned outcomes, and its preparedness for meeting the remaining outcomes, including the effectiveness of communication among all levels of the project and all participants.

A second external evaluation will review the project at three years, with a focus on educational and cost-effectiveness, both of the project as a whole and of the individual interventions that have been made. Extensive use will be made of evidence drawn from staff and students who have come into contact with the work of the project. On each occasion a project report will be provided to the steering group.