Note: This article was originally published in The Technology Source (http://ts.mivu.org/) as: Stephen C. Ehrmann "The Flashlight Project: Tools for Monitoring the Progress of our Hopes and Fears about Technology in Education" The Technology Source, July 1998. Available online at http://ts.mivu.org/default.asp?show=article&id=1034. The article is reprinted here with permission of the publisher.

Since the 1950s, people have been trying to use computers, video, and
telecommunications to revolutionize education. For decades, their vision was dominated by
several ideas about how to use technology to extend the teachers' reach. For example:

using television to make some of the world's greatest teachers available to students at
great distances or over time (e.g., for years after a great lecturer in philosophy had
retired, one could still watch her lectures);

using television to educate remote students or groups of students who could not
otherwise reach or afford a teacher;

using computers to provide self-paced, individualized instruction with tailored feedback
that could help students learn a greater variety of topics with greater speed and
effectiveness and yet at lower costs because of economies of scale. The wisdom and insight
of great teachers could be used to help create the software.

Although there are many examples of such uses of technology, they have not been
sufficient to constitute a revolution in the nature of education. The most important
barriers to using technology to help teachers to extend themselves have been economic
(e.g., the expense of developing courseware, computers advancing so fast that expensive
courseware becomes obsolete before ever finding a market, etc. [Morris et. al. 1994]).

The original hopes for revolution of teacher augmentation and replacement have been
stymied. Nevertheless, higher education does seem to be headed into a revolution, one that
will use technology energetically but not primarily as a way of extending our best and
brightest teachers. Flashlight is a flexible evaluation toolkit that can be used to study
the kinds of progress that are actually happening in most places. It can also be used to
check whether some of our most common fears about technology are coming true. As we'll
see, Flashlight can be used to start studies at the level of classes, courses of study,
services, and institutions, as well as enabling studies that cross institutional
boundaries.

The Emerging Vision

The revolution in education that is actually emerging is based on uses of technology
that are:

similar for on-campus and off-campus students (and for professionals in the larger
world);

helping a wider range of students (and faculty) reach a wider range of intellectual
tools and resources;

helping students learn real-world skills that are partly dependent on technology use
(e.g., statistical ways of thinking that are more visual and more recursive than possible
with paper-and-pencil techniques; ways of thinking about photography and graphic design
that make use of computer-based image processing and printing);

helping to widen access to education (e.g., to students whose time on campus is limited,
students with disabilities, etc.);

increasing the ability of an institution to compete for students;

used in ways that can improve faculty-student interaction, student-student interaction,
active learning, time on task, and other practices that research shows are usually
correlated with improved learning outcomes.

The software used to implement these visions is usually not the content-specific,
mass-produced teaching machine of the older vision. Instead it is most often empowering
tools or library-like resources, often the same as the tools and resources used outside
the academy. The vision on which Flashlight is based, and the methods used to identify it,
are described on The Flashlight Project Homepage.

The Flashlight Current Student Inventory

The Flashlight Current Student Inventory (CSI) can be used to collect facts
and opinions from currently enrolled students about the uses of technology in the bulleted
list above, as well as many other applications. The CSI is a toolkit of almost 500 indexed
questions that can be used to draft surveys, questionnaires, and protocols for interviews
and focus groups. Many of those questions focus on the ways in which technology is
helping, or failing to help, implement research-based principles of good practice in
education (Chickering & Ehrmann, 1996). It includes the Flashlight Evaluation
Handbook (1997) of guidance on creating studies, readings, resources, and case
studies. The Flashlight Current Student Inventory is available for site license
from the TLT Group.

Flashlight has a variety of applications, including:

guiding improvement of classes and courses of study (e.g., majors, minors, freshman year
skills development, writing across the curriculum) and strengthening the roles played by
technology in such efforts;

supporting integration of strategic thinking about the curriculum and technology
services;

preparing for accreditation;

helping faculty, departments, or institutions compare their uses of technology and
outcomes; and

redesigning student evaluations of faculty.

For example, faculty in a department might collaborate to develop a standardized survey
for their students (with some optional sections or questions for each course). They could
also use the CSI to design a regular program of group interviews with students, to ask
questions that are too open-ended to address with surveys. Over time, the information
could help the department see whether its changing uses of technology for students on- and
off-campus are helping (or interfering with) active problem-solving, student-faculty
interaction, student-student collaboration, student time on task, and other key
quality-related activities. This information will be valuable on its own and can also help
explain why learning outcomes (as assessed by other means) are or are not improving. If
there are no other means of comparing learning outcomes, then a finding that teaching and
learning practices are improving could stand in as a measure of quality.

The faculty can also use their survey and interviews to monitor changes in other issues
of interest (e.g., quality of training and support for technology and the extent to which
students are using certain technologies in other courses, in their jobs, and in their
leisure time). This kind of information, and the continuing discussion about the research
tools and findings, can help faculty collectively improve their instructional program and
teaching methods. The same findings can also show an accreditor how the department is
using data to improve instruction and use technology more cost-effectively.

The CSI and its accompanying handbook are the first components of Flashlight to be
released. Under development are a guide to cost modeling and an inventory of questions for
gathering information from faculty. Other instruments are also planned.

The Flashlight Approach

The Flashlight approach to asking questions is based on two basic premises drawn from
research on technology and learning.

First, technology per se does not determine learning outcomes (Educational
Technology..., 1994). Rather, learning outcomes are influenced by the choices that
faculty, students, and others make about the organization of teaching and learning,
including their practices, choices about content, and so on (Ehrmann, March/April 1995).
The role of modern technology is to expand the available choices (this is part of what we
mean when we say that it is 'empowering'). Technology also makes some of those choices
more feasible in certain circumstances. For example, all forms of presentation (e.g.,
lectures in a lecture hall with slides, lectures in a lecture hall with computer
"overheads," reading the lecture material in a book, hearing the lecture
one-on-one in a faculty office, reading the lecture material on the World Wide Web) have
the same learning outcomes, but some of those technologies make it possible to reach
people at a distance, some are more flexible for last minute changes, some are more
adaptable to highly visual materials, and so on. For this reason the CSI has separate
families of questions about teaching and learning practices (especially their prevalence),
technology per se (e.g., how reliable or unreliable has it been), and most importantly,
student reports on how frequently they use a specific technology to support a specific
teaching and learning practice, and how useful and/or problematic they judge the
technology to be when used for that purpose.

For example, Flashlight questions can be used to ask students how often they use the
local e-mail system to do homework with other students, and what advantages and
disadvantages it has relative to other means of working together.

A second premise underlying Flashlight's design: it is difficult and sometimes
impossible to evaluate local uses of technology by comparing learning outcomes. There are
many reasons for this, not the least of which is that the use of technology often involves
changes in the goals of the program and thus in how student learning is assessed.
Fortunately, research indicates that some teaching and learning practices tend to cause
improvements in learning outcomes (e.g., increased time on task; student-student
collaboration).

For that reason, Flashlight focuses many of its questions on such research-based
principles of good practice. If Flashlight data indicates that good practices are more
prevalent today than yesterday (or in a comparison group) and technology is judged to be
useful in implementing those practices to this degree, then a case can be made that
technology has probably helped improve learning outcomes, even in situations where such
improvements cannot be directly observed.

When improvements in learning outcomes can be directly observed, Flashlight data
can help indicate whether technology use played any role in that improvement (e.g., Brown,
1997).

Note: A table is also available which
illustrates the ways in which Flashlight and the CSI compare with the components of the vision
of higher education improvement and transformation summarized earlier.

What's Been Discovered So Far

Flashlight went into distribution in December 1997, so the only findings thus far are a
side benefit of the beta test process. Early studies using the beta version of Flashlight
uncovered instances where technology was not being used in ways that altered
teaching or learning practices. Gary Brown of Washington State University (1997) reports on a
lecturer who continued to lecture, but with better "slides" for example.
Harrington (1997) reports on sections taught in computer networked classrooms that were
encouraged to be taught in ways quite similar to sections taught in traditional
classrooms; faculty wanted to be fair to students in the traditional classrooms. In such
circumstances it is not likely that long-term learning outcomes improved.

On the other hand, Gary Brown used Flashlight to study a special seminar for at-risk
students. He documented
an improvement in freshman GPA and, with Flashlight, found evidence suggesting that
technology use had helped implement superlative teaching and learning practices in these
seminars, providing a plausible explanation for the improvement in grades.

Even these early uses of the beta version of Flashlight have had an impact on practice.
Brown reports that "[t]he expansion of the Freshman Seminar program can be partially
attributed to Flashlight findings from the pilot courses. The Flashlight report helped
persuade key members of the Faculty Senate to support the new program, which passed by
just a few votes."

Brown's findings strengthened prior research in the field that showed that the use of
multimedia is unlikely to improve outcomes if it is not used to improve teaching and
learning practices. As he noted,

A lecture, with or without multimedia, does not purport to help students manage large
complex tasks, exercise creativity, or even discuss content with peers or experts inside
or outside of educational institution... WSU, encouraged by findings surfaced and
articulated through the use of the Flashlight CSI, has begun to reexamine some of our
general education course objectives.

He goes on to say that "Several instructors who previously have used
multimedia-enhanced lectures are introducing, incrementally as access to resources
permits, more interactive and generative activities into their classes" (Brown,
1997).

There were practical implications from other Flashlight findings, as Brown reports:

For instance, the number of students who used the Internet to discuss subject matter
beyond the classroom walls pleasantly surprised administrators of the Freshman Seminar
program. Consequently, the revised syllabus now includes a project developed with partners
in other WSU colleges and at other institutions... Further, findings from the Flashlight
survey have helped to garner a $10,000 grant to develop a template version of [that] model
(Brown, 1997).

Other benefits are more subtle. One of the original objectives of Flashlight was to
provide the kind of information that would make it easier for educators to think
strategically, and that seems to be happening. President Pamela Pease of the online
International University writes that "International University has found the
Flashlight Project a process that has engaged the entire institution toward its goal of
improving student services, and teaching and learning within a 'virtual' learning
environment" (personal communication, March 17, 1998). Tony Fiddes of Qantas Airways
is preparing to evaluate their Web-based internal training system, Qantas College Online.
He reports that Flashlight is "prompting managers to think more strategically about
training and how they manage itmoving much more to the notion of having to discuss
and manage individual training plans and looking at expected behavioural changes"
(personal communication, March 9, 1998).

Flashlight use also seems to help individual faculty in thinking about instructional
design and teaching issues. In describing her uses of Flashlight in working with faculty
members at Indiana University-Purdue University Indianapolis (IUPUI), Elizabeth Rubens
says,

For me, the most powerful effect of Flashlight is that it confronts faculty with a new
set of criteria with which to judge the effectiveness of their teaching. After reviewing
the items, faculty begin to think how they might move beyond simply packaging lectures via
a new delivery mechanism and begin to help students structure experiences which promote
the principles of good practice. Seeing the yardstick change from simple exam scores to
thinking about student collaboration and creativity is a powerful motivator to promote
curricular change (Personal communication, March 16, 1998).

Flashlight's History, Home, and Benefactors

Flashlight was created by the Annenberg/CPB Project and is now headquartered at the TLT
Group (the Teaching, Learning, and Technology Affiliate of the American Association for
Higher Education). Flashlight planning was supported by the Fund for the Improvement of
Postsecondary Education (FIPSE). Development is being carried out by the Western
Cooperative for Educational Telecommunications of WICHE and Indiana University-Purdue
University Indianapolis (IUPUI). Other participating institutions are the Education
Network of Maine, Maricopa Community Colleges, Rochester Institute of Technology, and
Washington State University. The Flashlight Project is directed by Stephen C. Ehrmann, Ph.D., vice
president of the TLT Group. The associate director of the Flashlight Project is Robin Etter Zuniga of the Western
Cooperative for Educational Telecommunications.

Flashlight has a free electronic newsletter that publishes information about workshops,
availability of tools, and so on. To subscribe, send e-mail to listproc@listproc.wsu.edu with the one line
message "SUBSCRIBE F-LIGHT your name".

If your institution, consortium, or system is interested in site licensing the
Flashlight Current Student Inventory, obtaining a video (NTSC) about the project,
sponsoring a talk or workshop about evaluation, technology and the Flashlight project,
getting online training about evaluation and Flashlight, or obtaining consulting services
from the Flashlight team, please contact the TLT
group.

Educational Technology Research and Development (1994). XLII: 2/3.Washington,
DC: Association for Educational Communications and Technology.

Ehrmann, S. C., (March/April 1995). Asking the right questions: What does research tell
us about technology and higher learning?. Change: The Magazine of Higher Learning, 27 (2),
20-27. Retrieved March 15, 1998 from the World Wide Web: http://www.learner.org/edtech/rscheval/rightquestion.html