While each of the previous sections of this
guide has included examples of evaluation of
that specific component of program operations,
both providers and districts need to look
beyond process to outcomes.* The bottom line
in measuring course success is how well students
do. While partnering districts and schools
must track the progress and success of their
individual students, online providers will want
to track how the broad range of their online
students are performing as an indicator of their
own success. All data can then be used to make
program improvements.

VHS, for example, tracks and reports several
measures of student success, each of which tells
an important part of the story. For AP courses, in
particular, VHS tracks student enrollment, course
completion rates, the number of students taking
AP exams, and the how they do on those exams.
As part of schools' partnership agreements
with VHS, their students are required to share
their AP exam scores. Site coordinators receive
student scores from the College Board and submit
a copy to VHS. To protect students' privacy,
when VHS leaders receive score reports, they
record them using a random tracking number,
and the copies are destroyed.

Figure 6 shows a dramatic increase
in the number of AP course enrollments
through VHS over the past five years. As of
2007, VHS reported an 85 percent completion
rate for AP courses. 41 Figure 7
shows how students are doing on AP exams.
As enrollment has increased, so has the number
of exams taken — up to 350 in 2005–06.
The bottom part of each bar shows the number
of exams taken but not passed; the top
part shows the number of exams taken and
passed. The percentage of students passing
the exam has fluctuated some over the years,
starting quite high when only a few students
took the exams and dropping as more students
enrolled, 42 but has always exceeded the average
pass rate for the U.S. as a whole, which was
59 percent in each of the last two years. 43

Of course, the goal is for all students to master
the material well enough to take and pass
the exam. But, as noted earlier, some research
shows that taking an AP course is valuable
even if a student does not take or does not
pass the exam. 44 And there are likely to be
some transition steps as more students are encouraged
to take AP courses. Some students
may be more comfortable taking the course
without having to take the exam, so allowing
this option may encourage them to try taking
an AP course. Likewise, students may not do
well on a first exam, but may do better on
any subsequent exams they take. By tracking
all four measures, a program and a district or
school will gain valuable data that shed light
on areas to investigate in an effort to improve
outcomes over time.

The providers profiled in this guide also track
a variety of other outcome measures. COL contracts
with a private consulting firm, the Public
Good, to conduct an especially comprehensive
evaluation of program outcomes that includes
measures of enrollment, course completion,
grades, and student perceptions of what they
have learned. Course completion rates at COL
have increased each of the last four years, from
78 percent in 2003 to 92 percent in 2006. 45
When students themselves reflected on what
they had learned, many mentioned 21st century
skills that were enhanced by the online learning
process; that is, skills recognized as important
for success as citizens and workers in the 21st
century, like self-direction and communication.
For example, one student reported, "I think that
students should take online courses because it
teaches them how to be independent in their
studies." Another said, "This course taught me
more self-discipline in one semester than in all
of my other years combined." 46

Figure 6. Virtual High School's Advanced Placement Enrollment,
2001–02 to 2005–06 School Years

COL also looks closely at the characteristics of
the students being served. Charged with serving
schools with high academic needs and those
with high poverty rates, COL can report that
more than half the districts enrolling students in
2006 can be considered high-need, high-poverty
districts, up from an average of 34 percent
in the preceding years. This makes the high
course completion rate of 92 percent all the
more impressive.

In addition to collecting quantitative data (e.g.,
enrollment numbers, passing rates), these providers
and their partner schools or districts also
find it useful to collect qualitative data to help
them improve their program. Their evaluations
include satisfaction surveys of key stakeholders,
including parents and students, as well as
administrators, teachers, and site coordinators.
A number of providers ask students to complete
end-of-course evaluations (see fig. 8 for an example). CTY's survey asks about
quality of instruction, the student's level of interest
in course content prior to and after taking
the course, ease of technology use, and the student's
academic plans. COL also asks those who
withdraw from an online course to complete an
"early exit" survey, via e-mail. All providers indicated
an interest in learning more from students
who dropped an online course prior to completion
about their reasons for doing so. VHS and
FLVS also use their surveys to probe about unmet
course needs. FLVS increased the number
of electives it offers based on requests delivered
through its customer satisfaction
survey.

Figure 7. Virtual High School Students' Performance on Advanced Placement
Examinations, 2001–02 to 2005–06 School Years