Interactive Learning Online at Public Universities: Evidence from Randomized Trials

Interactive Learning Online at Public Universities: Evidence from Randomized Trials is not quite a policy essay, but is not a typical research article, either. The authors are associated with Ithaka S+R, which sponsored the study. Ithaka S+R, like JSTOR and Portico, is part of ITHAKA, a not-for-profit strategic consulting and research organization that focuses on the use of technology in higher education.

After sharing the context for the current push to provide online educational options in terms of access, affordability, and reduction of funding, the authors raise the concern driving their research—that “at least some kinds of online learning are low quality and that online learning in general de-personalizes education” (p. 7). The research presented is clearly limited to a specific, well-defined form of online learning—interactive computer programs that provide machine-guided customized instruction and guidance or “interactive learning online” (ILO). Many institutions do not utilize ILOs at this time, but the study’s research questions, methodology, and process notes can be informative to develop institutional research of relevant technologies.

The text is organized in three documents, all available as PDFs for download: the main text with two appendices, and two separate documents with the data collection instruments and the research protocols. A cost simulation macro is also available for download. The main text describes the research process, including four guiding research questions:

Can sophisticated, interactive online courses be used to maintain or improve basic learning outcomes?

Are these courses as effective, or possibly more effective, for minority and low-socioeconomic-status students and for other groups subject to stereotype threat?

Are they equally effective with not-so-well-prepared students and well-prepared students?

Are they equally effective in a variety of campus settings—community colleges versus four-year colleges, commuter colleges versus colleges with more students in residence? (p. 9)

Significant attention is paid to the methodology before the presentation of findings. A separate section is dedicated to questions regarding cost and estimates of potential savings; a more complete description of that analysis is in Appendix B. Details of the institutions and the sampling process are found in Appendix A. Explanations of the methodology are provided, such as how students were assigned to treatment or control groups, all of which are helpful in both understanding the findings and incorporating elements of the process in similar studies.

The authors address their decision to exclude community colleges from the study. Unfortunately, this limits the usability of the data to address the last two research questions. Bowen et al.’s definition of “not-so-well-prepared students” does little to capture the range of academic preparation reflected at many community colleges.

Are students who learn the building blocks, so to speak, with ILO hybrid formats able to use that knowledge to solve new problems at the same level as students who learn the building blocks in face-to-face formats?

The finding of the research is a resounding “no difference” in student mastery of basic learning outcomes. Multiple measures are used to establish student performance, including course completion rates, assignments, surveys, and student course evaluations. The importance of the analysis supporting the null hypothesis is clearly articulated in the report and is a point sometimes overlooked by those who only value change if it results in something “better.”

While course completion itself is not included in the research questions, persistence in online learning environments is a general concern. The authors include several tables in Appendix A that report learning outcomes, including course completion. There was no statistically significant difference in course completion within each institution between the ILO hybrid format and the traditional format.

While this study provides solid footing on basic content acquisition, researchers might want to expand the questions to address whether there are types of learning and mastery that ILOs support better or worse than face-to-face learning. For example, one of the challenges of learning new concepts is the ability to transfer knowledge from one type of problem to another, or “flexible knowledge.” Are students who learn the building blocks, so to speak, with ILO hybrid formats able to use that knowledge to solve new problems at the same level as students who learn the building blocks in face-to-face formats?

The second prong of the report is a “conversation” about potential cost savings. The difficulty of calculating the full cost of instruction proved daunting for the authors. After acknowledging the many variables needed for a meaningful analysis of cost, they offered “cost probes” instead. Rather than consider how much an institution would save by moving to an ILO hybrid format, Bowen et al. asked “under what assumptions will cost savings be realized, over time, by shifting to a hybrid format, and how large are those savings likely to be?” (p. 25). This is an important distinction as it excludes all development and initial costs. Their focus is limited to faculty compensation and productivity. While this addresses long-term cost savings, it provides an unrealistic estimation of the true cost of technology-based instruction. Even if institutions could find outside agencies to finance the cost of development or purchase and faculty training, as the authors suggest, there are still ongoing IT costs related to capacity, personnel, upgrades, continual training of new instructors, etc. The authors recognize that their cost savings findings are “too speculative” (p. 26) to provide actual calculations.

Interactive Learning Online at Public Universities can be useful for institutional researchers in a couple of ways. First, the findings themselves can inform research choices when studying online teaching technologies at our institutions. Second, because of its thorough presentation of methodology, extensive footnotes, and bibliography, the report can assist researchers in developing their own institutional research projects. Bowen et al. provide the details necessary to replicate their quasi-experimental design. The extensive documentation provides researchers with “red flag situations” to plan for when working with faculty. Even though it was not the piece’s strongest section, the inclusion of cost effectiveness in the larger analysis is a trend that more IR professionals may want to adopt. Online teaching technologies and modes of delivery are often considered inevitable for both increasing access and reducing cost, and Interactive Learning Online at Public Universities provides an important reference for IR professionals to address the fundamental issue of quality of learning and to document costs and related benefits.

Sally Jackson is the Director of Planning, Institutional Effectiveness, and Research at Spokane Falls Community College.

Comment

This is certainly an interesting study. While the "no difference in basic content acquisition" findings do not seem to suggest an advantage of ILOs for not-so-well-prepared students - I wonder whether the more flexible access to ILOs - as opposed to the more limited access to instruction(as in the limited face-to-face class time with instructor) may be a benefit. For example - a student can run the applicaiton multiple times to achieve mastery of the material - not always possible in time-limited in-class settings. This reminds me of the Khan academy videos - not sure if this is a similar delivery approach.

This is indeed a good resource for Institutions and individuals for deciding whether to use ILO or the traditional format.Interesting also is the"no difference" in basic learning outcomes as the research revealed.The decision to emphasize ILO or traditional face-t--face, lies now with institutions.With matured students(Post Graduates),ILO may be the best as they are able use proper time management and devote their time to their scheduled online course-time.However,am skeptical about ILO for the younger generation(under Graduates) as evidences show poor reading culture among the newer generation.Very impressive is the methodology as reviewed by Sally.The only area that was left out,in my understanding,is the cost effectiveness of ILO for institutions viz a viz traditional format.Further research is essential in this aspect as speculations may not offer a better ground for Institutional decisions.Sally thanks for a great job!!