Justin Reich is the executive director of the MIT Teaching Systems Lab, a fellow at Harvard’s Berkman Center for Internet & Society, and the co-founder of EdTechTeacher. Beth Holland is a doctoral candidate at Johns Hopkins University and an instructor at EdTechTeacher. Douglas Kiang has over 25 years of teaching experience at all grade levels and currently teaches computer science at Punahou School.

To Ensure Success, First Define the Black Box of Innovation

By Beth Holland on
February 17, 2017 6:06 PM

I first read about the notion of the "black box" in the context of the economics and education. Economic researchers frequently write about inputs and outputs. They examine what goes into the black box of schools in terms of time, money, or resources and then measure what comes out - student test scores, teacher evaluations, etc. However, economists rarely open that box.

Take the example of increased teacher salaries. An economic study may suggest that schools who receive more money to pay teachers (input) show increased student test scores (output). The causal relationship that higher paid teachers lead to greater student achievement may seem logical based on an inference such as higher quality teachers may receive a higher pay. But what actually happened inside the school? What did those teachers who received more money actually do to improve student performance? Can the improvement really be attributed to the teachers and their salaries, or did something else occur?

In their 2012 article, Theoretical Frameworks to Guide School Improvement, authors Evans, Thornton, and Usinger explain that most educational reform efforts occur absent a defined theory of change. Instead, schools and districts implement new plans or programs without defining why they believe that these change efforts should work; what they hope to achieve; and how they might measure success. In concrete terms, consider the technology programs, curriculum overhauls, and even STEM and Maker movements that continue to be implemented with a black box approach.

Schools roll out new devices and programs (inputs) but neither identify the outputs and outcomes that they hope to achieve nor the logic detailing the process for change. As a result, educators and administrators often lament that their new program failed to gain traction or abandon the efforts because they lack positive results. However, without first delineating the inner workings of the black box, it is virtually impossible to fully understand what happened inside. What actually failed: the theory or the implementation?

Though researchers Leviton and Lipsey wrote about the idea of "treatment theory" in reference to formal, empirical evaluation studies, consider the possibility if we applied their recommendations to programs in schools. First, before designing any sort of program, define the precise problem as it exists in context. If iPads or Chromebooks or Makerspaces are the desired input, what problem are they trying to solve? Next, establish exactly what should happen within the program. What activities will students complete and when? What professional development might teachers require and with what frequency? Then, with the process laid out, stipulate any factors that may impact your planning or mitigate the effects of your program. Contemplate everything from parental concerns to teacher buy-in to administrator support. Finally, describe the desired outcomes. What new knowledge, attitudes, beliefs, or skills should result from this new program?

It always saddens me when I hear that a school made an initial investment in a device, or initiative and then abandoned it because of "failure" -- especially when similar organizations have succeeded with the same endeavor. That prior success implies that the root of the problem may be the execution of the program and not the underlying premise. Imagine the possibilities for new innovations if teachers, administrators, students, and the broader community all understood the inner workings of the black box before beginning something new. Hopefully, it would ensure that the theory would support the implementation, and there would be nothing to blame.

Categories:

Tags:

Notice: We recently upgraded our comments. (Learn more here.) If you are logged in as a subscriber or registered user and already have a Display Name on edweek.org, you can post comments. If you do not already have a Display Name, please create one here.

Ground Rules for Posting
We encourage lively debate, but please be respectful of others. Profanity and personal attacks are prohibited. By commenting, you are agreeing to abide by our user agreement.
All comments are public.