…For Teaching ELL, ESL, & EFL

The Best Resources Showing Why We Need To Be “Data-Informed” & Not “Data-Driven”

Last year, two very talented educators — Ted Appel, the extraordinary principal we have at our school, and Kelly Young, creator of much of the engaging curriculum we use at our school through his Pebble Creek Labs — brought-up the same point in separate meetings with teachers at my school: The importance of not being “data-driven” and, instead, to be “data-informed.”

These conversations took place in the context of discussing the results of state standardized tests. Here’s the point made by Ted:

If schools are data-driven, they might make decisions like keeping students who are “borderline” between algebra and a higher-level of math in algebra so that they do well in the algebra state test. Or, in English, teachers might focus a lot of energy on teaching a “strand” that is heavy on the tests — even though it might not help the student become a life-long reader. In other words, the school can tend to focus on its institutional self-interest instead of what’s best for the students.

In schools that are data-informed, test results are just one more piece of information that can be helpful in determining future directions.

Since that conversation took place, I’ve written several posts about the topic. I thought it might be useful to bring together several related resources.

Here are my choices for The Best Resources Showing Why We Need To Be “Data-Informed” & Not “Data-Driven”:

The cartoon reminded of what the New York judge said earlier this month when he ruled that the School District can publicly release the names of teachers and their “Teacher Data Reports.” Here is what the judge said (and I kid you not):

“The UFT’s argument that the data reflected in the TDRs should not be released because the TDRs are so flawed and unreliable as to be subjective is without merit,” the judge wrote, citing legal precedent that “there is no requirement that data be reliable for it to be disclosed.”

An article in Educational Leadership is a year-old, but it’s new to me and certainly worth sharing. It’s called The New Stupid, and has the subtitle “Educators have made great strides in using data. But danger lies ahead for those who misunderstand what data can and can’t do.” It’s written by Frederick M. Hess.

…the key is not to retreat from data but to truly embrace the data by asking hard questions, considering organizational realities, and contemplating unintended consequences. Absent sensible restraint, it is not difficult to envision a raft of poor judgments governing staffing, operations, and instruction—all in the name of “data-driven decision making.”

and…
First, educators should be wary of allowing data or research to substitute for good judgment. When presented with persuasive findings or promising new programs, it is still vital to ask the simple questions: What are the presumed benefits of adopting this program or reform? What are the costs? How confident are we that the promised results are replicable? What contextual factors might complicate projections? Data-driven decision making does not simply require good data; it also requires good decisions.

He describes a class of antipsychotic drugs, whose effectiveness was demonstrated by several large clinical trials. But in a subsequent batch of studies, the therapeutic power of the drugs appeared to wane precipitously.

This is not an isolated case. “But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain,” Lehrer writes. “It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”

The world is fluid. Bias and randomness can creep in from all directions. For example, between 1966 and 1995 there were 47 acupuncture studies conducted in Japan, Taiwan and China, and they all found it to be an effective therapy. There were 94 studies in the U.S., Sweden and Britain, and only 56 percent showed benefits. The lesson is not to throw out studies, but to never underestimate the complexity of the world around.

“It’s amazing how much it’s possible to figure out by analyzing the various kinds of data I’ve kept,” Stephen Wolfram says. To which I say, “I’m looking at your data, and you know what’s amazing to me? How much of you is missing.”

This is the last paragraph of Robert Krulwich’s article at NPR, titled Mirror, Mirror On The Wall, Does The Data Tell It All? In it, he compares authors of books, one by Stephen Wolfram, creator of a the Wolfram search engine, and Bill Bryson, author of a biographical account of growing up in Iowa. The column, though not specifically about schools, hits a “bulls-eye” on our current data-driven madness.

It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?

At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.

David Brooks, who generally loses all coherence when he writes explicitly about education issues, has just written an eloquent case for the importance of being data-informed, and not data-driven. Read his column titled What Data Can’t Do. Here’s an excerpt:

At the risk of being accused of taking a “cheap shot,” I just can’t resist embedding two segments from The Colbert Show about the now well-known mistake by the two economists whose work has been cited endlessly to support austerity. And I can’t resist adding it to this list:

Well said. Being data informed is certainly driving a great deal of change within education, balanced with other important information about students. There is a lot of great work being done with data analysis and data linkage tools for the future of education with P20 and SLDS initiatives. Linking K-12 data with college and career data will certainly have a positive, significant impact on student achievement.

A single comprehensive and timely education data standard that ensures usability is key to the success of these programs. States need to take advantage of the data analysis that is being made available.

My Second Book On Student Motivation!

My Second Book On Teaching ELLs

My book, "The ESL/ELL Teacher's Survival Guide: Ready-to-Use Strategies, Tools, and Activities for Teaching English Language Learners of All Levels," (co-authored by Katie Hull Sypnieski) was published in the Summer of 2012