A place to write my way to understanding about issues related to teaching and learning. (Because of my experience, my focus is on mathematics education.) Please join me as I explore the changing educational landscape.

Pages

Monday, July 21, 2014

How compelling was that single study?

Dear Editor,

Ever
since Chancellor Fariña began advocating for using aspects of balanced literacy
in New York City Public Schools (New York Times, 6/26/14), she has come under fire. While her
only action thus far has been to ask Lucy Calkins, a proponent of a balanced literacy approach,
to do a seminar in August, the critics have cried foul. They argue that Chancellor
Fariña is ignoring the results from a 2012 study of a pilot program called
Core Knowledge Language Arts [CKLA] (New York Times, 3/11/12). Given the information provided about the
study, however, these critics are seemingly misrepresenting the results.

Yet the Education Department’s own three-year study comparing Core
Knowledge with balanced literacy in demographically matched New York City
schools showed the former “had significantly stronger gains than comparison
school students on nearly all measures.”

The New York City Department of Education recently did a
three-year study comparing 20 schools. Ten used the Core
Knowledge approach. Ten used balanced literacy. After analysis, the Core
Knowledge results were deemed to be far better, to a high level of statistical
significance.

[The Bloomberg Administration] tested things out and saw, at least in
that limited study, that Core Knowledge was working better for low-income kids
...

Whenever
someone states, "the study shows," related to something as important
as educating our children, please be wary. Our immediate response ought to be, "Where is
the research? I would like to examine it for myself." In this case, many
of the critics provide only a link to a New York Times article on
the study from 3/11/12. It describes the study thusly:

For three years, a pilot program tracked the reading ability of
approximately 1,000 students at 20 New York City schools, following them from
kindergarten through second grade. Half of the schools adopted a curriculum
designed by the education theorist E. D. Hirsch Jr.’s Core Knowledge Foundation. The
other 10 used a variety of methods, but most fell under the definition of
“balanced literacy,” an approach that was spread citywide by former Schools
Chancellor Joel I. Klein, beginning in 2003.

Based on
this description, it seems unlikely that the study did a direct comparison
between balanced literacy and CKLA. After an unproductive
search for the actual study, I only found a PowerPoint on Core Knowledge Foundation's own
website sharing the results from the CKLA Pilot. This is how the Core
Knowledge Foundation described the study:

There is
nothing about balanced literacy in this description. In fact, there is nothing
about balanced literacy or Lucy Calkins in the entire PowerPoint.

If there
is more to the study that supports the critics’ claim that a direct comparison
was done between balanced literacy and CKLA, then it is up to them to provide
that information. It is not enough to simply say, “a study shows.” Without the
actual study to anchor the discussion, arguments can drift far from the actual
results.

For
example, given the history of balanced literacy in New York City Public
Schools, perhaps the critics jumped to the conclusion that the comparison
schools were using balanced literacy. Therefore, in their minds, CKLA was
compared directly to balanced literacy. While this history might be enough to
support their opinion in casual conversation, research conclusions require
confirmation. And no evidence was provided that confirms that the comparison
schools were using the instructional approach described by several of the
authors under the banner of balanced literacy.

Every
study, and anyone who wants to appropriately cite that study, is limited by the
study’s design. The design of the CKLA Pilot Study limits how we can use the
results. If the critics are serious that they want Chancellor Fariña to use
data to inform education policy, then a new study ought to be conducted. That
is what the International Reading Association suggests:

With greater care in the design, analysis, and reporting, these
shortcomings could have been resolved. If we could have confidence that the
gains that students in the Core Knowledge schools showed on assessments of
reading comprehension, as well as assessments of science and social studies
knowledge, could be attributed to the resources that were introduced, then
there is cause for celebration. The Core Knowledge program could, in fact,
be an important element in a truly “balanced” literacy curriculum.

To his
credit, even E. D. Hirsch, Jr. supports this idea. At the end of his opinion
piece, he writes:

But I agree with the schools chancellor, Carmen Fariña. The study was
too small. We need a bigger one – and one that gauges long-term as well as
short-term effects.

In order
to conduct the study the critics want, a direct comparison of balanced literacy
and CKLA, some New York City Public Schools are going to have to adopt Lucy
Calkins’s instructional approach. Simply adopting some aspects, as Chancellor
Fariña suggests, will not be enough. Dr. Calkins will probably even have to
conduct professional development workshops for New York City educators. Ah, the
irony.

Let me
end on a serious note. If we are committed to doing what is best for students
and using education research to inform these decisions, then we need to start
having serious discussions around the data we collect. Using a single education
study to push an agenda or bash a potential competitor is irresponsible, as it
often ignores other important results. Intentionally misrepresenting the study in
an effort to suggest that our children are at risk is nothing short of propaganda and has no
place in the discussion. Let’s make “the study shows” the beginning of the
conversation, not the end of it.

About Me

I am a professor in the Mathematics Department at Grand Valley State University. Mostly, I teach future teachers but I also do some professional development with inservice middle school teachers. My six-word teaching philosophy is: "Agency and capacity fostering sustainable learning."
My wife, Kathy, is a first grade teacher. She is the person who keeps me grounded in educational reality when I begin to get too idealistic. I have also learned a great deal from her about comprehension strategies and instructional coaching.
I have three adult step-children (Hilary, John, and Andrew).