In my ongoing search to better understand how we reconcile the creative tension between subjective and objective measures of the world — including our ongoing (and thus far) elusive search for a better way of tracking how people learn — I took note of a recent New Yorker article that cast light on some emerging problems with the ostensible foundation of all objective research — the scientific method.

In the article, author Jonah Lehrer highlights a score of multiyear studies — ranging from the pharmaceutical to the psychological — in which core data changed dramatically over time. Drugs that were once hailed as breakthroughs demonstrated a dramatic decrease in effectiveness. Groundbreaking insights about memory and language ended up not being so replicable after all. And the emergence of a new truth in modern science — the “decline effect” — cast doubt on the purely objective foundation of modern science itself.

Without recounting the article in entire, there are several insights that have great relevance to those of us seeking to find a better way of helping children learn:

In the scientific community, publication bias has been revealed as a very real danger (in one study, 97% of psychology studies were proving their hypotheses, meaning either they were extraordinarily lucky or only publishing outcomes of successful experiments). The lesson seems clear: if we’re not careful, our well-intentioned search for the answers we seek may lead us to overvalue the data that tell us what we want to hear. In the education community, how does this insight impact our own efforts, which place great emphasis on greater accountability and measurement, and yet do so by glossing over a core issue — the individual learning process — that is notoriously mercurial, nonlinear, and discrete?

In the scientific community,a growing chorus of voices is worried about the current obsession with “replicability”, which,as one scientist put it, “distracts from the real problem, which is faulty design.” In the education community, are we doing something similar — is our obsession with replicability leading us to embrace “miracle cures” long before we have even fully diagnosed the problem we are trying to address?

In the scientific community, Lehrer writes, the “decline effect” is so gnawing“because it reminds us how difficult it is to prove anything.” If these sorts of challenges are confronting the scientific community, how will we in the education community respond? To what extent are we willing to acknowledge that weights and measures are both important — and insufficient? And to what extent are we willing to admit that when the reports are finished and the PowerPoint presentations conclude, we still have to choose what we believe?

Like everyone else who does education for a living, I read that Michelle Rhee is launching a new national advocacy organization, Students First. And after checking out the site and hearing how she articulates its purpose, I see some reasons to feel hopeful — and many more reasons to feel deeply concerned.

First, the good news: It’s hard to argue with Rhee’s four “we believe” statements for the organization. Who doesn’t believe all children deserve great teachers? Who would argue with the idea that students should not need luck to get a good education? Why not start allocating public dollars where they can make the biggest difference? And who would deny the need for more parental involvement and increased efforts to engage the entire community? So let’s all hop on the Rhee express, right? Well, maybe.

The Times article had nothing to say about school reform — it was about the Fed’s inability to decide whether to stimulate the economy now or later. And it was about how even in a social science flush with quantitative data, the “social” aspect of the science — i.e., human behavior — is sufficiently complex and nonlinear to make certainty a chimera. “One point I always make to my graduate students,” said Robert Solow, a Noel Prize winner and MIT professor, “is never sound more certain than you are.”

Would that such caution were commonplace in our current conversations about education reform!

Of course, the message is not that economics is a boundless free-for-all discipline that uses numbers to hide its own guesswork — charges that are sometimes made to rebut the growing push in education circles to embrace a greater use of student information to guide adult decision-making — but one message seems clear: beware the worship of “data” in your search for certainty, as long as human beings are part of the equation. “The entire question of how emotion will change people’s behavior is pretty much outside the standard model of economics,” said Dan Ariely, a professor at Duke. “Pride is not in the model. Fear is not in the model. Revenge is not in the model. Even simple things like disenchantment of people who are fired from their jobs — the model doesn’t account for how devastating that experience can be.”

In case you missed it, Steven Brill wrote a relatively balanced piece in this weekend’s New York Times Magazineabout the national education reform landscape — and how teachers unions are truly facing a sink-or-swim moment of reinvention.

As someone who feels neither allegiance nor antipathy toward either of the increasingly polarized camps (I actually like and respect both Linda Darling-Hammond and Jon Schnur), I see the partial truths in each side’s argument. On one hand, for example, it’s clear that K-12 teachers should not be granted lifetime tenure so easily — tenure, after all, was originally designed to protect the free-expression rights of college professors, and since the First Amendment barely even applies to public employees anymore, that point is moot. So I say bring on this reform.

It’s also clear that teacher evaluation systems need to be dramatically retooled. When educators can only be scored ‘satisfactory’ or ‘unsatisfactory,’ that’s a huge problem. So again I say yes to any reform that results in a new system that creates a reciprocal flow of feedback and helps educators improve the quality of their professional practice.

However, I see one massive problem — and it’s a problem that no one, Brill included, seems interested in addressing: Everyone wants to tie these new teacher evaluations to student performance data, but no one wants to talk publicly about the fact that we lack sufficient metrics for truly evaluating the full extent of whether or not young people are learning and achieving at high levels.

As I’ve written many times before, basic-skills tests in reading and math provide a single, useful proof point that is, in the current climate, dramatically overvalued. To help students learn to use their minds well, schools — and teachers — need to focus on not just basic- but also higher-order skills; they need to engage children in not just reading and math but also the arts and sciences; and they need to focus less on just data per se, and more on how well we’re equipping our teachers to respond to data in ways that improve the overall learning conditions (and outcomes) for kids.

A system, therefore, that incentivizes compensation and job security by using a single measure to count for as much as 50% of an evaluation will incentivize — you guessed it! — a relentless focus on basic-skills reading and math scores. But that’s not enough if we want the achievement gap to mean more than test scores. And more people need to start calling it out.

The good news is that although we may not yet have these more sophisticated metrics in place, at least we know what we should be looking for. This month, I’m finalizing the manuscript of a book that brings together 50 powerful stories about teaching and learning — selected from the many hundred that have been submitted by people across the country as part of a national campaign. The stories recount a wide range of experiences — from third grade classrooms to Outward Bound courses to church missions to prison sentences — but what they combine to make visible are the core conditions of a powerful learning environment. (See for yourself at rethinklearningnow.com.) And although they do not reveal a simple, single answer to the deeply complex question of how to improve our schools, they do clarify the question we should be asking ourselves at every turn — How do we create more challenging, engaging, supportive, relevant and experiential learning opportunities for children?

Imagine if more people started asking that question. Imagine what a new statewide teacher evaluation system would need to look like in response. And ask yourself — would you want that sort of environment for your child?

This Thursday marks the prime-time return of the NFL Draft — an annual smorgasbord of possibility when each team fills out its roster with the best talent the college ranks have to offer.

I’m a huge football fan, so I’ll be tuning in to see which players my beloved San Diego Chargers select to fill our current holes at running back and defensive tackle. I’m also a huge public education fan, so I confess that I wish the leading voices in my field — from Arne Duncan to Michelle Rhee to Joel Klein — would also tune in, and heed some of the most relevant lessons to be learned from the NFL.