Is there a "Cranfield paradigm"?

So I sat down this morning to finally put my discussion of the history of the Cranfield tests and of the phrase "the Cranfield paradigm" out of its misery by asking in what sense it is appropriate to talk of "the Cranfield paradigm". Fifteen-hundred words later and only half way through, I realized that I didn't want to have to re-read what I'd written, let alone write the rest, which I took as a pretty good indication that no-one else would want to read it, either. I'll try therefore to give an executive summary here.

The concept of a "paradigm" was introduced into the theory of science by Thomas Kuhn in his 1962 book, "The Structure of Scientific Revolutions". For Kuhn, scientific disciplines start in a pre-paradigmatic stage, with lots of competing schools and no agreed fundamental theory. In this stage (which, for instance, most social sciences are still in), individual schools can progress, but there can be no progress for the scientific field as a whole.

They, some watershed theory manages to unite the discipline, and it enters its paradigmatic stage. In this stage, the field spends long periods undertaking normal science, where information is steadily accumulated along the lines established by the fundamental theory. However, if sufficient anomalies in the fundamental theory mount up, then the field goes into a state of crisis. This crisis is resolved when, in an act of extraordinary science, a new fundamental theory is developed. Then all the old findings are recast in terms of the new theory, and normal science can resume.

A paradigm has many aspects, but as conceived by Kuhn (in 1962 -- he went wishy-washy later), it is most essentially a special kind of theory, what we have been calling a "fundamental theory", one which explains and predicts observed phenomena. We can therefore say that "the Cranfield paradigm" is most certainly not a paradigm in the 62-Kuhnian sense, because it does not establish any fundamental explanatory theory of the phenomena observed in information retrieval. On the contrary, Cranfield took the existing, competing theories of the day, found them all unsatisfactory, and left the field with a heuristic, black-box approach, representing at best a descriptive form of science and at worst no real science at all. (Everyone seems to be agreeing at the moment that I'm an expert on what real science is...)

However, Kuhn's categories are enlightening in some respects. We can certainly see information retrieval pre-Cranfield as being in a kind of pre-paradigmatic stage, with many competing theories all working off speculative, dogmatic, or anecdotal bases (see Robertson's 2008 essay, "On the history of evaluation in IR" for more on that). And the Cranfield tradition did ultimately lead to settled methodology (though not until quite some time had passed, as will be argued below). So we could say Cranfield gave the field the forms of a paradigmatic science without its content.

In my original draft, I then launched into an examination of a 1992 essay by David Ellis on "The physical and cognitive paradigms in information retrieval research"; this examination produced many words but no real conclusions, so I'll omit it here.

The next step was going to be Ellen Voorhees's 2001 CLEF paper, "The philosophy of information retrieval evaluation", which introduced the phrase "the Cranfield paradigm" to its current popularity. Voorhees uses the phrase multiple times in her paper, but her first mention is in the modified form "the Cranfield evaluation paradigm". Having (mercifully) stopped myself before I wrote this section, I can't tell you for certain what was going to be in it, but if I was forced to speculate, I'd say that I was going to say something along these lines:

The phrase "the Cranfield paradigm" is too glib, and the connotation is too broad. Speak, if you must, of "the Cranfield evaluation paradigm". However, the test collection methodology first introduced in a recognizable form at Cranfield, and then elaborated by the SMART project and elsewhere, was far from as dominant before TREC as one might suppose. In 1981, Sparck Jones counted only a couple of dozen laboratory IR experiments altogether to that date, not all of them using the test collection model, and more of them before 1970 than afterwards. Methodology was still quite fluid during this period. Jean Tague's 1981 article on "the pragmatics of information retrieval experimentation" runs for 44 pages, and identifies 10 separate decisions the researcher must make in running an experiment -- hardly an indication of a field working under the firm guidance of a dominant methodological paradigm.

By 1992, moreover, the test collection model had substantially fallen into disuse, the SMART project aside, and everyone was clamouring for user-centered studies. Of course, that was not to be; TREC was founded that year; and since then, IR experimentation has been static test collection after static test collection, paved from here to the horizon. But this is not the Cranfield paradigm; this is the TREC paradigm; and calling it the Cranfield paradigm implicitly rewrites history. It makes it sound as if IR evaluation has headed in one direction since 1957, when in fact it was beginning to wander all over the place, before TREC came along and placed its stamp on the field.

There, done. I thank everyone for their patience. I think I have now finished with this topic.

This entry was posted
on Wednesday, September 9th, 2009 at 9:46 pm and is filed under Uncategorized.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.