Defining a Pilot Study

Posted on September 21, 2006

Helena Kraemer (bio) shares her thoughts on what should and should not be called a pilot study.

I think the other question that came up - it's come up repeatedly in this group - is pilot studies. My understanding of the word pilot study is that it comes from the pilot boat that leads the ship liner into the dock. In other words, it's a teeny little boat. It isn't the ship liner but it's necessary to actually get that liner where you want it to be. A pilot study is supposed to be a very small study to allow you to basically try out the methods that you are going to use in a big study. What I hear people doing is calling an inadequately powered study a pilot study. And that's not what the word was supposed to mean. My envisioning of what a good [pilot study] and I think they are very valuable is that it's an opportunity to check whether the intervention that you are proposing to deliver, you can actually deliver. It's remarkable how frequently you set up a protocol and then it turns out not to be deliverable and you have to modify it in some way. You have to tweak it.

Whether the measures you are proposing to take, you can really take. You will find frequently that the subjects can't answer the questions that you thought they could answer very easily. Whether you can actually recruit the subjects or recruit them in the numbers you think you can do that. I mean it's not a good idea to start a big study, particularly if you've got funding to do that study, to start that big study without being absolutely sure that you can do all the things you are proposing to do. And the value of a pilot study is that it allows you try things out. The problem with what I've just said is that people want to publish their pilot study and when you use it for tweaking and adjustment and checking things out, it really isn't publishable material. My major concern is this notion that if I do an inadequately powered study and call it a pilot study, it's okay.

No, all it means is that it's an under-powered study and they muck up the literature. So, not a good idea.