A practice that is often seen in reports of randomised trials is carrying out significance tests on baseline characteristics, in the belief that this will provide useful information.The main reason for significance tests is to test whether the null hypothesis is true, and it is this that motivates testing of baseline characteristics.Investigators want to see whether there is a “significant” difference between the groups at baseline, because they have been brought up to believe that a “statistically significant” difference is a real difference. [I’ll leave aside the logical fallacy in deciding on the truth or otherwise of the null hypothesis based on a p-value – see other posts]. Obviously, with baseline characteristics in a randomised trial, this is pointless, because you already know that the null hypothesis is true i.e. on average there are no differences between the randomised groups, and any differences that are seen are due to chance.

Significance testing of baseline characteristics has been extensively criticised; for example the CONSORT guidelines say:

“Unfortunately significance tests of baseline differences are still common….Such significance tests assess the probability that observed baseline differences could have occurred by chance; however, we already know that any differences are caused by chance. Tests of baseline differences are not necessarily wrong, just illogical. Such hypothesis testing is superfluous and can mislead investigators and their readers.”

“For tables comparing treatment or exposure groups in a randomized trial (usually the first table in the trial report), significant differences between or among groups should be indicated by * for P < 0.05, ** for P < 0.01, and *** for P < 0.001 with an explanation in the footnote if required.” [my bold and underlining]

That is a pretty surprising thing to find in a top journal’s instructions, especially as the next point in the list says that “authors may provide a flow diagram in CONSORT format and all of the information required by the CONSORT checklist”.

The wording of the CONSORT guidance is less than ideal and I hope it will be changed in future revisions.It says “Significance tests assess the probability that observed baseline differences could have occurred by chance…”.This seems a bit misleading, as this isn’t what a p-value means in most cases, though it is more correct for comparisons of baseline characteristics in a randomised trial.The p-value is the probability of getting the data observed (or a more extreme result) calculated (and the significance test performed) if the null hypothesis is true i.e. it is based on the assumption that there is no difference.Obviously it can’t also measure the measure the probability that this assumption is correct.

July 25, 2008

This diary is partly for my own amusement, but also to record the enormously long drawn-out process of setting up and running a large scale randomised controlled trial.

We (which is the Warwick Medical School Clinical Trials Unit) were lucky enough to be awarded funding a few weeks ago by the NIHR Health Technology Assessment Programme for a randomised controlled trial of the LUCAS mechanical compression device for cardiac arrest. Briefly, this is a mechanical device for providing chest compressions for people who have had a cardiac arrest. Its suggested advantages over standard manual compression is that it doesn't tire so the quality of compressions remains constant, it can operate in situations where a paramedic wouldn't be able to do manual compressions (such as when a patient is being moved) and it frees up paramedics to do other important jobs, like saving other people. But there is no evidence that it actually does save peoples' lives and it's also possible that it kills them; for example, it might cause internal injuries that cause more people to die later on. Hence the need for the trial, which we are doing in collaboration with Coventry University, Leeds University and the West Midlands and Scottish Ambulance Services.

We were awarded this grant at the end of June 2008 but the project was already over a year old by then. The original idea to do this trial came up in meetings towards the end of 2006 and we submitted an outline proposal in May 2007. Just after we had done this there was a call for proposals in the field of Emergency and Prehospital care, and the funding body suggested to us that we should move our application into this funding stream, which we agreed to. However, it became clear that there was another application for a trial of the same device so we had some direct competition. As a result of the outlines we were invited to submit a full application, which went in in February 2008.

So we are now finally at the point of having the money and having to deliver this trial.