‘For every problem there is a solution: neat, plausible and wrong’. H. L. Mencken

Have we discovered it all?

Billions are spent on medical research, but we have entered an era of diminishing returns.

When Andy Burnham, the Health Secretary, admitted last week that he was going to have to “re-prioritise” £60 million of the Government's medical research budget, diverting it to help pay for social care for the elderly and disabled, it seemed a blatant example of robbing Peter to pay Paul. It is self-evident, after all, that today's research will reap dividends in the future, whether through new treatments, or novel ways of thinking about and preventing disease.

But there is something very peculiar about the current state of medical research, which suggests that the Health Secretary's decision is not quite as perverse as it might seem. Funding for such research is now very generous indeed – much more so than is commonly appreciated. Having doubled and doubled again over the past two decades, it is now estimated to be in the region of $100 billion worldwide – more than the GDP of many countries.

The main impetus for this rise stemmed from scientific developments in the early 1980s which permitted researchers for the first time to investigate the nature of common diseases – diabetes, cancer, arthritis and so on – at the most fundamental level, by examining their genetic causes and the inner workings of the cell. These investigations are now pursued in vast citadels of research, in which a dozen or more different groups of scientists will generate billions of megabytes of basic biological data every week, that require supercomputers of prodigious power to store and interpret.

And yet the more generous these financial resources become, and the more daunting the tidal wave of new facts and knowledge, the more striking is the paradox that the rate of medical innovation – in terms of discoveries that really make a difference – was enormously greater 30 or 40 years ago, when the scale of research funding was a fraction of what it has become. That era saw the advent not just of virtually every class of drug in common use today, but of open heart surgery, hip replacements, dialysis, transplants, test-tube babies, and much else besides.

By comparison, while the current research enterprise is certainly immensely productive, its practical benefits are scarcely detectable. As Professor Steve Jones put it in this newspaper earlier this year, writing about our investigations into the human genome, “the mountain has laboured and brought forth a mouse”. Or, in the rather terser words of the journal Science last month, “progress has been slow” and “has not brought with it any floodgates of understanding”.

This seemingly inverse relationship between research funding and the rate of medical innovation seems very puzzling. Is it the case that medical research has entered an era of diminishing returns, in which the successes of the past constrain the prospects for further progress in resolving the more intractable problems that remain? Or is it that the generous funding and priorities of current medical research are in some way inimical to the creativity that fuelled the therapeutic revolution of earlier times?

There are, of course, many pathways to scientific discovery. The current emphasis on laboratory-based research has certainly clarified the basic science of how biology works: the “nuts and bolts” of the genes and proteins. It then makes perfect sense to identify faulty genes and malfunctioning proteins and develop treatments that can correct them – a strategy that has resulted in therapies for Aids and some useful (if very costly) chemotherapy drugs.

But this approach, as Professor Jonathan Rees of Edinburgh University points out, leaves out of the equation all those other unknown factors that might also be involved in causing diseases, and provides little opportunity for stumbling on to unexpected findings. After all, what so often shone a penetrating light on an intractable medical problem was the moment when a researcher uttered what the science fiction writer Isaac Asimov called “the most exciting phrase in science”: “That's funny…”

The two central pillars of modern medicine, and the stories behind their discovery, epitomise the superiority of the surprising and unexpected over methodical, laboratory-based work. Research chemists acknowledge that they could never, in a thousand years, have come up with the magic molecule of penicillin, with its astonishing power to cure infectious disease, from first principles. The same is true of an equally potent wonder drug, the steroid cortisone, which revolutionised the treatment of more than 200 previously untreatable conditions almost overnight.

Instead, it was sufficient that Sir Alexander Fleming should have noted the surprising effects of the penicillin mould in destroying colonies of bacteria on a discarded agar plate – and appreciate its significance. And so, too, for steroids, which were discovered after Dr Philip Hench, an American physician, realised that it was “funny” that his female patients with rheumatoid arthritis tended to go into remission on becoming pregnant – hinting at the existence of some as-yet-unknown hormone with potent therapeutic effects.

And there is nothing in the least exceptional about penicillin and steroids (save for the scale of their benefits). In a systematic review of the circumstances leading up to the cornucopia of medical discoveries in the post-war years, Professor Morton Meyers of the State University of New York found that virtually all came about as a result of some chance, unexpected observation – “like looking for a needle in a haystack and finding the farmer's daughter”.

As he writes in his new book, Happy Accidents: Serendipity in Modern Medical Breakthroughs, “the major anti-cancer drugs were all discovered by independent thoughtful researchers from unexpected sources”. The same goes for the immuno-suppressant drugs that opened the way for transplants of the heart and kidney, the anti-inflammatory drugs that transform the treatment of arthritis, the blood-thinning drugs that minimise the damage caused by strokes and heart attacks, and the psychotropic drugs used to alleviate the symptoms of anxiety, depression and mania.

Professor Rees's own field of dermatology would, he points out, scarcely exist without the twin “happy accidents” of steroids and immuno-suppressants, used in the treatment of a host of skin conditions. Acne, too, whose rampant form at one time had such a devastating effect on the teenage psyche, is now curable by the fortuitous finding that a derivative of vitamin A, tretinoin, inhibits the secretion of oily sebum on to the skin. And psoriasis, whose severe forms used to be treated by several weeks in hospital spent covered in coal tar and bandages, is now treated using ultraviolet light – a therapy prompted by patients' reports of the near-miraculous improvement that followed a fortnight's holiday in the sun.

It is a similar story with the whole range of medical specialities. And as Prof Meyers points out, there is often nothing simple, or even accidental, about the process of turning “happy accidents” into treatments. On the contrary: bringing them to fruition can take years, or even decades, involving several blind alleys, much tenacity and a lot of serious science.

The difference, when compared with conventional research, lies rather in the nature of the discovery itself – as the “surprise”, in being unexpected, necessarily transcends the current state of knowledge. Today's citadels of research, however, deal primarily in the predictable generation of yet more megabytes of data, in expectation that its steady accumulation must eventually lead to those long-promised “breakthroughs”. “Those applying for research grants are expected to have a clearly defined programme for three to five years,” writes Prof Meyers. “Implicit is the assumption that nothing unforeseen will be discovered during this time – and even if it were, it would not distract from the approved line of research.”

There is no reason in principle why these two distinct pathways to discovery should not complement each other. But the multi-million-pound research programmes of Big Science are currently the only game in town. It is a common lament of those with a practical idea, or seeking to explore a fortuitous discovery, that there is simply no point in applying for support from the major grant-giving bodies, such as the Medical Research Council. That's how it is, and it's likely to remain so until the pressure grows to redirect at least some of those generous funds to potentially more fruitful fields of inquiry.