The clock beside my bed changes to read 04.00. The flicker from the glowing digital display is barely perceptible, but it is a change I have become tiresomely familiar with as I struggle to get to sleep. I have been lying awake for about an hour on this particular night after just two hours’ sleep, and there is no sign that I am going to drop off any time soon as my whirring brain ignores my exhausted body. Instead, I am making a start on writing this.

I have always had difficulty sleeping; even as a child I didn’t particularly enjoy being in bed. At university I suffered mild bouts of insomnia and have always been a night owl. In the past nine months, however, my sleep pattern has taken an almost ruinous turn for the worse. I wake up after just an hour or two of sleep and am unable to drift off again. It is infuriating and utterly draining.

During the day, I find myself unable to concentrate and feel fatigued beyond belief. My limbs ache after the most sleepless nights, and I am plagued by headaches. At work my colleagues have begun to comment on my mood swings.

As the Sunday Telegraph science correspondent, however, I decided I was in a position to find out more about my condition. With the help of Prof Kevin Morgan, from the Clinical Sleep Research Unit at Loughborough University, I became the subject of an experiment to find out what was going on when I slept.

Fitted with an actigraph, a watch-shaped device that I wore on my wrist at all times, my daily movements while awake and asleep were monitored for a week. I also completed a sleep diary, describing the quality of my sleep each night alongside the times I thought had I drifted off and woken up at. The results confirmed my fears.

“You meet the diagnostic criteria for primary insomnia,” said Prof Morgan. It is a condition that tends to affect younger people and is not associated with any other condition: many people who suffer from insomnia also have diseases such as cancer, chronic pain or asthma.

“Some nights your sleep is very light, and, on the worst night, you woke up around 4.30am, and there is as much activity as there was during the day two days earlier. You were incredibly restless. You have a very untidy sleep pattern, and there are clear signs you are suffering chronic fatigue during the day after a bad night. You have a sleep efficiency – the time you spend actually asleep while in bed – of about 66 per cent. For a young adult male, that should be about 80 per cent.”

Sadly, Prof Morgan had even bleaker news. “Unfortunately,” he said, “it tends to be a life-long problem. It is more prevalent in people who are under pressure, have lost their jobs, who are hurting in some way, who are depressed, who suffer from chronic conditions, and in older people. Generally, people who are unhappy develop insomnia, and people with insomnia are unhappy.”

Insomnia is the world’s most prevalent sleep disorder and affects about 10 per cent of the population. It costs the NHS more than £36 million in sleeping pills, and the same amount again, at least, is spent on over-the-counter remedies.

The impact on the lives of sufferers can be profound. It can affect their work, home lives and relationships, while also leaving them at risk of depression. Research has also shown that insomniacs are more prone to accidents.

More worryingly, however, researchers are now also piecing together evidence that suggests poor sleep plays a role in a range of serious medical conditions and is even linked to early death.

Prof Francesco Cappuccio, an epidemiologist at Warwick University, has found people who regularly sleep less than seven hours a night are at greater risk of developing high blood pressure, diabetes, heart disease and obesity than those who are well rested. In a recent paper that reviewed studies on 1.3 million patients, he found that those sleeping less than seven hours were more likely to suffer an early death.

“A lack of quality sleep can have huge impacts in the long term,” said Prof Cappuccio. “People are trading sleep to cope with longer working hours and to find more leisure time, but we are finding that it is really not a commodity you can trade without long-term effects on your well-being.

“A hundred years ago the average amount of sleep people managed to get was around 9.2 hours; now in the UK we sleep an average of 6.9 hours. Our findings are showing that about seven hours is the ideal amount of sleep.

“There is a large amount of evidence from the laboratory that, when you sleep less than seven hours a night for a prolonged period of time, you affect the hormone balances that make you more prone to diabetes and affect your appetite. You also affect the low level of inflammation in the arteries, which is one of the prime mechanisms in cardiovascular disease.

“Longer sleep is also associated with health risks, but the cause is far less clear.”

Yet, despite the apparent scale of the problem, insomnia is still relatively poorly understood, often being overlooked as the medical profession and neurologists have focused on other psychological problems. One of the reasons for this is that we still do not have a grasp on what insomnia really is. However, a study by the University of Pennsylvania and Glasgow University is beginning to shed light on what might be going on.

“It seems to be far more complicated than people being either awake or asleep,” said Prof Colin Espie, director of Glasgow University’s Sleep Centre. “People with insomnia may be asleep, but their brains are more active. When you wake them up from this light sleep, they believe they were awake and can tell you what they were thinking about. When you look at the microstructure of the brain patterns while they are asleep, at times their brains look more like one that is awake.

“It appears that people with insomnia are experiencing some of their sleep as wakefulness because they are aware of a lot of mental activity, which is, in fact, underlying this.”

The strange blur between wakefulness and sleep combines with another factor in insomnia that is now emerging as the overriding cause of the problem – worry.

Prof Espie said: “Our model sees insomnia as a disorder of selective attention that inhibits sleep. It is as if insomnia is a form of sleep stuttering – it is a disorder where the natural sleep process becomes inhibited by worry and concern about getting to sleep or quality of sleep.

“We are finding that people with insomnia are far more affected by words and pictures associated with sleep, such as beds and bedside tables, than those who have normal sleep patterns. As with speech, if you don’t think about it, it is easy. If you think and analyse every word, you can become incomprehensible.”

Those who suffer from insomnia will have some form of predisposition to the condition. Some researchers are noticing strong trends within families that are hinting at a hereditary link, but no one has yet found the gene or genes that may be responsible.Prof Morgan believes insomnia sufferers are often found in professions that attract individuals with a certain type of mind. “They have a particular personality that makes them at risk. They tend to be people with a certain way of thinking. They are very focused and obsessional, which means they have trouble switching their brains off.”

But not everyone with this predisposition will suffer from insomnia: it takes a trigger to set it off. In my case, the worsening of my sleep problems is clear. My personal life has undergone considerable turmoil in the past nine months, and 2010 has probably been the most challenging year of my life. I found I was unable to switch off my whirring brain at night. And the less I slept, the more I worried.

Psychologists now believe that this worry leads to an underlying preoccupation with sleep that can prevent insomniacs from dropping off. Rather than drifting peacefully off to sleep, they fret about the prospect of not sleeping. The way to get round this is to train sufferers to learn how to sleep again.

Prof Morgan will this week launch a new campaign to roll out a form of cognitive behavioural therapy across the NHS through the Improving Access to Psychological Therapies programme.

“Tablets are not the answer,” he said. “You have to relearn to go to sleep in your bed. We can do this by implementing a regime of sleep restriction so that you try to sleep only between certain hours and also by controlling the stimulus so that you learn to associate your bed and bedroom only with sleep.”

Prof Espie agrees. Over the next year he will be leading the biggest ever survey of British sleeping habits in an attempt to piece together new ways of promoting good quality sleep. Along with Boots, he has set up an online study – The Great British Sleep Survey – to analyse sleeping patterns.

As for me, I still haven’t managed to fall asleep while writing this. But, perhaps if I stop worrying about it, I might finally drift off.

* If you would like to have your own sleep assessed and to receive tips on how you can improve you sleep go to The Great British Sleep Survey at http://www.sleepio.com/