Tim Lott goes under the clippers at his brother's Soho barber shop. Photograph: Suki Dhanda for the Observer

It is some time now since I started to worry about baldness – somewhere between the retreat of the already fine hair at my temples in my early 30s and the final failing of the last growth of hair at my crown a few years back.

I had been trying to convince myself that things might not be too bad for the past 20 years. But at the beginning of this year, at the age of 55, an encounter with a ceiling-mounted mirror revealed to me what was doubtless obvious to others – a monkish, thinning crown. There was no longer any doubt about it. I was definitely more bald than not.

My wife, Rachael, wanted me to take it all off and be done with it. It was an option that made me nervous. My brother, Jack, a professional hairdresser for 20-odd years, advised me to hold on to what little I had. He had witnessed many times the shock, usually unpleasant, that men felt when they finally did clip or shave their hair.

I retained a sentimental attachment to what remained of my hair. After all, it had once been my pride and joy. In my teenage years, during the summer, it was cornstalk yellow, and I wore it long and wild. I considered it to be one of the few effective items of mating display available to me, and its relentless disappearance was a matter of grave regret.

But regrets were not going to get my locks back. So, against the advice of my own brother, I turned up at Jack's salon, determined, at last, to go for The Chop.

I may be ONE OF THE LAST generation of men who face this dilemma. In December last year, scientists at the Berlin Technical University revealed they had grown the world's first artificial hair follicles from stem cells. The leader of the research team claimed that within five years millions of hair-loss sufferers could grow new hair from their own stem cells and have it implanted into their bald spots. In January this year a study by the University of Pennsylvania suggested that bald men were not bald at all – it was simply that their stem cells were producing growths too fine to be visible to the human eye. According to the team leader, Dr George Cotsarelis, "The fact that there are normal numbers of stem cells in a bald scalp gives us hope for reactivating those stem cells."

Then, in February, researchers at the University of California, Los Angeles found that a chemical called Astressin B showed "astounding" results for hair regrowth after one jab per day for just five days. The tests were on mice, but the researchers were confident that a cure for baldness could be found in five to 10 years.

Finally, in March, a British company called Nanogen announced that their scientists had pioneered a brand new technology to combat hair loss. The new technology had "been designed to improve hair growth and even re-awaken dormant hair follicles". The company claimed to have developed a new "growth factor complex" (sh-VEGF) which would stimulate hair-follicle growth on balding scalps.

Taken together, this seemed to herald a revolution. Over the years there has been no shortage of unrealistic headlines promising an end to baldness – but most of this new research seemed to emanate from respectable academic institutions, and all of them seemed to be promising the same thing – the End of Bald.

Mopping up? ... James Nesbitt before and after his hair transplant. Photograph: Ken McKay/Richard Young/Rex Features

Even if the research proved to be disappointing in the long run, there appeared to be other developments in the world of hair repair taking place, too – not in biochemistry, but in the technology of hair weaving and transplants.

Actor James Nesbitt spent tens of thousands of pounds on what appeared to be a remarkably successful hair transplant. "They've changed my life. It's horrible going bald. Anyone who says it isn't is lying," said Nesbitt. Instead of the pitted sprouting potato look seen on many unfortunate recipients of hair transplants, Nesbitt's hair looks convincing enough to help land him – he believed – major new roles that he would otherwise have been denied.

The success of the transplant doubtless had something to do with the amount of money he could afford to spend on the job, but in the past hard cash didn't always solve the problem. "All that money and he's still got hair like a dinner lady," spat Boy George of Elton John, who has appeared to unsuccessfully confront his receding hairline with various ineffectual treatments over the years.

But this time, the transplants were actually pretty convincing. What was going on? There were rumours that a new hair-transplant technique – FUE, or Follicular Unit Extraction, which transplants follicles from the back of the head one by one instead of in a long strip – used robot technology to enable thousands of follicles to be replanted at once, thus producing a more sophisticated and convincing result. Perhaps a new crop of hair could be bought, right now, without having to wait for genetic science to take the necessary leap forward.

At my brother's hairdressing shop in London's Soho, I paced the floor. I'd spent the past week in a sunny clime gaining a tan in an attempt to minimise the impact, but listening to Jack revving up his clippers made me antsy in the extreme.

I lowered myself nervously into the chair. This would be the end of all choice – other than one, two, three or four, the settings on Jack's clippers. A "number one" was the bonehead cut, the number four the "suede head". I decided to go for the most modest option, the number four.

As I watched the hair cascade from my scalp, I was surprised how much of it I still had left. There was still bulk at the sides and the back, but that was now disappearing in clumps on to Jack's floor. I watched with a combination of fascination and anxiety. The procedure didn't seem to require too much artistry – clip, flip, drop, buzz, zip. Jack started with a channel down the centre of the head, just for fun I think, so I would look genuinely like a 50s mental patient, then took the rest off. The process probably lasted no more than 20 minutes. I kept asking Jack to change the music to something more calming.

Then it was over. Jack smiled, and dusted the back of my neck with talcum powder. I ran my hand over my head to feel the sharp, angry stubble, checked in the mirror then went into the bathroom and checked the mirror there.

How did I feel? I felt happy.

I had never felt so clean, and so… straightforward. It was no nonsense, it was real, it was me. I liked it.

I suddenly had no idea what the fuss about being bald was all about.

Why does baldness matter so much to men? The thinker and notable bald person, Alain de Botton, recalls his own grief when his hair began to disappear in his late teens.

"It was very distressing and frustrating," he says. "It stood in the way of being confident about myself and undermined my sense of how attractive I was to women. Even at the age of 40 I still think of it as an unfortunate thing in my life. But I see it as being almost analogous to a disability. It isn't simply vanity – most people do look worse without hair."

De Botton also notes that it is not subject to social inhibition in the way other shortcomings might be. "It is completely acceptable to laugh at a bald person in a way that it simply isn't at a fat person."

His distress is not unusual. Stories of men – and women – becoming suicidal after hair loss are not uncommon. This is not just about looks, but mortality, the passage of time.

"It is a paradigm of ageing, a signal of loss of control over our bodies on a continuum with, say, losing the ability to stand up or the loss of a faculty," says De Botton. "We suddenly become describable as just an old biddy or an old fool. Baldness quickly swallows up a person, like 'just' a fat person or an ill person."

Murray Healy, journalist and author of the book Gay Skins: Class, Masculinity and Queer Appropriation, points out that losing hair naturally is "seeming to fail, which is a bad thing. Thinning hair is, in a sense, the equivalent of a 'failed crop'." By which he means a failed agricultural rather than tonsorial crop.

He suggests unhappiness at not having hair is to do with a sense of shame, rather than any objective reality; and that if one is bold and does nothing to conceal the loss, and in fact emphasises it by shaving, it can signify the opposite of failure – confidence in one's own masculinity. This is a fairly recent phenomenon, which can be traced back to the hyper-masculinity advertised by the original skinheads in the 60s and 70s, a hairstyle which was then appropriated and lionised by the gay community in the 80s and 90s and turned into the opposite signal.

By the time Bruce Willis and David Beckham started to crop their hair, shaven-headedness had gone Mainstream Hetero. The white-van man and metrosexual alike, the baldness sufferer and the style surfer, were all flocking to have their hair clippered to the minimum length. "Celebrities shaving their heads brought a different interpretation to baldness entirely," says Healy. "From when Trevor Sorbie started to makeover men with cropped hair on daytime TV in the early 90s, the meaning changed."

But what about those who are not convinced by Trevor Sorbie or Bruce Willis? Is there really hope for them, with the new research discoveries and fresh techniques for hair transplantation? Probably not, according to GP and medical journalist Dr James Le Fanu, author of The Rise and Fall of Modern Medicine.

"You have to be incredibly sceptical about stem cells and anything where people say in five years' time we'll do such and such," says Le Fanu. "Stem-cell technology appears to offer the moon. Regeneration and regenerative medicine is often presented as the answer to everything in the universe. But actually there's nothing in the bank."

Balding, Le Fanu points out, is deeply mysterious, even counter-intuitive. "Nobody knows why you get balder as you get older. It's an anomaly. It doesn't fit." Because, Le Fanu explains, people with lower levels of testosterone (which usually comes with ageing) should theoretically enjoy better hair growth. After all, castrati never lose their hair, because they don't produce testosterone. "There is some link between hair loss and testosterone, but no one knows what it is."

Le Fanu isn't all negative. He claims that hair transplants are pretty good nowadays and recommends drugs such as Regaine (Minoxidil), which induces hair growth in 30-40% of people. He notes that "increasing circulation to the scalp also appears to promote hair growth. The Japanese have a special hair brush which they use to massage the scalp 200 times a day to increase blood flow. This is said to improve hair growth."

Le Fanu, though knowledgeable, is not a hair-loss specialist. The closest I could get to an independent expert, Dr Bessam Farjo, is medical director of both the Institute of Trichologists and the Farjo Medical Centre in Manchester, which specialises in hair transplants. Farjo respected calls for scepticism, but was cautiously optimistic that real progress is being made.

"The importance of George Cotsarelis's research is in showing that stem cells are not lost when people lose their hair," says Farjo. "It's just that a certain kind of cell – the progenitor cell – is missing. If the hair is still there and the cells are still there, the hope is that the cell can be 'kick started' to produce new hair growth.

"The UCLA experiments with Astressin B underline the case for linking hair loss with stress – the mice in the experiment lost hair when put under stress and regained it when the Astressin B was introduced. Furthermore, the German research is the first time to my knowledge that artificial hair follicles have been grown in a laboratory. This could be very promising."

The "growth factor", sh-VEGF, that Nanogen claims to have discovered, and which can supposedly "reawaken dormant hair follicles", is also significant, according to Farjo. "A growth factor sets off protein in the body that triggers various processes... they flow in the body naturally. But a growth factor can also retard growth as well as promote it. The research is all about trying to figure out how to introduce more of the positive growth factors."

Farjo emphasises that "curing" baldness – which, after all, is completely natural and therefore doesn't require a cure – is still a long way off. "We definitely have a lot more information now than five or 10 years ago – but curing baldness could, in the long run, be as difficult as curing cancer."

Farjo recommends hair transplants for those seeking a quick solution – which, he says, have greatly progressed. However, the idea that James Nesbitt has been the benefactor of a new robotic FUE technique is simply wrong. Nesbitt had a conventional "strip harvesting" process. He was just lucky, had a good surgeon, and it took well.

"Robot technology has simply not been achieved yet to the level where it can be used," says Farjo. "And there is no real difference in quality between strip harvesting and FUE – it's just a matter of how serious a scar you leave at the back of the head."

Talking of scars – psychological rather than physical – the big test for me was going home after my crop. It was, after all, my wife who had nagged me for years to get my hair taken off and, as usual, I cravenly sought her approval.

Somewhat predictably, for the first day, she didn't even notice. I had to resort to blocking her off in a corridor and asking her the pointed question: "Well?" while raising my eyes to the crown of my head.

"Well what?" Rachael said, scanning me abstractedly, to work out what exactly I was talking about.

"Well?" I said, even more pointedly, lowering my head slightly so she might be able to get a better clue.

"Oh, your hair!" she said finally. She grimaced, then said, "I was hoping it would be shorter." Then she rushed off to an important appointment.

Clearly she was unimpressed, but then I was sure the style would grow on her – so to speak. Sure enough, when she returned from her appointment, she examined me critically, as if for the first time, and smiled. She clearly loved it.

"You look like a cancer patient," she said, cheerfully, and went to make some tea. Perhaps a comb over wouldn't be such a bad idea after all.