After about 18 years of living in the city of Cambridge, Alexander Masters is experimenting with living in the country, which means more time in a car, which means, "because of my old man-ish ways", audio books, which at the moment means Trollope's Phineas Redux scattered all over the floor in front of the passenger seat. It was Dickens for a while, but he came a cropper with Oliver Twist. He can't stand the sentimentality of Dickens's good characters. "Die, Oliver, die!" he says, barrelling down the streets of Cambridge, looking for a parking space.

This roughhouse anti-sanctimony is one of the things that saves his biography of a chaotic, violent, ex-homeless man called Stuart Shorter from sentimentality. More than saves it, in fact: makes it counter-intuitively respectful and invigorating. Stuart: A Life Backwards has been shortlisted for the Samuel Johnson Prize, the Whitbread biography prize and optioned for TV by Sam Mendes's company. Last night, it also won the Guardian First Book Award.

One gets a sense that the recognition both surprises Masters - who, during the four-and-a-half dogged years while he was writing it, rarely earned more than his subject - and validates a confidence he had all along. But it also reminds him that there is something - someone - missing: Stuart himself, who walked in front of the 11:15 from London to King's Lynn on July 6 2002. It was not entirely unexpected: the first time they met, when Masters was researching an article on the homeless for a local Cambridge paper, Stuart, scrunched in a doorway near Sidney Sussex College, announced he wanted to kill himself. But it had to look like murder because, he said, with a typical mix of sensitivity to others and extreme self-destructiveness, "Me brother killed himself in March. I couldn't put me mum through that again. She wouldn't mind murder so much."

Stuart, as Masters subsequently discovered, had ample reason to feel as he did: an absent, violent father, whose only real legacy was muscular dystrophy, which made Stuart a target for bullies; a ruinous stint at a special-needs school, where he was sexually abused; rape from his brother and babysitter; a talent for psychopathic violence; alcoholism; heroin addiction and a prison record so long and tumultuous Masters simply lost track ... but also a gentle, perceptive side, and a natural flair for the brutally accurate sentence. But putting it on the page proved more difficult than Masters bargained for.

There were obvious traps: condescension, which Masters mostly - mainly by letting Stuart deflate him - escapes, and pontification, of which he has a horror. "I think it just hides a lot of stuff," he says over lunch in Browns. "If you start ... trying to play on the heartstrings, you actually lose people." So he adopted deliberate methods of puncturing windbaggery, such as letting Stuart speak for pages on end, capturing both his cadences and his illogical disruptiveness. Masters put signs up on his wall, as he was writing, that said things such as: "You are not interesting." And he used humour: it's a tale of rampant misfortune, but it's also very funny. "Well, Stuart was very funny," he says. "And I think it's important to try and convey a sense of the situation. It's an exaggerated life, on the humourous side as well as the romantic side, or the tragic side. All these elements are exaggerated, magnified. And Stuart often found himself in ludicrous situations, or we would find ourselves in uncomfortable situations, upsetting situations, where there was no way of release, no way of making sense of it except to be ridiculous about it. That really worked. I hate saying that sort of thing - what can you do but laugh? - but it is a bit of that."

Masters and Stuart first met in 1998, though they did not begin to get to know each other properly until the following year; by the time Stuart died, they had become friends as well as biographer and subject. Masters admired the determination it cost Stuart to drag himself to the drug dependency unit every week, despite the increasing dystrophy-related weakness of his legs; Stuart admired Masters's determination to disentangle his life. Stuart taught Masters how to make convict curry and prison hooch, and in one surreal scene, Masters took Stuart to a house belonging to grand friends of his in Norfolk, where Stuart was allowed to use a ride-on lawn mower and assault the giant hogweed. "This afternoon is one of the few times I have seen Stuart unreservedly happy."

Masters gave Stuart an early draft of the book - scientifically thorough, buttressed everywhere by studies - and Stuart pronounced it "bollocks boring". What he wanted was "a bestseller, 'like what Tom Clancy writes ... something what people will read ... Make it more like a murder mystery. What murdered the boy I was?' " Hence Stuart: A Life Backwards, which Stuart did not live to see published.

Another objection Stuart had was to Masters' need to pin down and explain Stuart's Stuartness: pointless, he thought, when he didn't understand it himself; doubly pointless when they came from such different backgrounds.

Masters was born in New York and can, if he wishes, trace his lineage back to the Mayflower; his father, Dexter, established the Consumer's Union (the American version of Which?) and was a friend of Alger Hiss. He was a novelist, too, as is Masters' mother, Joan Brady. The family left for Paris when Dexter Masters came under scrutiny from McCarthy's FBI, but never made it, ending up in Totnes, Devon, where Alexander had a "really, really nice childhood. It was completely the opposite of Stuart's. It was lovely. I even enjoyed school." And the fact that his parents wrote - "I can't overestimate how valuable that is. It gives you a sort of presumption, and it makes you realise the way you need to work. How hard you need to work. If you decide you're going to do it, then you do it all the time. It's not something you dally with on a Sunday, it has to be your life."

But he did not actually write for a long time. Bedales was followed by a degree in physics at King's College, London. It was an unhappy, lonely time, partly self-inflicted: wanting to learn Russian, he took a room with a Russian landlady in Gants Hill, about as far from college and other students as he could be. He had artistic instincts, but did not like arty students; his last year was spent not in class, but "wandering around London, scuffing my heels". Somehow, with the aid of other people's notes, he got a first and went to Cambridge for a further degree in maths, and then the beginnings of a PhD in the philosophy of quantum mechanics. He obviously idolises mathematicians and physicists, who do what he considers to be real things - "the magnitude of the problems they were tackling was so exciting" - and a kind of mathematical clarity underpins his laconic, expletive-strewn prose.

At Cambridge, he was happy again, throwing himself into the social life - though not into the work. He began a relationship with a don at the same college, Dido Davies, "which was nice and scandalous." Although they are no longer together, Davies, the author of a well-received biography of William Gerhardie, is still his first and strictest reader. They had collaborated before, in a series of guides Davies wrote under a pseudonym, Rachel Swift, with titles such as Satisfaction Guaranteed: Or How to Be the Sort of Man Every Woman Wants in Her Bed. On one website, Masters is described, amusingly, as an "independent sex researcher".

When he ran out of money, he went to work for a rough sleepers' day centre, Wintercomfort, whose director, Ruth Wyner, and her deputy, John Brock, were famously imprisoned for supposedly allowing heroin dealing. Masters ran the campaign that achieved their release, and he and Stuart began to work together, doing public appeals, and stunts such as a three-day occupation of the pavement in front of the Home Office (Stuart's idea). Masters became "aware how often I'd take his stories back with me and tell them to my friends. And I thought, well this is ridiculous, I'm getting all this material and wasting it - I'm not recording it."

He knows that the book and its many gory details "caused a lot of disruption in Stuart's family - though not to Stuart's mother and sister, who were always very strong-minded. They've been through a hell of a lot. But the rest of the family - Stuart was a disgrace to them. They didn't want the book to be done, and who can blame them? But Stuart was adamant." Which did not mean it was easy for Stuart. If Masters picks the wrong day to talk to Stuart about being raped, for example - if it is not "at the end of an untroubled week, when he has money to spend from the dole, so there's a distraction to look forward to in the evening" - Stuart "drinks, cuts himself, and then injects citric acid".

I wonder if Masters thinks his questions ever made Stuart worse, maybe pushed him over the edge? "I don't know," he says. "It has occurred to me. He was questioning himself all the time, anyway, but that doesn't get me out of it, because I was focusing the question and giving it a sort of a push. And there were times when it was uncomfortable, and he'd have to tell me to stop. It could have made him worse. I hope it didn't. I don't think I would have gone on had I felt it was making him worse".