It's always tricky interviewing a nice man. First, there's the problem that, not being particularly nice oneself, it's hard to know what makes him tick. Then there's the little matter of copy. I interviewed Jonathan Miller once and his enmities could have filled a book. I threw away my list of questions and just watched as the machine gun mowed down a lifetime of enemies. Ferdinand Mount, novelist, journalist, former head of Margaret Thatcher's prime ministerial policy unit, former editor of the Times Literary Supplement and quintessentially nice man, is going to be harder work.

He knows it, too. At one point he apologises for the lack of edge in his answers: "I've said a lot or un-urgent things to you, I think," he says, explaining why these days he prefers writing books to the mock-urgency of the newspaper column. This, though, is partly my fault. I was enjoying reading his new memoir, Cold Cream: My Early Life and Other Mistakes, so much that I didn't have time to draw up anything like a systematic interview script, and am even more halting and discursive than he is. I console myself with the thought that it can't be any worse than the time in the mid-1960s, recounted in hysterical detail in the book, when he interviewed Edward Heath drunk. (Mount was drunk, that is.) "I didn't realise this was going to be such a superficial interview," the chilly Heath told the journalistic tyro.

The section on journalism - one of five long, free-form parts of a book that manages to be both loosely structured and perfectly rhythmical, symphonic almost - offers a portrait of a drink-fuelled, seat-of-the-pants, less spin-obsessed age when it was also possible for Mount to get an exclusive interview with prime minister Harold Wilson. It took place at 9pm one Saturday evening in Wilson's study at No 10, which was in virtual darkness because a fuse had blown. "The atmosphere was rendered more sinister still by the fact that the prime minister was wearing a large black eyepatch," writes Mount. "The single lamp that was still functioning gave a weird cinematic effect. It was as though I had tracked down some war criminal in his last redoubt."

Typically, Mount's tape recorder ran out of tape and he had no idea how to put in a new one. Wilson's press secretary did it for him. Oh, and what sounds a magnificent set-piece interview never ran. The Daily Sketch had just bought the life story of the first Miss World and plumped for that instead. The paper ceased publication a few years later, which is at least reassuring.

Self-deprecation, as David Sexton pointed out in an admiring review in the London Evening Standard, is the dominant note in Cold Cream. "If ever you want to know what an expensive classical education fits you for, it is for writing 250-word leaders in a tabloid newspaper," Mount writes of his time on the Sketch. Attempting to become a Conservative MP just before his early excursion into journalism, he has high hopes of becoming Tory candidate in rock-solid Wells, where he is up against a man who was badly injured during the war and had to have extensive plastic surgery. "If this was to be a beauty contest, I evilly calculated, I must be in with a chance." He lost out at the selection meeting - "The Man Without a Face romped home and I packed it in".

Mount's own face is hard to read. He is still cherub-like at 68, though with rheumy eyes and a red swelling on his cheek that may be a volcanic spot activated while shaving. His light-green shirt is crumpled. I like to imagine he has just come to his publisher's Soho office for this mid-afternoon rendezvous from a three-bottle lunch, but he tells me he has been killing time at the National Portrait Gallery.

His background - Eton, Christ Church, "semi-dormant" baronetcy - is that of a toff, and when in 2004 he wrote a polemic on social inequality called Mind the Gap his bona fides for the task were derided in some quarters. But his saving grace, he says, is that his family were forever short of money. He calls the sub-section of the upper class from which he hails "Hobohemia". Mount's father, Robin, appropriately given the family name, was a gentleman steeplechase jockey who once rode the mighty Golden Miller, five times winner of the Cheltenham Gold Cup. His mother, Julia, starting point and presiding spirit of the memoir, was one of the extensive Pakenham clan (immortalised in his uncle Anthony Powell's seductive novel sequence A Dance to the Music of Time). Mount, living with his parents in an isolated village on Salisbury Plain, didn't go to school until he was almost nine.

"My mother taught me everything I knew academically," he says, "and my father taught me everything else. My father having no occupation after the war, he was a sort of magical companion. It wasn't simply a question of going to see him riding in steeplechases; he was a tremendously inventive and enchanting talker, and I feel even now that I'm still living off the memory bank of his old jokes. He had a very strong influence on me that way."

Mount's relationship with his mother was complicated: intense and loving, but also with a certain detachment, which he says reflects a Pakenham family coolness. "They always found it difficult to express emotion. My Aunt Violet said that when you came down in the morning you'd hear the crash of these Pakenham skulls as they attempted to embrace but were lacking the skill to do it." He puts it even more strongly in the book: "I was brought up all wrong and so was my mother ... We should not have needed a jar of cold cream to touch one another's cheek."

This familial unwillingness to open up and his mother's death from cancer when he was in his teens left Mount emotionally adrift, and while everyone else in the 60s was obsessed with sex and revolution he went in search of respectability. "I think I was in a mulish mood during the 60s," he says. "I was in a kind of sullen revolt against it, and wasn't part of the swinging feeling at all." His father's irresolution and fondness for drink embarrassed him then; hence the flight from Hobohemia. "I liked to do all the things he didn't do; I liked to pay the bills as soon as they arrived. I've been trying to join the petit bourgeoisie all these years really."

Those uncertainties - the rejection of Hobohemia and hankering after security, emotional as much as financial - ended when he married Julia Lucas in 1968. His memoir covers his early life, so this Julia is conspicuous by her absence, but one passage in the book conveys her importance to him through 40 years of marriage. "I somehow knew, though I knew virtually nothing else worth knowing, that marriage was the last refuge of the undeveloped heart, the warm-water port that welcomes every iceberg. Marrying Julia was the end of my early life and the beginning of my present."

He chooses not to explore that present life in the memoir. "Though two Julias have made my life," he writes, "in this book there is only the first one." His stated reason is that "it is impossible to recollect the present because it is still going on". But perhaps, too, as Martin Amis has said, it is hard to write well of happiness - and the Mounts are, by all accounts, very happy, in a large, "civilised" (in the words of a friend) house in Islington, the family un-Pakenham-like in its ability to embrace without clashing skulls.

Mount has three children - William, a musician turned record producer; Harry, following his father as journalist, author and classics buff; and Mary, an editor at Penguin. Another son died in infancy, at a time when upper lips were still stiffened and it was not done to grieve too openly. Mount does not refer to this private pain in the book - and I was surprised that some of his long-standing colleagues did not know of his loss - but one friend said the trauma had, if anything, made a close family even more tightly knit.

Marriage released Mount from his insecurities, and he came to admire his father's unwillingness to conform, and pays him a moving tribute in the book. "I remember only rare occasions in my childhood when it was clear that his attention was fully engaged, when he was making mayonnaise or breaking a yearling in the field beyond the beech hedge. His inability to concentrate on practical affairs used to drive me mad. Now that I am older than he ever lived to be, I look more kindly on his willingness to be diverted. He seemed to inhabit the world more fully than people who looked straight ahead."

Mount reckons he inherited a "butterfly mind" from his father, though whether this is real or contrived, part of the baggage of the old Etonian, is a moot point. Behind the unironed shirt, upper-class drawl and comic self-deprecation lie a man who has had three significant careers: as a serious writer, with a hefty output of novels (I apologise for not having read them ahead of the interview: "That would be diligence carried to extreme excess", says Mount); as a figure who, over more than 40 years, has engaged seriously with what passes for political life in Britain, culminating in an 18-month spell as the head of Mrs Thatcher's policy unit in 1982-83; and as editor from 1991-2002 of the Times Literary Supplement.

Mount denies at least two of these heinous charges of being successful. "The only one I did perfectly competently was editing the TLS, but then I think that's quite an easy thing to do because it runs on its own." His novels, he says, have been "bought only by my friends and relations". As for politics, he says he never quite had the Thatcherite will to succeed at that grinding trade. "I haven't got the physical stamina," he says. "That was what I was struck by: how they can go on. In politics repetition is half the battle. You have to go on saying the same thing, making the same speech, and that is a very stern discipline. But if you don't do it, the message doesn't get through."

Mount first encountered Thatcher when he was working at the Conservative Research Department in 1964. "She was then 39 years old and ... looked more like one of the over-age milkmaids in the chorus of the Bath panto than someone in training to be an Iron Lady, although you could not miss the willpower," he writes. "Her voice was sharper then than it later became, when she had had voice training to bring it down a semitone after criticisms that she sounded shrill. To me, anyway, she sounded unbearably sharp as she began to slice my papers into pitiful shreds."

Despite this unpromising beginning, in 1982 Mount was invited - thanks to the intercession of Thatcher's economics adviser, Alan Walters - to be head of the No 10 policy unit. When they met again, Thatcher instantly launched into a lecture, free-wheeling but powerful. "What we really have to address are the values of society," she told him. "This is my real task, to restore standards of conduct and responsibility. Otherwise we shall simply be employing more and more policemen on an increasingly hopeless task. Everyone has to be involved. At one time, women's magazines played quite a constructive role. Now they've just caved in. Personal responsibility is the key. That was what destroyed Greece and Rome - bread and circuses. It has to stop. Ferdy, it has to stop."

Mount was swept along on the tide of her conviction. Generally seen as a Tory "wet" (he is a cousin of David Cameron and endorses his "liberal Conservatism"), he says that in those confident early days of Thatcherism he shared many of her ideals: curbing trade union power, reducing the size of the state, encouraging enterprise, rebuilding the family. "I confess I found all this both startling and thrilling," he writes of her call to arms at that audience in Downing Street. "The naked zeal, the direct, unabashed appeal to morality, the sheer seriousness ... There was none of the weary, professional cynicism I was so used to and had myself become so weary of."

He was hooked, though in the book and in person he makes a distinction between Thatcher the politician and Thatcher the human being. "She was extremely good to work for, but horrible, especially to colleagues," he says. "She'd ring up nice John Biffen and give him a tremendous rocket for refusing to appear on the Today programme because he said it was too early in the morning. Or give Francis Pym a dressing-down for saying the Tories shouldn't have too big a majority. She was very brutal."

A "strange, tense, ruthless but deeply honourable and usually honest woman" is Mount's adroitly double-edged summing up of Thatcher in the book. "She remained heroic, intolerable often, vindictive, even poisonous sometimes, but always heroic. Equally, I never became fond of her. That insistent, harsh concentration could never become endearing. 'I'm not here to be nice,' she would say, which was just as well." After a year and a half, during which he had written the Conservatives' 1983 election manifesto (he of course dismisses it as "the dullest of all time"), Mount decided he'd had enough. "You needed to have this flat, clear, blinkered approach," he says. "It's not easy to live with that over a long period."

He admired Thatcher's zeal and tenacity, but recognised - or at least recognises now - that her cult of individualism damaged Britain. "Her startling success and her relentless personality had a long-term corroding effect on her party," he writes. "It was an individualist age, and she was both the midwife and the emblem of it," he tells me. "Cameron is very much trying to reform the little platoons." Mount belies his languidness by seeming almost obsessed by the unromantic issue of decentralisation.

Is he giving cousin Cameron the benefit of his experience? "The only thing I can claim to have contributed to him is the phrase 'slow politics'," he says, "on the analogy of slow cooking. Trying to move in the direction you want to go consistently and deliberately, but not ... [as usual, like Alfred Tolland in Powell's Dance, he doesn't quite finish the sentence] ... Aim to hit the ground strolling. You shouldn't go charging in."

As with his early decision to abandon hopes of becoming a Tory MP - "If I'd really, really wanted it, I'd have plugged on and stood in the Rhondda, done things like canvassing and taken lessons in how to speak, all of which I was too proud or idle to do" - leaving No 10 after 18 months, with an election just won, was a reflection of an inability to grind it out, to press on relentlessly, the love of the detour he inherited from his father. He says he is the sort of person who, looking for a definition of "focus" in a dictionary, invariably finds himself transfixed by "foc'sle".

Butterflies, he implies, get bored easily. The truth may be that he has always been, first and foremost, a writer, but life - the need to pay the bills on time, an abiding interest in politics, a tinge of self-doubt (he imagines himself "tiptoeing along the edge" of the true artist's world) - kept intervening. His other lives almost certainly damaged the reception of his novels; made him appear less than monkish in his devotion to his art. Writers must suffer, and old Ferdy wasn't suffering enough. What rot, as the habitues of Hobohemia might say after a few gin-and-oranges. Now at last, with this loving, lyrical, life-filled memoir, he may get his due.

· Cold Cream: My Early Life and Other Mistakes is published by Bloomsbury, price £20.