"We'll talk in the library," Clive James says, as if that would make our destination obvious. I look around the hallway and all I can see are books - walls and walls of books, in half a dozen different languages, themed in alphabeticised rows along handmade wooden bookcases lining every room. It is like standing in Waterstone's. As far as I can tell, the room we settle in can only have been designated the "library" on account of being the biggest, thereby containing even more books than anywhere else. Later I discover that upstairs is even bigger - but then, that is because it's a ballroom, devoted exclusively to James's enthusiasm for tango. The place is less like a flat than a rather literal-minded film set, commissioned to evoke this character called Clive James.

Some discrepancy between a public figure's professional persona and private self is normal, but James turns out to be even more like Clive James than the one we have known for nearly 40 years - humorist, critic, poet, presenter, polyglot and prolific polymath. Talking to him feels like being on one of his chatshows, for the conversation skims from sex to Arnold Schoenberg, tango to Abi Titmuss, always enthusiastic and often self-deprecating, never lingering long on any one thought.

In the space of an hour he quotes Karl Popper, a South American president, Dorothy Parker, TE Lawrence, and several other luminaries I have never even heard of. Even when James isn't laughing - which isn't often - it feels as if he is, for he has the sort of mischievous smile that teeters on the brink of mirth. His breathing is a little laboured, for he is thickset, with a neck like a bull's and a body like Tony Soprano's - but at 69 he remains energetic, and improbably unaltered. Or, as he puts it, "The smartest move I ever made on TV was to start off looking like the kind of wreck I was going to end up as."

It is nearly a decade since James retired from our screens, but he is still making television here in his library, interviewing famous literary and thespian friends for broadcast on his website, clivejames.com. It is an extraordinary site, archiving a body of work that includes video footage of old interviews with, among others, the Spice Girls, audio broadcasts of lecture series and his current Radio 4 show A Point of View, innumerable articles and song lyrics, plus four novels, four memoirs, eight poetry collections, and 14 volumes of critical essays. When I tell him it is intimidating, he insists it is not meant to be, but looks rather pleased.

I wonder whether, having presented countless primetime TV shows in his career - The Clive James Show, Saturday Night Clive, Clive James On Television - the interviews he now films in his library can feel like making real television. "No," he laughs. "Thank God, no. It takes a lot less time, and a lot less hoo-ha. The guests can just drop in, you do it, then have a Chinese meal. Tom Stoppard won't do that for you in a TV studio." But he admits he still sometimes wishes he hadn't turned his back on the medium which made him an international household name.

"Yeah, often, because it's a lot of fun, and you do get spoilt. You get used to getting on to the aeroplane and turning left. Who was it who said that? I think it was someone like Elizabeth Hurley. It's usually Elizabeth Hurley," he laughs. "She gets credited with calling people not in show business 'civilians'. Actually, I was the first to say that - but I doubt if you trace it back that I really was the first, just because I thought I was the first. Scarcely anything is original - it's very hard to be totally inventive, so I'm not terribly interested in originality. Vitality is all I care about."

This cannot be quite true because James is famous for coining what he calls the "resonant phrase"; Arnold Schwarzenegger he described as "a brown condom full of walnuts", and Barbara Cartland's eyes as "looking like the corpses of two crows that had flown into a chalk cliff". He grins happily - "Yes, that one took off" - but still has no idea where they come from.

"There's just a click in the head. You can't tell where it's coming from, it just visits you, it's just like poetry. Inspiration is too big a word, but you don't know exactly how it's done. Which is why people who can do it get nervous, they think it might not be there tomorrow. Although you'd better have a fall-back standard. As the great philosopher Martina Navratilova once said, it doesn't really matter how well you're playing when you're playing well, what matters is how well you're playing when you're playing badly. You've got to have a standard you can hit. The real fun starts when you get above that - but that's beyond your control. Still is. And I've been doing this a long, long time."

James's writing career began in earnest in 1972 as a television critic for the Observer, where he says he soon became unpopular for writing his reviews at great pace in an open-plan office, laughing out loud at his own jokes. Has it got easier or harder over the years? "Well, Thomas Mann, he said - and this is great, this is writing - he said a writer is someone for whom writing is harder than it is for other people.

That line is perfect in every way. Not only is it perfectly written, but it's absolutely true.

"The only thing I've got better at as the years have gone by is I've grown more resigned to the fact that it comes hard. You realise that hesitation and frustration and waiting are part of the process, and you don't panic. I get a lot better at not panicking. I get up every morning early if it's a writing day and I will do nothing else but write that day. But the secret is not to panic if it doesn't come."

Unlike many writers, James does not describe the process as agony but it certainly seems to be a compulsion, if anything only intensified by age. "Well, the option not to do it isn't there," he agrees. I'm curious, then, to know if he thinks it is possible to "promote" writing; we meet during the launch of the BBC's Poetry Season, when the airwaves are exhorting us all to "Let poetry into your life".

"No, I always thought poetry should be banned," he jokes. "It would be the only way to make it popular with the young - to ban it, make it a criminal offence to own it or to deal in it. But," he adds, when his chuckles subside, "Carol Ann Duffy will do things for it, I suppose. She'll be a good laureate, because teachers love to teach her poetry to the kids. No, the real disaster is the Oxford professorship."

Suddenly he is very serious, referring to the allegations of sexual harassment that forced Derek Walcott to withdraw his candidacy. "I think it's a catastrophe," he says, "because Derek Walcott is a great poet. But what male teacher is going to escape a sexual harassment case? All you've got to do is stand there, and you're sexually harassing them." Does he mean he thinks the claims against Walcott are false?

"I don't know, I don't know the case. But it sounds to me like a David Mamet play where you've got an imaginative girl, thinks she's been approached, she may not have been. But who knows? It's a very bad reason to stop a 79-year-old man who has all the qualifications, including [the fact that] he would write brilliant lectures. It means a whole generation's going to miss out on his wisdom. For what? For a couple of cases that have been mouldering for 20-odd years."

Walcott's withdrawal left Ruth Padel to succeed to the professorship, and James has some unexpected advice for her. "She's in a difficult position now. She should recuse herself instantly and ask for a new election, because for five years the media's going to be talking about nothing else." I'm not sure if he's joking, and am only half-serious myself when I ask if he would want the job.

"You know - and this is strictly between you and me and millions of readers - it's the only job I want." Really? "Oh, yeah," he says intently, nodding slowly. What, professor of poetry? "Yeah. But you can only dream."

As a populist polymath, James has always been something of a surprise in this country - a class and cultural puzzle. He was born in Sydney in 1939, an only child, but lost his father at six in a plane crash when he was returning from a Japanese prisoner of war camp, and was raised by his mother, a factory worker.

Arriving in London at 22, he drifted for a few years, then studied at Cambridge, became president of Footlights, and established himself as a critic and writer in the 70s, before becoming a permanent fixture on our screens throughout the 80s and 90s. His enchantment with trash TV and the eccentric talents of a Cuban singer, Margarita Pracatan, seemed to sit oddly with someone who taught himself Russian because he "could no longer bear not to know something about how Pushkin sounded" - but he says there is simply no aspect of cultural life that fails to fascinate him. "Well, maybe rap music. But nothing else, no, not really."

His conversation is an odd mixture of highbrow dissertation and ironic wit, punctuated by the occasional intervention of chatshow-host cheese. It is like talking to FR Leavis and being interrupted by Les Dennis.

"Thomas Mann? Great guy. At school with me," he quips, pausing as if to wait for a studio drum roll. When he says he is crazy about his website, he adds, "But I may - get this - be crazy!" I am not sure if it is a highly sophisticated self-referential kind of kitsch, or just a sense-of-humour lapse.

For a liberal public intellectual he has also been unusually lecherous, drooling over supermodel guests with a helpless gusto more familiar to Loaded than Poetry Please. In the days before lad-mag culture, it could just about pass as charmingly daring, but I wonder if he felt less indulgent about all his ogling after the advent of new laddism in the 90s.

"Oh no," he chuckles. "You see, what you think doesn't really matter very much. What depends is what you do about it, and I had a fairly strict code of practice: look but don't touch. Heterosexual men probably die just as randy as they were when they were young - there's just less and less they can do about it. I've always thought the whole thing was funny, and tried to make humour out of it.

And I've always tried to make poetry out of it, too, because it's also sublime. Sex is ridiculous. But on the other hand," he adds, after another carefully timed pause, "it's an almost unbeatable way of reproducing the human race." Cue that drum roll again.

Suddenly he stops smiling. His features darken to an expression I can't read, and he says something quite odd. "I'm frequently told that I talk far too rapturously about women. There are worse crimes. There are worse crimes. I'll tell you one worse crime. In-jur-ing them. It never occurred to me." He repeats it softly, almost under his breath, as if talking to himself. "Never, ever, ever occurred to me."

A second later the joking is back, and he is laughing about his weakness for addictions, which have included cigarettes - 80 a day - booze and marijuana, the last of which he gave up after becoming "the first person known to science to OD on it". He has since quit smoking too, drinks only very occasionally, and counts himself lucky that cocaine was not around when he was younger, as "I would have been an obvious first choice for cocaine death, no question about it". There is unquestionably, he confirms, such a thing as an addictive personality. "I said I could use up a lifetime's supply of anything in two weeks - and it's true." He thinks for a second. "Actually, that line is a bit too good, I may have lifted that. Was it mine?

I can't remember. Though I don't believe in sex addiction, I think it's a crock. Sex addiction is a very big phrase for bad behaviour."

If he could go back through his life and edit out the bits of which he was least proud, which chapters would go? "Oh, without number. Whenever I was cruel or insensitive." Has that been a theme? "Yes. Casual, focusing only on my own needs and requirements, yes. Inability to know that other people are truly alive as I am. That's probably built into the male ego, and I believe in men it's especially strong, more of a warrior ego, and it has to be civilised, usually by women. That's what they're for - they're to civilise men. Not to give them pleasure, but to civilise them."

What the women in James's life make of that remains a mystery, for they have banned him from ever talking about them. His London penthouse may look like a bachelor pad, but he has in fact been married for 40 years to a Cambridge scholar with whom he has two daughters, a portrait painter and a civil servant now in their 30s. He is said to spend the weekdays in London alone, and the weekends in Cambridge with his family, but he will only confirm that he spends "a lot of time alone writing" and "a lot of time" with his family. "These are forbidden zones - not by me. They do not want their movements or existence discussed in the media."

Successful men are often warned that, on their death bed, they will not wish they had spent more time in the office - only with their family. For James I cannot help doubting this will be the case. All he will say is, "If I hadn't been married, I can tell you this much: I would have been a much less civilised man - or even less civilised, put it that way. In fact," he laughs, "I'd probably already be dead".

In our third Haycast, Joan Bakewell talks about writing her first novel at 75, Clive James reads us some poetry and AC Grayling, Claire Armitstead and Sarah Crown discuss the ongoing Oxford poetry professorship scandal