There may only be one real star in the choreography of budget day – the chancellor, with his red box, his photocalls, his theatrical giving with one hand, taking with the other – but the members of the chorus also have parts to play: they must line the benches behind him, looking fascinated, engaged, and if not 100% approving, at least noncommittal. They are not meant to fall asleep, heads gently drifting down towards their ample chests. Not the most ringing endorsement of his colleague, was it? The lord chancellor and secretary of state for justice laughs, flushes and takes it head-on. "It wasn't anything to do with George's speech. It was early on. I sat there, snuggled in alongside my colleagues, and I was conscious of nodding for a moment. I wasn't conscious of going to sleep, actually. I don't want to cause a dispute with Ladbroke's [who paid out, on odds of 16/1, to anyone who bet that this would happen], but my head was dropping and – [I was] wide awake by the end of the speech. I assure you it wasn't lack of interest. Nor was it George's delivery, or anything about prime minister's questions – I listened to them quite carefully. But I had had a bad night the night before – I'm not sure I was going to tell you, because it sounds like I'm whinging, but I'm having gout in my wrist and the blasted thing kept me awake the night before. So I was generally short of sleep."

The goutish hand does look painfully swollen. It is somehow too fitting, a cosmic failure of irony, that a man renowned for enjoying the finer things in life, who says he still settles down to his red boxes each night with a brandy and "quite a good cigar" (even when he has been at a jazz club until 2am), should suffer from this particular ailment. But otherwise Ken Clarke is looking better-kempt than he has for a while: returning to cabinet after an absence of 13 years obviously suits him.

Having been chancellor himself – the zenith of his 18-year run of ministerial jobs under Thatcher and then Major; he left that post having halved the budget deficit – what does it feel like to watch another Conservative do the job? "It's very hard work, the budget – six months of very hard work. And I used to take the view, on budget day, that apart from anything else, one can enjoy it a bit. So if I felt in the mood I gave quite lively budget speeches and quite enjoyed the occasion. I think George enjoyed yesterday." Whether he sounded like this when he was growing up behind his father's jewellery and watchmaking shop in Nottingham is anyone's guess, but Clarke has perfected an interruption-proof mode of speech: the words, delivered in a posh, life-is-a-comedy tone, slide into each other, forming a sinuous line that drops in volume mid-sentence and rises whenever there's a chance someone else might speak. Having said that, he, unlike many politicians, does manage to conduct a proper conversation: he may be talking over you at the time, but he hears the question, and answers it.

And while he may often claim that the reason he has survived when so many colleagues succumbed to stress and turmoil is because he is "so laid-back as to be almost horizontal" this has the ring of a useful overstatement: the body might be entirely still, leaning back in the chair, but the eyes are watchful, steady, all there. Just as important to his longevity, I suspect, has been the self-confidence which means that the answers he gives are – refreshingly – neither anxiously focus-grouped nor a ventriloquised party line. Then there's the fact that he had another job before that he enjoyed (as a barrister), and could at any point, presumably, return to. And there's the famous hinterland: football, cricket, motor-racing, birdwatching, the Romanesque churches he explores with his wife Gillian, who gave up a promising career as a historian at Cambridge to raise their children. And jazz: his Jazz Greats on Radio 4 (he hopes to record a ninth series later this year) are notable for their chatty, unforced enthusiasm and lived knowledge – he really did see, for instance, the Miles Davis quintet perform in Croydon. "Coltrane didn't turn up."

Does his longevity mean that Osborne, for instance, comes to him for advice about his budgets? Not this time, apparently, though he is quick to say that he likes Osborne; that in opposition, particularly, they used to go for lunch and "put the world to rights"; that if he had been asked, he would have advised what Osborne did this week: fiscal neutrality, cost-of-living concessions (to middle England), no Gordon Brown gimmicks. Of course, he would say that, but the thing often forgotten by people desperate to hug a Tory – to believe that some of them, especially the charismatic, clubbable ones like Clarke, might just be misguided liberals – is that in many ways he really is a Conservative. They also forget that when he was health minister in 1988 he had good claim to be the most hated man in Thatcher's cabinet: quite an achievement. So he was always for Plan A – cut hard and fast – and still is.

It must be strange, though, to watch a chancellor who is younger than both his children. "George occasionally takes pleasure in telling me that I was elected a member of parliament before he was born, yes. And that he is older – sorry – he's younger than either of my children. Cor blimey." Lots of laughing. "It's part of a wider issue – I'm part of this fashionable generation that's now working beyond what my parents' generation regarded as the ordinary date of retirement. And it hasn't crossed my mind to retire. But it does mean that you've probably got to work a step down from where you were. I'd like to say I think I can handle the work of being a cabinet minister, but this is a medium-sized department. This isn't the old home office. This isn't the treasury. I'd probably claim I could handle those, but I wouldn't be surprised if my friends said, 'Mmmm, no' – he laughs again – 'I don't think so.' Because those are immense jobs. So you're suddenly further down the pecking order than you were, with a smaller range of responsibilities."

It is not inconceivable that this will be his last high-level job in government: there's his age, and the fact that his Europhilia and relatively liberal attitudes to law and order mean a distinct section on the Tory backbenches wants his head. "I enjoy it all, and when I say I'm in the mid-term of my career, I only do it to make my colleagues go pale. Obviously, at this age, I might not be allowed to carry on, or there might come a sensible point when I shall no longer be a minister. It gives me different perspective. I am no longer a young, ambitious guy. I'm not trying to catch the selectors' eye. I look around for people who are much older than me who are still in office around the world – Europe's a bit thin at the moment. It's a slight embarrassment, because the only one much older than me is Silvio Berlusconi."

Surely this would be a real disappointment – after all, he ran for leader and lost three times; does the fact that he now never will be leader rankle? "Of course I was disappointed, but it was not one of these traumatising, character-changing experiences. But given that in every walk of life you see people who look slightly damaged by being disappointed, insofar as I have any sufficiency of insight, I don't think I'm damaged at all. I'm certainly not a Vicar of Bray – and the longer I've gone on, the more I've stuck to my guns on things, and I've been candidly indiscreet in public simply because I want to say what I actually believe, all that kind of thing."

It's a flattering self-portrait, but it is also true that he has been remarkably consistent and frank – on Europe, for instance, when it would have been far easier for him to concede to scepticism; repeatedly in his defence of the rule of law and the independence of the judiciary (if he has a core belief, this may be it). Thus the recent fuss about the European Court of Human Rights, sparked by judgments that some prisoners should have the vote, and that paedophiles should be able to appeal to have their names taken off the sex offenders' register. The government has established a commission to look at the relationship between Strasbourg and London and attempt to improve it, but withdrawal is not part of the terms of reference – "it's never been part of the political policy of either of the two political parties".

On law and order, Clarke, never a darling of the rightwing press (they call him "soft-cell Ken") set out his stall early: drug addicts and the mentally ill should be sent to charities and the NHS, which could help them get better; it was "virtually impossible" to rehabilitate prisoners within 12 months, and so there should be fewer short sentences and more restorative justice and community sentencing; three prisons are to close, and the prisoner population to drop by 6,500 (something of a necessity when his budget is also going to drop by 20%). The trouble with all this, and with his proposal that charities and private ventures should take up the slack, being rewarded for success on a payment for results system, is that there is no money: the Community or Custody Inquiry, chaired by Ian Blair and Anne Owers, concluded that Clarke's laudable push for rehabilitation over sentencing risked collapsing into a black hole for lack of cash.

Also facing a catastrophic cut in funds is the legal aid system – £350m, mostly, according to the Law Society, affecting divorce, housing and child-contact cases. With regard to family law, Clarke insists that mediation should take up the slack (though as this paper pointed out not long ago, forced mediation is something of a contradiction in terms; it is also not viable in cases where there is abuse, or a massive imbalance of power, or a breakdown in communication). But what is really revealing is the angle from which he argues for less legal aid in general – not that this cut, like many others, will disproportionately (in fact, almost entirely, by definition) fall on those least able to bear it, those who in the days ahead risk losing their homes and their jobs and will now have no recourse; rather, he argues that we live in a society that has become too litigious, and the people likely to be sued need to be protected. "I see it in people in all walks of life. Most of the doctors I know are very conscious of being sued. Most of the businessmen I know spend a fortune getting legal advice to avoid litigation. Most of the ministers I know take very careful note of the advice they get about the risk of judicial review."

On foreign policy, too, he argues from a legal point of view. He voted against going into Iraq, but for intervention in Libya, "to stop a humanitarian massacre and to stop a dictator attacking his own people and to restore order so far as we can. What we seem to have almost established in international law is the humanitarian basis, which can in exceptional cases justify intervention by the international community. What you have actually got is intervention on humanitarian grounds authorised by a UN resolution, carried out by the international community. It is a significant event in the evolution of the world order."

He knows it's controversial, but doesn't hide the fact that he would like Gaddafi gone. "We do have one particular interest in Maghreb, which is Lockerbie – if other people want to get rid of the curse of Gaddafi, the British people have reason to remember the curse of Gaddafi – Gaddafi back in power, the old Gaddafi looking for revenge, we have a real interest in preventing that."