There is a curious ritual for every new guest who arrives at Michelmersh Court, the country retreat of Sir David Frost and his wife, Lady Carina. Despite the pouring rain, I am marched out across the croquet lawn for an eccentric twist on grandmother's footsteps. "Whatever you do, don't look back," Lady Carina exhorts, pointing out a Cypress swamp tree.

"Right, are you ready?" We've reached the far end of the lawn, observed by two deer in the adjoining paddock. I feel a bit like Orpheus, instructed not to look back at Eurydice. We pause, then spin around together. It's a breathtaking sight: the symmetrical south elevation of the house, which has been described as one of the finest Queen Anne rectories in the country.

"So peaceful, so humble," whispers Lady Carina, daughter of the 17th Duke of Norfolk. I'm not sure humble is the first word that springs to mind. The Grade II* listed house has just come to market for £5.25m. But I know what she means. There's something about the exquisite doll's house proportions that gives it grandeur without ostentation. In early summer, when the wisteria and magnolia are in full bloom, the sight is even more stunning.

(Jonathan Stone)

"We do that with everyone who visits," Lady Carina adds, as we walk back to the house, where Sir David is waiting for us. Which is quite a thought, when you think of the circles that the Frosts mix in. Witness the great and the good who turn up every summer to Sir David's garden party at his London home in Chelsea. The television inquisitor, famed for his languid delivery, has interviewed the past eight prime ministers and seven US presidents, most famously Richard Nixon. When George Bush Senior visited, he didn't just play grandmother's footsteps when he came to stay in 1993, he knocked a football around on the lawn – keepy uppies and all.

"There he was, playing soccer on a Sunday morning," Sir David recalls, as we settle down in front of a log fire in one of the property's six elegant reception rooms. "I had no idea he could play, but it turns out he learnt the game when he was at Amherst College, 70 years earlier."

The former president – "a great raconteur and delightful guest," according to Sir David – stayed in the White Bedroom, one of eight in the main house. There is also a three-bedroom lodge at the foot of the drive and a two-bedroom apartment above the garages. Bush was accompanied by his Secret Service security detail in a fleet of six black Cadillacs, which had to negotiate the narrow country lane that runs up past the local church to the house's white gates.

(Jonathan Stone)

But that's one of the many features of Michelmersh Court that appealed to the Frosts when they bought the property 25 years ago. It's set in 21 acres of Hampshire woodland, paddocks and meadow, three and a half miles outside Romsey in the Test Valley, but accessible enough for friends from London – and America – to drop by at weekends. The capital is 80 minutes by car, and Winchester, 10 miles away, has a 55-minute train journey into Waterloo. The Frosts spend Monday to Friday in London and retire to Hampshire at weekends.

Sunday lunches lie at the heart of their country life and take place in "the school room", my favourite part of the rectory. It's a light and airy space, off a curious Fifties Formica-clad kitchen. (For years, the Frosts had a Portuguese cook and housekeeper, who lived in the lodge, but Lady Carina does the cooking these days.) There are some plain wooden tables and benches at one end of the school room, a large ping-pong table in the middle, and at the far end some sofas and a piano. Everything you need, in other words, for a riotous Sunday gathering in the country.

Sir David is too discreet to offer names, but I eventually gather the identities of some of the guests who come to lunch. They include Stephen Fry, who has written the screenplay for a new version of The Dambusters, produced by Sir David (and featuring a squadron of Chinese-built replica Lancaster bombers); Lord Lloyd-Webber and his wife; and, in the past, Baroness Thatcher.

"The Lloyd Webbers live about 30 minutes away [near Newbury], which counts as neighbours in the country," says Sir David.

At 73, the presenter shows no signs of easing up in his career. It began in the Sixties satire boom when he hosted That Was the Week That Was, and The Frost Report (starring then-unknown comedians including John Cleese, Ronnie Corbett and Ronnie Barker).

(Jonathan Stone)

Every Monday morning, a chauffeur drives him up to London, where he presents a weekly television current affairs programme, The Frost Interview, for the satellite television station Al Jazeera. The programme replaced Frost All Over the World, which ran from 2006 until last year. Both are similar in format to his earlier show, Breakfast with Frost, which ran on the BBC for 12 years until 2005.

His eyes are still bright with mischief and, even in our brief encounter, he has an uncanny ability to make me relax and reveal more of myself than planned. (How old are my children? What sports do they play and so on.) The irony of taking a journalist around his private house is not lost on the host of Through the Keyhole, a mystery tour of celebrity's houses, which he presented from 1983 to 2008. (A revival of the show is reportedly on the cards, with Keith Lemon presenting.) In the spirit of the show, I look for a detail that might reveal the owner's identity. How about the cushion on a sofa, embroidered with the message: "If I cannot smoke cigars in heaven, then I shall not go."

The well-stocked wine cellar is another clue (he is a lover of fine wine), but a more obvious giveaway is the spacious downstairs lavatory, where the walls are adorned with photographs of Sir David interviewing a pantheon of world leaders. These include Tony Blair and Bill Clinton (together on the lawn, at Camp David), Vladimir Putin, Nelson Mandela and John Major. Which British prime minister does he most admire, I wonder, having interviewed so many?

(Jonathan Stone)

"Tony Blair was a brilliant interview subject, because he could make policy and ad lib on the air, which is appealing to an interviewer," he says. "John Major must be one of the most decent people who has been PM. And his sense of humour grows. His after dinner speeches get better and better – sparkling. He's very big in the US."

But it's Richard Nixon's name that will forever be linked with Sir David's, after he persuaded the former US president to admit to a cover-up over the Watergate scandal. The series of combative interviews that led to the confession was recently dramatised in Frost/Nixon, a play and later a film written by Peter Morgan. "When I went to say au revoir, after all our interviews were finished, Nixon opened the door and said, 'Come in, David.' It was the first time he had used my Christian name. For about 20 minutes, he was so carefree, then the shutters came down."

The interview made Sir David's reputation, and he went on to become a household name in Britain and America, producing films as well as hosting his own television shows (co-founding LWT and TV-am along the way). His life today seems idyllic, commuting between his town house in Chelsea, and Michelmersh Court.

The son of a poor Methodist minister, he is worth £200m, happily married for 30 years to Lady Carina, a charming former L'Oreal model, and they have three sons. Which raises the question: why are the Frosts selling their country house?

"The children don't live here any more and we miss them," says Lady Carina. Certainly, when we go on a tour of the house's bedrooms (master bedroom and three principal bedrooms on the first floor, five bedrooms on the second floor), the place feels deserted – a clear case of empty nest syndrome. The plan is to move to a farmhouse in Hambleden, in the Chilterns, where there will be a barn for Sir David's Sunday lunches and cigars plus a cottage for the boys when they come to stay.

It's not so much downsizing as a change of venue. New beginnings, no memories, no missing the children. Lady Carina, who decorated Michelmersh Court and likes "doing houses", also wants a new project, although she is the first to admit that there is much she will miss about this property: the working wooden shutters, the dado panelling, high ceilings, hard wood floors and sash windows.

"I shall miss it, too," says Sir David, who fell in love with the house as soon as he saw it - even though he had given Lady Carina strict instructions to look for a small cottage in the country.

"This is a very private house surrounded – protected – by 21 acres." A keen sportsman, Sir David will also miss the secluded outdoor swimming pool and tennis court. Not to mention the five-a-side football matches held after Sunday lunch on the lawn between the library and the sheep field (where guests have been known to arrive by helicopter). But who will inherit this country idyll?

"Ideally, the property would suit a young family moving to the country from London or abroad," says Ed Cunningham, a partner in the country department of estate agents Knight Frank.

"Or perhaps a mature couple wishing to entertain on a grand scale in a stunning peaceful and private location. It would also be ideal for those in the art world such as film directors or actors, due to its natural seclusion and warm, welcoming feel."

Whoever chooses to look at it must remember to play Lady Carina's game. Don't look back until you reach the far end of the croquet lawn, even if it's raining. The view that greets you will not disappoint.