Vince Cable appeared to confirm that the Duke's position was in doubt this morning as he declined to give the royal his firm backing in a radio interview.

"He is a volunteer, he has offered to perform these roles, and I think it is down to him essentially to judge the position he wants to be in," the Business Secretary told the BBC Radio 4 Today programme.

"Obviously there are conversations that will take place with him about what he is to do in future. That is simply a matter of managing the relationship."

But asked later whether Mr Cameron had full confidence in Prince Andrew as a trade envoy, the Prime Minister's official spokesman replied: "Yes."

He added "The Prime Minister thinks he is doing an important job and is making a major contribution and he is supportive of him in that role.

"We are not reviewing that role in any way. The Government's position is very clear: that we support him in his role as trade envoy.

"He has made a very important contribution to UK trade through the role and continues to do so. We think he makes a valuable contribution and so does British business."

Speaking during a Cabinet away-day in Derby, Chancellor George Osborne also gave the Duke his backing, telling reporters: "We have confidence in him doing the job and we think he's done a good job in recent years.

"He's promoted British exports. What we want is everyone promoting British exports at the minute and Prince Andrew has done that."

Epstein, who played host to the Duke at his Florida home, a mansion where under age girls were sexually abused, gave £15,000 to the Duchess’s former personal assistant after the Duke allegedly approached him for help last December.

There is no suggestion the Duke had any sexual contact during his visit to Epstein’s mansion.

Government sources said the Prime Minister “won’t shed many tears” if the Duke quits as Britain’s special representative for trade and investment, and five ministers in the departments overseeing the Duke’s role refused to give him their backing.

A Cabinet minister told The Daily Telegraph that the Duke’s responsibilities would be the subject of a “tasking review” that would lead to a reduction in his responsibilities.

The minister said that there would be no official announcement of the review, as the Government was wary of publicly putting pressure on the Duke to quit, for fear of antagonising the Queen and the Royal family.

Privately, however, government insiders increasingly believe that the Duke’s position is becoming untenable. One senior Conservative figure who has dealings with the Duke described him as a lonely and misguided figure with poor judgment.

“There appears to be no discernible mental activity,” the senior Tory said. “I feel sorry for him. He has no friends and so is surrounded by these vile people.”

Another government source said: “We won’t be giving a full-throated defence of him. There won’t be many tears shed if he resigns.”

The Duke’s role is overseen by UK Trade and Investment, a government body under the charge of the Department for Business, Innovation and Skills, with input from the Foreign Office.

Vince Cable, the Business Secretary, failed to give the Duke his backing when he was asked if he should continue in the role, and four other ministers in the two departments also refused to voice support for the Duke yesterday.

Mark Prisk, the business and enterprise minister, would say only “no comment” when asked if he still backed the Duke, while John Hayes, his departmental colleague, said it was “not appropriate” for him to comment.

The Foreign Office ministers Jeremy Browne and Lord Howell of Guildford also replied “no comment” when asked if the Duke had their support. William Hague, the Foreign Secretary, said the Duke still had his confidence, saying that he had done “a lot of good” for the country in his roving trade role. But sources later said that Mr Hague had not been properly briefed on the situation when he spoke.

On Saturday the extent of the concern felt by Buckingham Palace became clear when an aide mistakenly sent an internal palace email to The Daily Telegraph which read: “Will UKTI stand behind him? We need some government backing here.”

David Cameron faced calls in Parliament last week to relieve the Duke of his role as trade ambassador following questions about his “close” friendship with Saif Gaddafi, the son of the Libyan leader, and with Tarek Kaituni, a convicted Libyan arms dealer.

But it is the Duke’s links to Epstein, who is alleged to have abused up to 40 young girls, that now pose the greatest threat to his position.

The Duke, who has counted Epstein as a friend for 16 years, was a guest of Epstein at his Florida mansion and was pictured with his arm around a 17-year-old “masseuse” who later claimed to have been one of Epstein’s victims.

Last month a photograph was published of the Duke strolling through Central Park in New York in conversation with Epstein, during a private visit which he has since described as a “mistake”.

The photograph was taken in December, and it is now alleged that the Duke took the opportunity to discuss his ex-wife’s £5 million debts with Epstein.

A source close to the negotiations with the Duchess’s creditors has told The Daily Telegraph that the Duke was “sorting the offer” of financial help from Epstein at the time, a claim that has not been denied by Buckingham Palace.

Last night the Duchess’s spokesman admitted that Epstein had paid £15,000 to Johnny O’Sullivan, her former personal assistant, as part of the £78,000 she owed him in wages and expenses.

The Duchess’s spokesman described the payment as an “independent arrangement” between the two men and said the Duchess intended to repay the money.

Asked if the money had been paid as a result of the Duke’s meeting with Epstein in December, the Duchess’s spokesman said: “I don’t know.”

It has also been disclosed that Gordon Brown resisted demands to sack the Duke from his role as trade ambassador during his time as prime minister.

Chris Bryant, the former Foreign Office minister who was responsible for Kazakhstan, where the Duke has forged friendships with controversial businessmen, urged Mr Brown to force out the Duke. Mr Bryant said: “I tried to raise my concerns with No10 but a Labour administration tackling the Royals would have led to charges of republicanism. Perhaps it might be easier for the Coalition to take a stand.”

The FBI is said to be considering reopening its case against Epstein, who was jailed after admitting soliciting an under age girl for prostitution in 2008.

Sources in America said investigators could seek to interview the Duke, who is reported to have been given daily massages paid for by Epstein.