Prince Andrew's role as Britain's special trade representative is to be downgraded as ministers seek to distance themselves from his controversial dealings with discredited business figures.

Amid growing unease in Downing Street and ministerial circles at the conduct of the Queen's second son, ministers indicated over the weekend that a review of the Duke of York's role would be used to lower his profile.

One cabinet minister said a hard-headed assessment would be made during a "tasking review", which is held every six months to assess the prince's work as Britain's special trade representative.

"The gas can be turned up and the gas can be turned down," the minister said, but stressed there was no question of removing the prince. "The royals go on, that is what they do," he said.

Ministers have been alarmed by the prince's links with Jeffrey Epstein, a billionaire US financier who served a prison sentence for soliciting an underage girl for prostitution. The Mail on Sunday reported that the FBI is to reopen its investigation into Epstein over allegations of his involvement with underage girls.

A picture was released last week of the prince with his arm round Virginia Roberts, then aged 17, who claims she was sexually exploited by Epstein and his friends. The teenager said she met the prince who allegedly enjoyed regular massages at Epstein's home. The prince categorically denies any wrongdoing or impropriety.

The prince also faced embarrassment over the weekend when the Guardian reported that he hosted a leading member of the deposed Tunisian dictatorship at Buckingham Palace three months before the regime collapsed. Sakher el-Materi, the son-in-law of Tunisia's deposed president Zine al-Abidine Ben Ali, attended a lunch with more than a dozen executives from British multinationals hoping to win business in Tunisia. Materi, 29, fled Tunisia after his father-in-law was ousted.

It was also reported that Epstein helped to pay off debts owed by Sarah Ferguson after an alleged intervention from the prince, her former husband. Epstein is said to have given £15,000 to a former assistant of Feguson's after the prince allegedly made an appeal as part of efforts to help her avoid bankruptcy.

As unease grew at senior levels in Whitehall about the prince's links with a convicted paedophile, the prince held talks last week with Sir John Cunliffe, the prime minister's most senior adviser on overseas business and Europe. The meeting was used to assess his future as part of the six month "tasking review".

There were suggestions over the weekend that Prince Andrew could be given more of a ceremonial role rather than his official position as trade envoy on behalf of the government body, UK Trade & Investment. Government sources indicated that they would be relieved to see the back of the prince as special trade representative. "It really is up to him," one senior source said. "Will many people be shedding tears if he decided to go? I doubt it."

William Hague, the foreign secretary, said the government remained fully supportive of the prince. "I've seen a lot of benefits that he has brought in countries that I have visited where he's been performing that role," Hague told the Politics Show on BBC1. Asked about the prince's embarrassments, Hague said: "I'm not an expert on what you say are the embarrassments, I haven't had time, with everything going on in the Middle East, to look at those things. But certainly I've seen around the world a lot of good that he has done for this country."

Other Tories were privately less supportive of the prince, reflecting unease in Downing Street about his conduct. One senior Tory source said: "There appears to be no discernible mental activity upstairs as far as the duke is concerned. I feel sorry for him. He has no friends and so is surrounded by these vile people."

But Ed Perkins, the prince's press secretary, had pleaded with Whitehall in an email to show some support.

In an email, leaked to the Telegraph, Perkins said of the disclosure that the prince had hosted a relative of the then Tunisian president: "[I] am deploying the line that he [Materi] was vice-chairman of the chamber of commerce. Will UKTI stand behind him? We need some government backing here."

Alastair Watson, the prince's private secretary, also took the unusual step of commenting in public to defend his reputation. In a letter to the Times, Watson wrote: "The duke has known Mr Epstein since being introduced to him in the early 1990s. The insinuations and innuendos that have been made in relation to the duke are without foundation."