Ministers are to scrap the right of appeal for more than 80,000 relatives of British families who are refused visas to visit them each year, according to a leaked Home Office policy paper seen by the Guardian.

Senior Whitehall officials have warned that the move is considered highly controversial, particularly within Britain's Asian communities, as well as being legally risky.

Home Office ministers have been told they need to "warm up colleagues in government for these potentially controversial changes", starting with the Conservative party co-chairman, Baroness Warsi, the only British Pakistani in the coalition government.

The immigration minister, Damian Green, has been warned to expect protests from "some Commonwealth countries", implying the move could trigger a renewed row with India and Pakistan in the wake of the recent controversy over the cap on immigration.

Immigration welfare groups have condemned the move as "discriminatory and mean" and said such visits often involved weddings, funerals and visits to dying relatives.

The move echoes the robust approach expected later this year when ministers unveil plans to curb the number of family members coming to settle in Britain as part of the push to cut net migration to the "tens of thousands".

The leaked Home Office submission to the immigration minister outlines a bid for secondary legislation in the new parliamentary session starting this autumn to abolish appeal rights for family visitors. It also discloses that ministers want to scrap the right of appeal for thousands of skilled migrant workers in Britain who want to extend or renew their visas under the points-based system.

More than 420,000 visa applications were made for temporary visits by close relatives of British families in 2010, at a cost of more than £70 each. Of the decisions made last year, 350,000 family visit visas were granted, 88,000 were turned down. More than 63,000 of those who were refused, appealed against the decision and around 36% were allowed to come to Britain on appeal.

The policy paper concedes the appeal success rate "might be perceived as high" but claims a large proportion of appeals are allowed because they involve the submission of further evidence. Officials argue that, in these cases, relatives should have to pay for a new visa application rather than make an appeal.

The paper – written for Green by the UK Border Agency's (UKBA) director of appeals and removals, Phil Douglas – says the move is a "crucial part of plans to reduce the number of appeals and resultant cost to the taxpayer". He says family visit visas are the only visit visa decisions taken by entry clearance officers abroad that still attract a full right of appeal.

"We can expect this move to be controversial – in particular with some Commonwealth countries and UK communities with families overseas – and in previous advice we recommend warming up colleagues in government for these potentially controversial changes, starting with a conversation with Baroness Warsi."

Habib Rahman, chief executive of the Joint Council for the Welfare of Immigrants, said: "There is an entirely justifiable expectation of British and settled people to be able to welcome family visitors. The fact that 36% of appeals are successful demonstrates the paucity of decision-making in this area.

"If a refusal for someone to attend a wedding, a funeral, to visit a dying relative or to be with their loved ones for short periods is refused, the right to an appeal is the only fair way to settle such a matter. This idea is discriminatory and mean and should be abandoned before it gets any further."

Abolishing the right of appeal could save the UKBA between £8m and £12m a year, and the justice ministry up to £24m in the cost of immigration judges and tribunals.

Douglas's letter acknowledges there are "possible challenges" to the move, including on human rights grounds.

"Removing family visit appeals is not without legal risk," he notes. "This is mitigated to some extent by the fact that potentially some residual rights would have to remain to allow for appeals on the basis of the Human Rights Act 1998 or the Race Relations Act 1976."

Appeal rights for family visit visas have a charged political history. They were abolished by Ken Clarke as home secretary in 1992, when he told MPs that while he recognised those who were refused were greatly disappointed they could not come to Britain, this "was not a matter of life or death".

The decision caused many problems, especially in Britain's Asian communities, and Labour made restoring it a manifesto commitment when they came to power in 1997. When Jack Straw reintroduced appeal rights in 2000, he told the Commons that MPs "with Afro-Carribbean or south Asian constituents knew at first hand the strong sense of injustice that the abolition of the right of appeal has engendered amongst many of our constituents".

Family members coming to visit British relatives can apply to come as tourists, but they have no right of appeal if their application is refused. In 2009, a system of "sponsored family visits" was introduced, under which families whose relatives fail to leave when the visa expires face penalties up to £5,000, a prison sentence and a ban on future sponsorship.

Family visitors include grandparents, aunts, uncles, nephews and first cousins, as well as immediate family.

A recent report by the independent inspector of immigration criticised the quality of initial decision-making on family visit visas applications, with more than 50% failing a sample test.