More than 100 universities, further and higher education colleges, and independent schools face losing their right to sponsor overseas students under a fresh drive by Tory ministers to arrest the latest unexpected rise in net migration to Britain.

The drive was signalled by immigration minister James Brokenshire, in a speech in which he accused the Liberal Democrat business secretary, Vince Cable, of telling "falsehoods" on immigration and claiming it was the "haves like Vince" who were among the winners from immigration, while working-class families lost out.

The minister framed his announcement as a response to a letter published in the Guardian last week and signed by more than 160 academics in protest at being used as an extension of the UK border force as they come under increasing pressure to check the immigration status of their potential overseas students.

He told the Demos thinktank: "I'm sorry, but is it really unreasonable to require universities to ensure students are genuine? After all, they would hardly admit a British student without checking their A-level results."

He not only rejected the criticisms that universities should not be required to act as border police but went on to warn those that did not do the job properly that they could face losing their "highly trusted sponsor" status and their right to bring in overseas students.

Brokenshire applauded the fact that student visa applications had fallen by 29% since 2010 but said he wanted to clamp down even further by reducing the proportion of visa refusals that educational institutions are allowed before their sponsor licences are suspended.

He wants to see the threshold of a maximum of 20% of foreign students who are offered places turned down by the Home Office for a visa reduced to 10%.

"Clearly, the existing rules which allow such a large margin are extremely generous," he said.

The Home Office said reducing this refusal rate threshold from 20% to 10% could lead to 105 of the 1,700 educational institutions able to recruit overseas students losing their licences.

About 70 of the 105 are higher and further education colleges, and the rest include universities alongside independent schools and English-language schools.

The Home Office refused to name any of the institutions that may be affected.

Brokenshire, who is politically close to his boss, the home secretary, Theresa May, also ramped up the Tory attack on their coalition colleagues, the Lib Dems, putting in question just how much of a joint coalition policy on immigration actually remains.

In his first speech as immigration minister following the resignation of his predecessor, Mark Harper, after he employed an illegal migrant, Brokenshire indicated a far more combative approach to the subject.

Although he insisted he could still work with Cable, he accused him of being condescending and spreading "falsehoods" on immigration.

He said he was "simply incorrect" to claim the 58,000 rise in net migration to 212,000 a year was "good news" as it could force down wages and displace local people from the job market.

He said Cable was also wrong to claim the rise in net migration was due to more people emigrating when a doubling of EU migration lay behind the rise.

He also said Cable had been wrong to say overall evidence was that immigrant workers did not put British workers out of a job.

Brokenshire seized on the sudden publication by the Home Office of the "suppressed" cross-government report on immigration to back up this claim.

"As I have explained, the evidence actually suggests that immigration – especially when it is in large numbers and especially when we're in difficult economic times – can displace British workers.

"That's what the migration advisory committee concluded and it's also the finding of the so-called 'secret report' we have published today. So, wrong again."

In fact the report's main conclusion found: "There is relatively little evidence that migration has caused statistically significant displacement of UK natives from the labour market in periods when the economy is strong."

It does, however, say there is evidence of some displacement of British residents, particularly by non-EU migrants in recent years when the economy was in recession but declines to put any figures on how many British workers were unable to get jobs because of the use of migrant labour.

The researchers add that the greatest impact is likely in lower skilled jobs but the effect gradually declines.

The Home Office-published report says there is little evidence that migration from within Europe has any significant impact on the employment of British workers during good times and bad.

Brokenshire, asked by the Guardian if there was any evidence of British workers losing out to migrants now at a time of economic growth and falling non-EU migration, acknowledged that the evidence was only historical but said it could not be ruled out.

Cable was due to hit back during his Mansion House speech, observing that he could not speak positively about the contribution of migrants to Britain without feeling the need to put on a reinforced tin helmet.