The exchange came as Mr Hollobone used a Westminster Hall debate to highlight the cost of locking up 10,834 foreign nationals – around 13 per cent of the prison population.

Guy Opperman, a Conservative MP for Hexham, asked whether Mr Hollobone agreed money from the Department for International Development and Ministry of Justice should be used to "improve" foreign prisons, thereby enabling deportation of criminals based here.

"I have to say I don't think my constituents in Kettering are that fussed particularly about the human rights of foreign nationals who commit crimes in this country," Mr Hollobone said.

"I think it does make sense to use the huge and increasing international aid budget to build suitable prisons in countries who provide us with a very large number of prisoners."

The Tory MP named Jamaica, Pakistan, Nigeria, Somalia, India and Bangladesh as countries on the "list of shame" when it comes to foreign criminals in Britain, having around 2,900 nationals in English and Welsh prisons and costing taxpayers £100 million. He noted they received £973 million in British aid each year, according to 2012 figures.

"I would go so far as to say I think we should make our international aid to these countries conditional on their acceptance of a prison building programme," Mr Hollobone said.

"I don't think we should give these countries international aid if they are not going to cooperate with us in this regard."

Mr Penning said he did not think aid money should generally be spent on new prisons. He later added that Britain should deport foreign criminals even if the prisons they are going to "happen to be not quite up to the standard that they are in this country".