The Local Government Secretary suggested plans being considered by the Liberal Democrats to increase council taxes on holiday homes were “virtually unenforceable”.

Deciding whether someone lived in a home or used it as an infrequent bolt hole would require a “new Stasi-police” staking out properties and recording “how many times milk bottles are removed from the steps", he said.

Last month the Liberal Democrat leader revealed his party planned a fresh tax on holiday cottages because second home owners are “gutting” rural communities.

The Coalition has already granted councils the power to charge the full rate of council tax on holiday homes but Nick Clegg said he will “constantly look” at how they can go further.

More than 165,000 Britons own a holiday home, according to the 2011 census. Of these, 23,000 have properties in Cornwall.

The area with the highest concentration of holiday homes is Gwynedd in North Wales. There are 121,874 usual residents and 7,784 holiday homeowners – meaning 64 “townies” for every 1,000 locals.

In North Norfolk, there are 4,842 holiday homeowners and 101,000 residents – a ratio of 48 holiday dwellers for every 1,000 locals.

Speaking at the Local Government Association Mr Pickles added that although the Government technically had the powers to confiscate second properties in a “Che Guevera” style land grab the Conservatives would also be “reluctant” to consider that option.