Let church bells ring, and this time not for the Queen. Ed Miliband has admitted Labour got its immigration policies wrong in office. That's true – blindingly obviously so. Assorted ex-ministers have been saying so for some time now (after they'd safely left the government), but it's good to hear it from the man in charge.

Tory governments have also made mistakes on immigration policy since the Empire Windrush landed the first of what we then called West Indian newcomers in 1948. The Attlee government was gearing up to restrict the flow when it lost office in 1951, and the incoming administration settled for cheap labour in an expanding economy. We are still "addicted" to the habit today, the CBI told David Cameron last week.

When the Tories imposed restrictions on arrivals from the Caribbean and Indian subcontinent in the 60s Labour squealed "racism", but it did similar things in office. Remember Labour's curbs on the expelled Kenyan Asians in 1968, and how much more decently Ted Heath behaved when a similar fate befell Uganda's Asian citizens a few years later. Remember, too, that a lot of Labour voters backed Enoch Powell's grisly warnings of the period.

One of the more interesting points made on Miliband's behalf came from Matthew Taylor, the Blair-era No 10 boffin who now runs the RSA thinktank. The immigration debate was no longer about race, he said, because the target of concern is now the east Europeans who have flocked here since New Labour's big miscalculation that gave citizens of new EU member states such as Poland unrestricted access after 2004.

Nice try, Matthew, and you're half right. But it isn't so for some people, any more than it's credible to say those nasty grooming cases in Lancashire – another dreadful one in the news today – have little or nothing to do with the exploitative cultural attitudes of some Pakistani immigrants towards vulnerable young white girls. Such attitudes cut both ways and the posh people running MigrationWatch sometimes seem almost as happy to stir the pot as the BNP.

As everyone now knows, the influx of Poles and their neighbours into Britain (not France, where "Polish plumber" is a swear word) after 2004 was not the 13,000 once predicted, but more than half a million. Not all of them went home – why should they? – once they'd saved up for a car or house, or once sterling slid against the zloty, making the UK a less attractive place to work and save. That was predicted, too.

Did ministers lie, as Lord Glasman (remember him?) now says? No, they deceived themselves, as people often do. Jimmy Carr has repented his tax arrangements this week. Now it is Labour's turn on immigration. Can Ed Balls's confession to errors of economic management be far behind? Well, since you ask, I'd guess miles behind. But it will have to be done one day if Balls is not to waste the rest of his career.

In his Guardian interview with Patrick Wintour and his speech today, Miliband is saying Labour was too quick to dismiss the concerns of ordinary people as prejudiced – "bigoted", in Gordon Brown's famously self-damning phrase, which Miliband tactfully omits. His answer is to try to reform what he calls a "brutish" labour market. How? By requiring firms with 25% non-UK employees to register, by tightening up employment agency "local connection" rules to prevent discrimination against locals, by not repeating the Blair/Brown access error when Croatia joins the EU and – we hope – by making higher skills for the unskilled pool of British workers a greater priority.

Good, though coalition ministers promise similar things. I heard Theresa May this week saying she would succeed in cutting net immigration to tens of thousands while not excluding the skilled people the City is always asking for (they never fill their quotas, May said). Good luck. In a mobile world where travel is relatively easy and a liberal, English-speaking, open economy is a magnet, it won't be easy.

Half a million French people now live here, most young and talented, and François Hollande's new policy pledges has provoked David Cameron to invite a few more while France gets our sun-seeking OAPS. That sounds like a bargain. Unfortunately, hot money refugees from Russia, Egypt, Greece and other troublespots make some middle-class Londoners, the younger ones, feel as threatened in their search for homes, jobs, schools and doctors as the poorer Labour voters Miliband hopes to reassure.

Today's Guardian leader sets out with admirable fairness the case for action and the dangers of not taking any. It quotes a report (pdf) by the National Institute for Social and Economic Research to support the view that an influx of newcomers willing to work does not depress local wages nor push up local unemployment, as Tory papers routinely assert.

Employment rates that peaked at a startling 74% in 2004 remain very high, but unemployment rates have tripled to more than 8% and official analysis seems to confirm that a large proportion of new jobs created in the economy have gone to people born overseas. Put it another way: "How can my unskilled constituents compete with a highly motivated, multilingual Pole?" as a Glasgow Labour MP asked me years ago.

It's a good question, and at a byelection in Glasgow a local baker explained in my hearing that he always tried to employ fellow Glaswegians to make his bread (he competed with five supermarkets within two miles) but ended up employing Poles and other foreigners: benefit withdrawal for paid work didn't make it worth employing local people.

I've heard this time and time again in different parts of the country, along with complaints that foreign workers are undercutting local wages and being employed on terms that English people would not accept. "How can I compete with a group of Poles sleeping in a van?" a Welsh electrician once asked me. "They're taking our jobs," they say on the big farms of Lincolnshire.

"The locals don't have the skills or the motivation, they have too many excuses," counter employers of the kind Peter Walker interviewed for today's Guardian. Why can't they both be right? Plenty of people look bloody hard for work and succeed against amazing odds. But some people are content not to bother or – so Iain Duncan Smith now suspects – work for cash and top it up with benefit.

That component of the debate was the factor missing from Matthew Taylor's exchange with the leftwing Labour MP Jon McDonnell on Radio 4's Today programme, and with John Denham, Miliband's righthand man. It was Denham who alerted me to the evidence that people most likely to be English nationalists and anti-EU (he is a Southampton MP, one of few Labour MPs in the south) are those most fed up with things in general.

So Miliband is right to say that cost-cutting employers are part of the problem. So are agencies who look for docile foreign workers who may not mind sharing a crowded caravan or working for less than the minimum wage. But – as Peter Walker also found – skills and motivation are also part of the remedy.

Blaming migrants for shortages of houses or undercutting wages is wrong, Jon McDonnell says. His remedy is restored trade union rights to enforce decent wages (anyone for the BMA or the London bus drivers this week?) and a resumed programme of house-building. Everyone benefits from immigration ("I am the grandson of Irish immigrants"), the MP says.

Good for him, but it is in the short-term that rents rise, school rolls and GP surgeries are under pressure and the worst losers are usually near the bottom of the pile. Miliband was right, belatedly, to acknowledge their concerns. Now to the hard bit. You surely don't need to be a multilingual Polish graduate to pull a pint or serve a meal. But all over Britain it sometimes feels that way.