Mainstream voters routinely tell researchers that they are both angry and mistrustful of Labour over the surge in immigration levels since 1997. Hardliners see the record as a deliberate attempt to impose a multicultural society on a nation that hadn't asked for one. Enter stage left, Alan Johnson, trying to lower tempers in his speech on Monday.

Angry talk was hardly new 41 years ago when Enoch Powell predicted those "rivers of blood" in a speech with inflammatory language that even Nick Griffin might hesitate to use today. What has given it fresh salience is Griffin's appearance on Question Time and a hastily written newspaper article it prompted. In it Andrew Neather, a young civil service speechwriter turned Fleet Street journalist, set out to celebrate multicultural Britain and upbraided Labour ministers for lacking the nerve to do the same. In the process he claimed to have seen "an early draft" of a long-forgotten No 10 analysis paper that saw inward migration as a means to "rub the right's nose in diversity".

That was enough for conspiracy theorists and some people such as Frank Field MP, who should known better, to detect a secret plot. Neather quickly backtracked and called for a less polarised dialogue. Johnson's speech on Monday was his contribution. The text included a large wedge of apology and explanation: apology for "maladroit" handling of sensitive issues too long ignored by governments of both parties, explanation of what Labour has done to put things right since 2004 when Charles Clarke, briefly home secretary, issued a reformist white paper, including a points-based system (PBS) for economic migrants; an e-borders system that is gradually logging who enters and leaves Britain; and a much more efficient means of processing asylum claims within six months, although "rooms full of files" remain to be cleared.

His apology startled some liberal observers. But Labour policies in office justify it. In 1997 Jack Straw abolished the "primary purpose rule" that blocked many marriages on suspicion that many fiances were really economic migrants. As asylum applications rose UK ministers also declined to join EU neighbours like France in restricting workers from the new member states, notably Poland, heading west in 2004. That was almost certainly the moment Labour was seen to lose control of its policy. Instead of the predicted 13,000 new arrivals, up to 1.5 million eventually arrived.

Nor did Labour ever sustain a confident story to promote the wider benefits of welcoming hard-working, law-abiding ambitious newcomers to core voters who feared for their crowded schools, hospitals and jobs. "Legitimate concerns," Johnson called them.

Even that offended some analysts and senior Labour colleagues who insist such problems are exaggerated by extremists and the media. "Have we had race riots or social breakdown? No," says one. Mainstream voters are open to persuasion and the recession has sent some Polish plumbers home. But the line between hysteria and complacency is a dangerously fine one.