Perhaps no topic in British politics is more poisonous. Just ask Gordon Brown, who saw Labour's 2010 election campaign crumble as the nation heard him brand Gillian Duffy a "bigoted woman" after she had collared him to ask why her native Rochdale was now home to so many eastern Europeans. The emotive power of the subject was clear againtoday, as Ed Miliband discovered that a speech by the leader of the opposition can, after all, gain serious media attention – just so long as it's about immigration.

Part of that power lies in the polarised nature of the debate. In one corner stand the likes of Migration Watch, Ukip and the Daily Express, trenchantly arguing that there are too many foreigners here and it's time we closed the borders. Opposite stand what one expert calls "Guardian liberals", who recoil from any such talk, hearing in it an ugly xenophobia.

Yet the debate is not confined within neat lines of left and right. Immigration's defenders find allies among Economist-reading economic liberals who favour free movement of labour, if only so that employers can hire whom they like. Labour is internally divided on the matter too. While Tony Blair and his disciples preached the globalisation gospel, in which mass migration was an unavoidable fact of life, the Blue Labour tendency seeks to channel the anxieties of the party's traditional, low-income supporters, who fear that new arrivals are changing their communities and threatening their livelihoods. (Gillian Duffy was a lifelong Labour voter.)

In a further twist spotted by Sunder Katwala, founder of the new thinktank British Future, the two sides in the national debate tend to wear each other's clothes. The left borrows neoliberal arguments about prosperity to defend migration, just as the right complains about the wage squeeze suffered by those at the bottom, discovering a concern for the urban poor rarely expressed when not discussing immigration.

Those jumbled lines of division suggest another paradox: that on this most contentious of questions we might agree more than we realise. That, indeed, there is a consensus to be had on immigration – one that could even favour the liberal side of the argument. True, the polling shows stubborn camps at either end of the spectrum – those who would pull up the drawbridge tonight versus those who would keep it open and unchecked. But in between is a majority of Britons who may, depending on the question, appear to pollsters and headline writers as "anti-immigration" but whose views, on closer examination, turn out to be far more nuanced. The question is, who will win over this middle ground?

If those who defend both immigration and migrants are to prevail there are several steps they need to take. The first is to do what Brown so audibly failed to do – listen. That means understanding voters' anxiety of that middle ground, whether it's Mirror-reading fears over jobs and housing, or Mail-stirred concerns over cultural continuity and cohesion. The Labour leader waded into the first, economic area today, perhaps regarding that as safer territory, though even that brought trouble. Mischievous messages of support came from Ukip's Nigel Farage and even the BNP's Nick Griffin, the latter gleefully retweeted by Miliband's critics on the left, as if to suggest that the leader had taken the first step on the grimy slope towards fascism. But such guilt-by-association is not really defensible. Ed Miliband cannot stay off awkward topics just because a thug like Griffin will try to exploit it if he does.

Instead, he has to hear and address the worries people genuinely have. Nor is it good enough to listen, nod politely, then tell the troubled voters they've got it wrong. There are, to be sure, powerful stats and graphs that show migration has been of net economic benefit to the UK. Yet one of Miliband's strongest moments came when a questioner asked about such "aggregate" data. With a smile, he said: "The aggregate is not where people live their lives."

What matters is people's experience. Maybe the economic benefits of migration ultimately outweigh the costs, but that's no comfort to someone who's lost a job and believes he's been edged out by migrants willing to work for a pittance. Yes, the line on the graph may be upward, but that doesn't help if you're on the wrong side of it.

Nor is it sufficient simply to offer tea and sympathy. Reassurance requires action. That means a commitment to make the system work and to police the borders properly. Katwala says, rightly, that it's immigration liberals who should be demanding rigour the loudest: it's those "who want immigration to remain toxic who have an interest in chaos at the airports", he says. Fascinatingly, an Oxford study last year found that, yes, 69% of Britons want to see immigration reduced – but of those, a majority "only" or "mostly" wanted cuts in the number of illegal arrivals. Deal with illegal immigration and much of the poison of this issue would drain away. Next there needs to be action at the sharp end, where the impact of immigration is felt. Miliband correctly criticises Labour in government for failing to enforce the minimum wage or give equal protection to agency workers, lest they be seen as meddling with the labour market. But now, while the Tories are mulling further deregulation, as proposed in the Beecroft report, Labour can demand protections that would simultaneously protect migrant workers from exploitation and reduce that downward pressure on the livelihoods of British-born workers.

And those who want Britain to remain an open society should not assume the public is unbendingly hostile. That same Oxford study found that foreign students arouse little opposition – yet they account for a large share of immigrants to Britain. Those categories whose presence gets people riled – asylum seekers and extended family members, according to the survey – actually represent a very small share of the overall number of new arrivals. Ask Britons whether they want to keep out skilled workers – doctors, nurses or care workers – and most say no.

Gradually the zone of disagreement shrinks. It shrinks further when people accept, albeit grudgingly, that there is little they can do about EU migration, not without giving up the benefits that enable them or their children to work or study in Europe. Eventually the numbers argument comes down to what we should do about the 20,000 or so migrants who come from outside the EU and are currently subject to a government cap.

In the US, some 75% back immigration reform that would tighten the borders but would also in effect see an amnesty for millions of illegal newcomers. Such a consensus should not be impossible to reach here. With proper borders and a labour policy that mitigates the sharpest effects of immigration, we could keep our door open to those who yearn to bring their talents here or need a shelter. It cannot be beyond us.