In the movie Talladega Nights, motor-racing champion Ricky Bobby celebrates his victory with an obscene gesture. When his team’s owner complains about the money and prestige his profanity has cost him, Ricky Bobby replies: “With all due respect Mr Dennit, I had no idea you’d gotten experimental surgery to have your balls removed.”

“What did you say?” asks Dennit.

“Well, I said it with ‘all due respect’,” replies Ricky Bobby.

“That doesn’t mean you get to say whatever you want to say to me,” retorts Dennit.

When it comes to catch-all caveats, “I’m not a racist” ranks alongside Ricky Bobby’s “all due respect”. They’re the magic all-purpose words designed to draw poison from the inevitable offence to come; a necessary declaration of self-absolution before the gratuitous sin.

No one’s ever racist nowadays. But somehow racism still seems to exist and find its target. Few doubt that certain views pervade and practices persist but even fewer will own up to holding them or following them. A system of discrimination remains, yet no one, apparently, is running it. So while those who take responsibility for it are rare, those who suffer the consequences are many.

So it has been over the last week with the Roma in Britain. The deputy prime minister, Nick Clegg, provided his disclaimer. “I am a liberal,” he said. “I think one of the great things about our country is that we are open-hearted and generous-minded.” Describing the Roma as sometimes “intimidating” and “offensive”, he also said: “We have every right to say if you are coming to live in Britain … you have got to be sensitive to the way that life is lived in this country.”

Sensitivity is a great thing, but mythology is dangerous. One wonders how the Roma should indicate their inculcation into British values. By encouraging their teens to get hammered on a Friday and Saturday night and start brawling? Or by stigmatising the next group of immigrants who come after them? For that too is the way “life is lived in this country”.

Meanwhile, another Sheffield MP, Labour’s David Blunkett, warned of “explosions” on a par with the northern riots over a decade ago if the Roma didn’t integrate. Ukip’s Nigel Farage praised Blunkett for his “courage”.

There is nothing courageous about slandering a group of impoverished, marginalised people. They’re too poor to sue and too isolated to effectively resist. There can be no comeback because they have no power, so where’s the courage? But there is everything racist about denigrating a group of people as though their shared ethnicity means shared values and implying collective responsibility for the actions of individuals in their community.

Nor does there seem to be much basis for this moral panic. The Guardian recently reported that South Yorkshire police say crime has not increased significantly since the Roma moved to town a few years ago. After a chip-shop owner swore two teenagers tried to sell him a baby, the police scoured CCTV footage and records of babies born in the area and found nothing. A police spokesman said it “could have been a joke in poor taste”. That didn’t stop it making the front page of the Express. What’s left, according to Blunkett, is littering and hanging out in large numbers. But there are laws against littering and obstruction. They shouldn’t be applied specifically to the Roma – but nor should the Roma be specifically spared.

To be against demonisation is not to be in denial. The arrival of a large number of poor people does demand resources to facilitate their integration. Those challenges are most likely to fall on working-class communities that are least equipped to meet then, their capacity further diminished by the swingeing cuts of Clegg’s government, including – as Blunkett pointed out – the Migration Impact Fund. But the challenges are because the Roma are overwhelmingly poor, not because they’re Roma.

The truth is the Roma have far more to fear from non-Roma than vice-versa. Gassed by the Nazis, forcibly sterilised by the Swedes, recently expelled by the French, they have long been persecuted. In the last six weeks, two Roma families in Ireland, accused of stealing children because they didn’t sufficiently look like them, had their kids taken away from them by the state only to have them returned. In Serbia, skinheads tried to snatch a blond child from in front of his house for the same reason.

This discrimination is not legitimised by the fact that the villification comes from a multiracial group, as is the case in Sheffield. Racism is about power not colour. A mob is a mob, whether it’s decked in the union jack or looks like the United Colours of Benetton. The Residents Association of Page Hall, the area of Sheffield where the Roma are concentrated, is on patrol, to monitor and correct their behaviour. To his credit, Blunkett also called on local communities to reach out. If there is anything to fear it’s not that there’ll be a riot but a pogrom.

I saw this play out in 2002 in rural Hungary, where the Kolompar family, a young Roma couple with two kids, had been chased from a new home by a mob rallied by the local mayor of Nemetker. With the crowd threatening to burn their house if they moved in, the Kolompars turned back. The villagers then trashed the house, damaging the roof and destroying the doors and windows so that it was uninhabitable.

Paradoxically, the plight of the Roma in eastern Europe was so bad that securing minority rights for the Roma was a precondition for countries from the region joining the EU. Polls show that 91% of Czechs had “negative views” towards them while a survey of Hungarian police officers revealed that 54% believed criminality to be a key element of the Roma identity. In the Czech Republic, 75% of Roma children were placed in schools for people with learning difficulties; in Hungary it was 44%. The mayor of Mendez, a small town in Slovakia, said: “I am no racist … but some Gypsies you would have to shoot.”

To counterbalance integration against the threat of riots is basically the Tebbit test without the sport. “Where you have a clash of history, a clash of race then it’s all too easy for there to be an actual clash of violence,” said the former Tory party chairman.