'Anyone who has taken on both the BNP and the imams who wanted her to veil up is an interesting figure' … Baroness Warsi. Photograph: Linda Nylind for the Guardian

Today a bunch of racist hooligans will march in Luton. They are simply stupid men in casual sportswear, and nothing to do with anyone who reads this paper. The English Defence League is just a revamped version of the BNP. Right, is that good enough? Have I passed Baroness Warsi's "dinner table test"? Have I shown that I am not an "Islamophobe" with these sentiments? I fear not. Indeed, I fear the slow response to a fast-growing movement requires more sophistication.

The EDL did not arise out of some political vacuum. They are themselves a clever, post-modern response to our muddled discourse about race, culture, identity and religion. For me to use the word "clever" about the EDL will mean I am automatically branded some kind of fascist. So batten down the hatches. I saw this week the amount of "lefty" self-congratulation floating around after the EDL leader Tommy Robinson (the man has several different aliases) appeared on Newsnight. Jeremy Paxman is back to his sneering best, even using Naughties's four-letter word about cuts. Maybe it was thought that Paxman versus a clueless thug would be the best way to destroy this organisation's credibility. But it wasn't. No one was there to stop the flow.

The man may be inarticulate (ie, working–class) but he made several sharp points that are often found in the mainstream media: about sharia law, "Muslim paedophile gangs" and the treatment of women. This was a coup for the EDL, while the chattering classes mostly amused themselves by commenting on his chavdom.

Many of the same people who thoroughly condemned Robinson's "racism" had earlier sat through My Big Fat Gypsy Wedding, and had been appalled at the way traveller women were treated. Is it racist to say that the traveller technique of "grabbing", a courtship ritual leading to the mad "fairytale" wedding, is not something to be approved of? Is this another cultural difference that we can't comment on? We now watch the white working class in programmes like The Only Way is Essex with incredulity. Aren't these idiots with their perma-tans and stupidity ridiculous? Look at those pikies!

So I don't agree with Sayeeda Warsi that anti-Muslim sentiment is the last socially acceptable form of bigotry. I think there are all kinds of bigotry out there, but simply some are spoken about more than others.

The reaction, though, of both the left and the right to Warsi's speech about prejudice against Muslims told us a lot. The right wing dismiss her basically as a dumb token who is a disaster when she speaks out. The left does much the same and points to her "homophobia", as if this negates all that she has to say. Personally I think anyone who has taken on both the BNP and the imams who wanted her to veil up is an interesting figure in the current political landscape.

Some of the things that Warsi was getting at in her speech, and indeed Robinson did in Newsnight, is the stuff that many people say when they think no one is listening. It is everyday speech. Remember, the EDL formed in 2009 after a demonstration against the returning troops from Afghanistan. Banners were held up saying "baby-killers" and "butchers of Basra". This demonstration was organised by Al-Muhajiroun and included members of Ahlus Sunnah wal Jamaah. These are extremist groups. This is the bit where it is obligatory to say that most Muslims are not like this. The EDL evolved, if that is the right word, into the "United Peoples of Luton" in response. Many of its members have BNP pasts, criminal convictions and come out of the hooligan firms of the football casuals. This is also the bit where I will say that not all EDL support is so straightforwardly thuggish, either.

I agree with Jon Cruddas, who has fought the BNP and won in Barking and Dagenham, that the EDL may crash and burn. "But it may not, because it taps into a politics born out of dispossession but anchored in English male working-class culture of dress, drink and sport." These people may formerly have been traditional Labour supporters, but I am afraid the language they speak echoes eerily all that New Labour told us about extremist Islam. To see the far right use women's rights as an excuse for a ruck would be funny if it were not simply dangerous. The EDL is apparently very concerned about the treatment of Muslim women. And so was our government once. Isn't that one of the reasons we went into Afghanistan? The EDL are now cutting their interviews with footage of a horrific stoning of a woman in Afghanistan. Meanwhile, our leaders are indeed talking to the Taliban. The EDL also speaks of the civilizing force of Israel in the Middle East, and in this momentous week I can't help but note that at the beginning of the protests all reporters were raising the subject of "radical Islam", while the people on the street were not. This fear of radical Islam is not conjured out of thin air. The leaders of the EDL live in a violent world. It's not in their heads. They spread fear and hatred. They intimidate and are intimidated in return.

While the BNP was racist, anyone can join the EDL. Don't all rush. They boast of their Jewish division, of their Sikh leader, their lesbian and gay contingent. Aren't they the very model of diversity? Unlike the BNP, they don't target purely racial identity. This is why it is too simplistic for organisations such as Liberty to dismiss them as modern-day "black shirts". The EDL are using the language of inclusiveness so beloved of liberals, whilst so-called "progressives" sit on sidelines shouting Islamophobia or Islamofacism at each other.

The failure of successive governments to deal with what is euphemistically called "social cohesion" is precisely what leads to extremism. The complete inadequacy of particular kinds of multiculturalism is yet to be properly acknowledged. In education, for instance, many "multicultural" approaches have been entirely top-down and imposed in increasingly inept ways. The EDL can exploit ignorance and fear because in many of the places they find support different communities are still leading "parallel lives".

The EDL slogan of No Surrender, a Loyalist slogan (or just possibly the title of a Bruce Springsteen song) also tells us a lot about their roots and influences. Far-right movements will, of course, thrive during a recession. Certainly, one of the things we need to do is to understand the new right. For the left to rise again in any meaningful way, we have to deal with the concept of Englishness, and stop the official silence around race and culture. The crude stereotyping of the EDL plays into their hands. As with their Dutch counterparts, they are using a language of libertarianism, modernity and fake inclusiveness. They know what they are doing, just as Al-Muhajiroun does. Such groupings may in fact be mirror images of each other. Any anti-fascist movement cannot take on one without the other. That is indeed complicated. But any other way is indeed surrender.