Today Lady Warsi, the first female Muslim member of a British cabinet, will say in a speech that more needs to be done to combat "bigotry" against Muslims. She will also warn that such prejudice is increasingly seen as socially acceptable in Britain and that it even passes "the dinner table test", ie that comments directed against Muslims pass without question in polite society.

While this shouldn't be blown out of proportion, Warsi has a point. This country has made huge progress in recent years with regard to attitudes towards ethnicity, sexuality, gender and disability. And Britain remains one of the best countries to live in if you are a member of a minority group. But all is not well.

The rise of informal anti-Muslim social movements, such as the English Defence League, is one of the big stories of the last two years. Although these groups claim to be against "Muslim extremism", many of their protests are simply against the building of new mosques – in other words against ordinary Muslims and religious buildings that have no relation to extremism. The sudden popularity of these groups is one indication of rising antipathy towards Muslims among some sections of society.

Similarly, in the press, most media stories that involve Muslims and Islam tend to be negative – although the same could probably also be said of Christianity and Christians. The difference however is that most Brits are much more comfortable with Christianity than they are with Islam. For instance, while ordinary British people can understand that radical Christians (such as the pastor Terry Jones who was yesterday banned from the UK) are a fringe minority, Islam remains a largely unknown quantity, the exotic "other". This means that ordinary British people reading overblown press reports, for instance about the antics of Anjem Choudary, may not know that such individuals are fringe self-publicists who are barred from every mosque in Britain.

International events also contribute towards negative perceptions of Muslims, especially with the actions of so-called "jihadist" groups around the world in recent years. It is not surprising that Muslims are viewed with suspicion when groups such as al-Qaida claim to act in the name of all Muslims worldwide. In comparison, moderate groups of ordinary Muslims often struggle to be heard.

Warsi's comments are important, however, because they bring the discussion of anti-Muslim prejudice into the mainstream. Too often, mainstream parties have been reluctant to talk about these issues – often due to a general uncertainty about how to approach issues dealing with Islam and Muslims.

In this political vacuum, extreme Islamist groups have tried to hijack the "Islamophobia" debate for their own political ends. This is dangerous as it allows these groups to pose as defenders of Muslims and to urge Muslims to remain in their ghettos, fearful of the hostile outside world, with Islamists as their gatekeepers. Ironically, though, these groups are themselves responsible for much of this "Islamophobia" – by presenting their intolerant fringe interpretation of Islam, which they believe in imposing on everyone else, as being the religion's default version rather than as the unpopular minority reading that it undoubtedly is.

However, that is not to say that Warsi is entirely right. The stirring-up of hatred and prejudice against all Muslims needs to be clearly distinguished from criticism of aspects of contemporary Muslim practice. Her reluctance to divide Muslims into "moderates" and "extremists" also seems counter-productive. Extremists exist in every religion and we shouldn't be afraid to say so.

Indeed, much suspicion of Muslims arises from the fact that many British people are not sure which Muslims are extremists and which are moderate. If anything, therefore, we need to clearer about who extremist Muslims are, what they want – and why such extremists differ from the majority of ordinary British Muslims.