A Sikh man weeps as he hears about the shooting at the temple in Oak Creek, Wisconsin. Photograph: Scott Olson/Getty Images

"It's a common thing," says Balvinder Kaur Saund, a Labour councillor for the London Borough of Redbridge and chair of the Sikh Women's Alliance, "to walk through an estate in [some parts of east London] and young boys throw stones and shout 'Taliban' at you. I have seen that myself." She was once with a man – a Sikh who wore a turban – when some youths shouted "Osama" at him. "I wanted to retaliate but he said 'Just ignore it and carry on walking'." She thinks many more incidents go unreported.

The terrible events in Wisconsin at the weekend, where a gunman shot dead six people at a gurdwara, is an extreme, and thankfully rare, attack, but there are numerous instances of Sikhs being targeted in the US and the UK. The first victim of a backlash against Muslims in America after 9/11 was Balbir Singh Sodhi, a Sikh who was shot dead. There have been other physical attacks on people, and on gurdwaras. In March this year in the US, a letter was sent to a Sikh family – addressed to the "Turban Family" – claiming to know they had links to the Taliban. Mitt Romney, potentially the next president of the US, twice referred to Sikhs as "sheikhs" in expressing his condolences over the temple shooting.

After the London bombings in 2005, a gurdwara in Kent was firebombed, and another in Leeds. After 9/11, former Metropolitan police officer Gurpal Virdi remembers: "I dealt with a lot of complaints – we had attacks on individuals, even on women. When it happened, I was going around stations to tell the officers the differences between the Taliban and a Sikh because they didn't know either. And this is London, a multicultural area. It doesn't help when they kept showing images of [Osama Bin Laden] and in the UK, it's the Sikhs who wear a turban."

And then there are the everyday occurrences witnessed by Balvinder Kaur Saund, where Sikh men in particular are eyed suspiciously, as if they are proud members of al-Qaida.

Varinder Singh, co-founder of the Turban Campaign, which tries to counter negative reporting of the Sikh community, says: "Racism has always existed, but certainly September 11 didn't help. It created a suspicion of the Sikh community, particularly because of the appearance of the male members. Inevitably that leads to physical and verbal attacks, but I think the recent attacks in America are a step beyond that. That was deliberate and I think the argument of mistaken identity can't really be used here – there were big signs outside saying 'Sikh temple'. I think a general hatred has developed. Islamophobia is rife and applies to everyone of colour, regardless of religious background. In the eyes of people who wish to hate or belong to fascist organisations you are the enemy and seen as a potential target. In this country, there is a lot of emphasis placed on extremism in certain corners of the Muslim community, however the government and councils tend to overlook far-right organisations such as the English Defence League."

In May, the EDL tried to ally itself with the Sikh community by hijiacking a Sikh protest in Luton outside a police station after allegations that a Sikh woman had been raped by a Muslim man. "When we see groups such as the EDL try to align themselves with the Sikh community, it's a relationship that can't last," says Varinder Singh. "This is a move to isolate the Muslim community who need our support."

Is there a danger that in trying to differentiate themselves from Muslims – and there have been historical tensions between some Sikhs and Muslims – Sikhs are themselves contributing to Islamophobia? "It does happen and it is dangerous," says Sabby Dhalu, joint secretary of Unite Against Fascism. "When I put the argument to people, a lot of them agree with me in opposing Islamophobia and the need to work with Muslims. It's all the more reason for the decent majority to speak out."