"Judged by the intensity and sheer volume of the anti-Warsi vitriol it is difficult to come to any other conclusion than that her critics don’t like her because she ticks three very un-Tory boxes: she is female, Asian and Muslim. Since it is 2012 and they can’t say as much in public, her right-wing opponents target instead her alleged lack of “competence” and “ability”."

"The anti-Warsi camp is very sensitive to the charge that it is motivated by racism, sexism or any other prejudice. It is all just a question of political effectiveness, they insist. That is plainly a bit disingenuous. There are plenty of white Tory men who would love a seat in the cabinet and flatter themselves by thinking they have been passed over because of a positive discrimination policy in favour of ethnic and gender diversity."

The New Statesman is not the only place where you can read similar suggestions. Monday's Independent, Ian Birrell in The Guardian and ex-Statesman hack James Macintyre have all tip-toed through not dissimilar territory. I've ignored it up until now – some of it directed to me and ConHome – because you hesitate to give legs to such sulphurous stuff but it's being repeated too often to ignore.

Let me deal with the racist thing first. Run a poll of Tory members and you'll find that Priti Patel is one of their very favourite new MPs. They aren't interested in her ethnicity but love her straight-talking Tory politics. Similarly there is much enthusiasm in the grassroots for the likes of Sajid Javid MP, Paul Uppal MP and Shaun Bailey as future stars. At the last election non-White Tories were selected in many heartland seats. Helen Grant, Sam Gyimah, Kwasi Kwarteng and Nadhim Zahawi being stand-out examples of candidates prospering among the bluest of blue selectorates. It's true that there are still some racists inside the Tory Party (and in too many other walks of life) but Cameron has repeatedly acted against them when they are uncovered and they are far from representative of mainstream party views.

Critics of Sayeeda Warsi may be wrong. I may be wrong. Perhaps you don't need an MP as Party Chairman. Perhaps it's unfair to think the Tory Chairman should know the level of party membership. Perhaps it's unreasonable to wonder why it's always the super-competent Michael Fallon who is wheeled out to defend sticky wickets on the Today programme. Perhaps it's unfair to want a Tory Chairman who represents the grassroots to the leader rather than the leader to the grassroots. Perhaps it's wrong to want a party chairman with the experience and authority necessary to rethink the electoral strategy that failed two years ago. Perhaps, perhaps, perhaps but these are legitimate concerns and they are not motivated by racism.

Baroness Warsi has great skills. She is very likeable. She's a good speaker. Her conservatism is deep and real. She gets high praise for her work in Pakistan. Her performance against Nick Griffin on Question Time and her recent remarks about the Rochdale grooming scandal are proof that she's able to raise issues that are difficult for other Tories to raise. But I still don't think she's an effective Party Chairman. I, like many activists, think she'd be better suited to another role.

I suggest to people on the Left that they are much more careful about making thinly-disguised charges of racism. I don't expect them to change their ways. They know that mud sticks. They know that they are winning as soon as Tories have to start insisting they aren't racist. The tactic is still gutter politics though – even when its authors use long words in the New Statesman.

PS My colleague Paul Goodman addresses the issue of Sayeeda Warsi's future in The Telegraph: "The architect of this mess isn’t so much Lady Warsi as David Cameron. There is a terrible circularity in the story of their relationship. Her original appointment protected the Conservative Party from accusations of racism. However unfairly, her plight exposes the Prime Minister to precisely that charge. In 2007, Mr Cameron rushed into the politics of ethnicity to get his party out of a tight spot. He may now have the opportunity to repent at leisure.