An embarrassing admission perhaps, but I have only met Charlie Whelan on the one occasion. It was not auspicious. A mutual acquaintance introduced us at a Labour conference. We shook hands and then a split-second later, he spotted someone more important over my left shoulder.

Suddenly he was gone. Politics is like that at the apex. Either you matter or you don't.

Thus I hold no brief for him or his union Unite. But I don't think you have to owe any allegiance to him, his union or even to the Labour party to see that in recent days we have all fallen victim to a conjuring trick inspired by Conservative central office. We have Lord Ashcroft, goes their script, eagerly recited by the rightwing papers and their commentators. But so what, Labour has its own Blofeld figure in the shape of Whelan and its paymasters Unite. We may be a bit sleazy but so are they is the closest thing to a narrative that one can draw from this. It's been quite effective. Hardly anyone is talking about Ashcroft any more. But basically, it is rubbish.

Not even the Tories can seriously believe that it is reasonable to equate the actions of Whelan, who – like him or not – represents many thousands of people, who have democratic channels through which they can object to his stances and activities if they so wish, with the sole force that is Ashcroft.

Unite takes subscriptions from its two million members and channels some of that in a way that should serve their best interests. Ashcroft personally uses his millions as he sees fit to fund those candidates that appeal to him, those that campaign as he says they should. He, a non-dom, has made a personal decision to try to bring about the election of a government of his choosing. One that, God willing, might remember where all that money came from and could cede to him some influence. There is nothing illegal in that. The rich are free to do as they choose with their wealth and to support whatever political movements take their fancy. But only through a seriously distorted lens can one look at Labour's relationship to Unite and the Tory link to Ashcroft and conclude that they are the same.

In truth, this is merely a small part of a larger conjuring trick that we fell for a long time ago, because even Labour voters have bought the illusion that the link between the party and trade unions is intrinsically evil. This is the core belief that has allowed the Conservatives to spin their way out of trouble.

Even people who form part of the trade union movement have bought the idea that the relationship with Labour is best treated as the dirty little secret. And handled that way, the party does indeed pick up a vote or two from Mail readers who commute to the capital from Godalming. But the downside has been ruinous. Look at constituency parties up and down the country and it is clear that the party has hollowed out. The unions and many of the sorts on the left despised by New Labour provided the link for the party to operate in many poorer, inner-city communities. New Labour cut the cord and in so doing left those communities to drift.

If Labour is to rebuild itself in these communities, it will have to do many things, and one will be to reshape its relationship with the unions; organisations that connect with real people, relaunching it as something other than the existing shady, loveless marriage. Indeed, the right answer to those who accuse Labour of being close to Britain's biggest union is to say: "Yes we are, so what"?