Baroness Warsi! I'd forgotten about her. I always liked the cut of her jib; truly an iron fist in a velvet glove, that one. Cuddly, smiley and intensely rebellious. Just my cup of tea, ie a cup of tea with a shot of vodka in it. But enough about how I take my tea.

A couple of years ago, Baroness Warsi seemed to be everywhere. I don't think I quite noticed that she'd been demoted from her position as co-chairman of the Conservative party; it's tricky when they shuffle a job share. But Warsi was back in the papers, last week, after a mischievous appearance on The Agenda.

The Agenda, in case you don't know it, is a cheery political discussion show on ITV. At the end of each episode, guests show off a mocked-up front page they have created with their "dream headline" for the next day's newspapers.

Warsi's front page featured a pudding bowl containing the heads of David Cameron, Oliver Letwin, Ed Llewellyn and Jo Johnson under the headline: "Number 10 takes Eton mess off the menu."

A low blow. An easy joke. When people say the top brass in the coalition are just a row of identikit posh boys, they are being very unfair. Between them, David Cameron and George Osborne actually represent a pretty wide cross-section of the Bullingdon Club membership.

They are further democratised by Nick Clegg, surely the least posh son of a bank chairman ever to be captain of a Cambridge college tennis team. When Clegg is hit with the "toff" slur, do you know who turns in her grave? His grandmother, Baroness Kira von Engelhardt.

Politics is a pure meritocracy. That's why Gordon Brown's cabinet had two brothers and a married couple in it. They just happened to be the best people around.

There; I think I've exhausted my own weakness for the easy joke. The truth is, I feel sorry for the Old Etonians. Everybody should be judged on his or her own merits. Assuming that toffs are "out of touch" is more modern and fashionable than assuming they have a "natural fitness for government", but it's no fairer.

Anthony Eden went to Eton; Churchill went to Harrow; these schools aren't incapable of turning out great men. It must be incredibly frustrating, now, to enter parliament with a genuine desire to do good, often actually to do it, and be constantly dismissed on the grounds that your easy ride in childhood negates anything you achieve in later life.

Many moneyed children grow up with no drive at all. I feel sorry for them too, but find it bizarre that people often consider it nobler and more honest to be a trustafarian junkie, stuffing the inheritance up your nose and passing out on your drum kit, than to seek election as an MP. There simply isn't enough money or fun in politics to assume that everyone goes into it for purely selfish reasons. You have to trust there is an element of social conscience.

It may be easy for me to make the empathic leap into this frustration; I admit I get quite bored that, after 20 years of writing newspaper columns, people still shout that I've only got the job because of my father. They wildly overestimate the money involved (I subsidise it with gambling), the security of the industry (none) and the power my father wielded as an erstwhile magazine humorist (if any, he didn't know it).

However, I'm not entirely self-deluding. I know that I'm probably far more pedestrian and less talented than many who dreamed of becoming writers but couldn't see the road so easily.

My father never had to worry about that. The son of a handyman and a hairdresser, he had to fight even to stay at school beyond 15. He won scholarships to Oxford, Yale and Berkeley and became editor of Punch in the teeth of widespread establishment snobbery. He was obviously special.

Which brings us back to Baroness Warsi. She was the first female chairman of the Tory party, the first Muslim to serve in a British cabinet and the first Muslim woman to serve as a minister in the UK at all. One of five daughters of an immigrant Pakistani mill worker, she went to a secondary school in Yorkshire whose only other "notable" alumnus is a boxer I haven't heard of.

Just think of the prejudices this woman is trapped between. Having broken through those glass ceilings, she has then lived with her every appointment being described as "tokenistic".

When the OEs gnash their teeth in frustration at being dismissed for their social background, they might pause to imagine what it's like to be Warsi. No Muslim woman had ever got there; she gets there and they say it was easy because she's a Muslim woman.

That's some deep bigotry from inside her own party; meanwhile, she's been pelted with eggs by a protest group that says she's not a "proper" Muslim because she supports a militaristic UK government.

How much backbone must that woman have, how strong a sense of purpose, how much diplomacy and drive and wit, to keep pursuing her goals along this tightrope? You can't assume that someone's a mediocrity just because they had no odds to overcome – but when they overcome odds like hers, you know that they are not one. I disagree deeply with many of Warsi's opinions, but you'd have to be an idiot not to recognise she's special.

And yet they demoted her. When she pleaded to keep the chairman's job, nobody deferred to her judgment. Why not? What did they fear in this woman's power? How difficult is it to see what they're failing to see? That is the question we should be asking, when we judge their mettle, not where they went to school.