Chris Grayling's defence of the right of B&B owners to turn away gay couples is an echo of the bad old days when some landlords used to stipulate: "No blacks, Irish, gays or dogs."

The equality laws exist to protect everyone against discrimination.

But Grayling apparently believes that some people – homophobic people – should be above the law. Why this exception? After all, he does not agree with B&Bs refusing accommodation to black or Jewish couples. If race discrimination is wrong, why is Grayling saying that homophobic discrimination is right?

According to the law, no person providing services to the public, such as B&B accommodation, is allowed to discriminate. Quite right too. If a Christian or Muslim were turned away by a B&B, I would be first in line to oppose such discrimination. Why should discrimination against gay people be treated any differently?

Permitting some forms of discrimination opens the door to more and more demands for further exemptions from the equality laws, as we've seen with religious registrars refusing to perform civil partnerships and religious solicitors refusing to handle gay asylum cases – all in the name of their "conscientious belief".

Grayling is not a religious fundamentalist. He is in the shadow cabinet and wants to be home secretary. Yet he opposes the comprehensive protection provided by the equality laws. He is against the law as it stands.

His high-level support for anti-gay discrimination by B&B owners raises serious doubts about the Conservative party's commitment to lesbian and gay equality.

David Cameron needs to clarify urgently whether he agrees with Grayling that B&B owners should have a right to refuse accommodation to gay couples.

Cameron's silence is worrying. Many voters – gay and straight – will be disturbed by his failure to swiftly disown Grayling's support for homophobic discrimination. What does this say about the sincerity and seriousness of his commitment to gay equality?

Some people fear that if the Conservatives win the general election they might amend equality legislation to allow some forms of homophobic discrimination and permit further opt-outs by religious organisations and individuals.

These fears are not unfounded. The Tories don't have any official lesbian and gay rights policies. In fact, they support the ban on same-sex civil marriage and the ban on gay and bisexual men donating blood. The Conservative party annual conference has never voted for gay rights, and gay rights policies do not feature in any Tory policy document. The Conservatives are offering the gay community no new measures to remedy the remaining vestiges of homophobia and transphobia.

The new gay-friendly Conservative image is seriously damaged by Cameron's own voting record in parliament. He voted against gay couples being allowed to adopt children in 2002 and against the repeal of section 28 in 2003. In 2008 he voted against giving lesbian couples access to IVF fertility treatment. In recent years the vast majority of Tory MPs have voted against gay equality, including a third of the present Conservative shadow cabinet. This does not bode well for what a future Conservative government will do on gay rights.

Cameron's pro-gay image has also taken a battering from his decision to team up with Poland's homophobic Law and Justice party (PiS) in the European parliament. The PiS has banned gay pride parades and derided same-sex relationships. It has strong links with the misogynistic and antisemitic Catholic radio station, Radio Maryja. Jewish, women and LGBT – lesbian, gay, bisexual and transgender – voters are rightly appalled to see the Tories co-operating with such a nasty, bigoted party.

Even worse, the Conservatives are in a European alliance with Latvia's notorious rightwing nationalistic and homophobic Fatherland and Freedom party. It borders on neo-fascist, with its annual commemoration of Latvians who served in the Nazi Waffen-SS during the second world war. These dodgy political alliances call into question the sincerity of Cameron's professed conversion to progressive, pro-gay Conservatism.

By their backsliding on gay equality, the Conservatives are missing out on lots of potential pink votes. If they promoted positive lesbian- and gay-rights policies, the Tories could outmanoeuvre Labour and win over more gay voters. Our support could influence the outcome in a tight election contest, especially in marginal constituencies. Gay people – and our straight friends and allies – have got ballots too, and at the next general election many of us will use them to support parties that support gay rights, not parties that want exemptions from the equality laws.