Earlier this month I returned my degree to Oxford University. This was in protest at the failure of my old college – Exeter – to adequately address the widespread offence caused by them hiring out the college to a fundamentalist Christian group. The college authorities fail to grasp that the defence of the rights of all students requires affirmative action, not mere lip service to past achievements. Exeter's refusal to either cancel the booking or to apologise for it has provoked widespread condemnation, both within the University and the national press.

The arrival this weekend of The Wilberforce Academy will be met by student protests. The conference is run by Christian Concern, who state that the conference's aim "is to raise up the next generation of Christian leaders who will take a bold stand for Christ within their spheres of influence." Christian Concern are a political pressure group, whose adherents hold views at odds with the majority of Christians, including those in Exeter College chapel, of which I was once a part. They oppose gay marriage and gay relationships as unnatural and immoral, support reparative (cure) therapy, and want abortion to be made illegal. Its CEO believes Islam to be the work of the devil.

While I welcome their decision to set up a working group to examine the procedure for future bookings, they could have begun by stating that Christian Concern would not be back next year. They have not. The working group has no timetable to report and no assurance has been given that its recommendations will be implemented. They could have issued an apology. They could have asked Christian Concern to withdraw and made the fact public, regardless of their response. They could have co-ordinated a positive demonstration of inclusivity and tolerance for their existing students. They could have flown the rainbow flag at the end of LGBT History Month, rather than become one of a handful of colleges that failed to do so.

Furthermore, to say, as some have, that the conference should go ahead on grounds of free speech is erroneous. This is not a debate in the Oxford Union, nor is it an academic conference in which the views of Christian Concern will be debated alongside opposing views from both within and outside the church, with repudiation from psychiatric bodies and academics. This is a private conference, hiring space, making full use of their prestigious association with Oxford University but going utterly unchallenged.

I, along with many others, would defend the freedom of people to speak out in the public sphere. But this is not the public sphere – it is a private community in which the consideration of its members and the defence of their rights should be paramount. It is the difference between defending the rights of the BNP to be heard while not wishing to invite them into my living room.

This is not just a minor skirmish within an ivory tower but speaks to our ongoing national debate over conflicting rights. At its heart should be an understanding that there is a fundamental difference between who we are and what we believe. There is a difference between turning away from a community an organisation that looks to intrude on the rights of its members and the conflicting views of scientists and astrologers, say. There is also a difference between espousing values that some regard as intolerant and looking to impose them upon other people; the difference between preaching against homosexuality and telling a gay couple in the street that they will "not inherit the kingdom of heaven" (implying they would burn in hell).

Every bit as worrying as their continuing attack on gay rights is their support for Nadine Dorries's recent campaigns for a reduction in the abortion time limit and a change to the rules on abortion counselling. It is sad that the first previously all-male Oxford college to have a female rector, back in 1993, should provide an unchallenged platform for those who oppose a woman's right to choose.

I know that Exeter College is not homophobic. Yet the intransigence of the current college leadership would suggest they'd rather upset their past and current members than a paying client. A community that upholds the rights of women and gay people isn't just something you can assert into existence, or something you can be sure will exist tomorrow. It requires each of us who support it to stand up and say so, to reaffirm those values through what we do, rather than merely by what we say.