Like many non-believers, I remember faith – and not just on Good Friday. I remember the pleasant mumble of liturgy, the talcum scent or drunk sweat of the person kneeling next to me, the weight that absolution took off and the chalky taste of redemption on my tongue. I remember the woozy intellectual delight of meditations on eternity – until the day when it became simpler and clearer to not believe in things because they were impossible, but just accept they were conceptual garble. It wasn't just my own struggles with sexuality and gender identity – it was the pain I saw religious dogmas inflicting everywhere. Faith was breaking my heart, but faith broke first.

Also there was a growing realisation of what believers do and have done in the name of faith, and my refusal of the comfort of religion if it made me complicit in those crimes. But it's people, not religions, who commit these crimes. Similarly, it's not religion that kills people, encourages them to despise their neighbours and makes small children go to bed whimpering for fear of hell – it is people with religion. As Swift said: "We have just enough religion to make us hate, but not enough to make us love one another." It is sometimes tempting to argue against religion as if it were a weapon placed in the hands of the potentially murderous.

But we don't judge truth by totting up a big ledger of atrocities and altruisms. In the end, the argument against religious belief has to depend on things other than the behaviour of believers. I've often pointed to the inconsistency between the core ethic of believers and their actual behaviour, and that is a valid point against the pretensions of religious leaders, but in the end it's not a conclusive argument against religion itself.

Pointing to that inconsistency does, though, bring us to one of the major problems with existing religions and their structures. The idea that texts written in a specific time and social context in human, often poetic, language with clear artistic intent can be the inerrant declaration of the mind of an eternal god depends on a leap of faith so vast that many of us cannot make it.

Without revelatory texts, most of the religious structures – the churches, the departments of theology, the wandering saints – lose much of their authority to tell us what to do. This does not mean that those texts cease to have much wisdom in them – or that individual men and women of exemplary virtue cease to be people to whom it is sensible to choose to listen.

Some of the best people I have ever met were priests and ministers, just as some of the others were charming hedonists with a gift for kindness and joy. What we call holiness is real even when it points to and derives from nothing real.

I know, as a poet and critic, that sometimes it feels as if something beyond me is helping move my fingers on the keyboard; as a novelist, that sometimes characters speak to me; as a mourner, that sometimes my memories of the dead seem alive in the same way; as someone who lives next to a park, that sometimes the smell of rain on sunlit grass is as close to paradise as I expect to come, except for the opening bars of Mozart's clarinet concerto. One of the reasons why I am reluctant to describe myself as an atheist is that we do get hints of a transcendent realm, just little reason to associate it with anything so definite as even the most nebulous idea of a god.

I am an agnostic partly because I don't think it is part of the human condition ever to have very much certainty about anything but moments of pleasure and of imminent and immanent death. I don't think we have a language, will ever have a language, that can describe transcendence in any useful way and am aware that that transcendence may be nothing more than the illusory aspiration of a decaying piece of meat on a random rock.

The thing is to be humble enough to be content with that while acting to other people as generously as if better things were true, and making art as if it might survive and do good in the world. Because what else are we going to do with the few short years of our life?