Misconceptions surround the word ‘occult’. A string of images we might immediately associate with it, plucked from lurid horror cinema: buxom covens of scantily-clad women forgoing Christ and loving it; men in cowled robes mumbling in an arcane tongue, faces lit by flickering candlelight in a secret place of worship; pentagrams marked out in chalk set up to summon a horned and hairy beast who speaks in a rasping voice.

You can find all of these these things in speculative fiction should you want it. And why wouldn’t you? Demons, naked witches and cowled men lend themselves so well to the pulpier end of fiction and cinema. They’re exciting, the imagery compelling. But on a deeper level there’s a bit more to this excitement and intrigue than mere titillation gleaned from the transgressive. That illicit aspect of the occult is where the real appeal lies – the sense there could be more beyond our own limited understanding of the world, that perhaps there’s more to discover about reality than what we’ve been taught. The notion of a hidden, heretical, culture stretching back millennia that we could tap into if we only knew how – it’s a seductive idea.

What’s even more appealing is that there is some truth to this. All the word occult (from the Latin occultus: ‘clandestine, hidden, secret’) means is ‘knowledge of the hidden’. In common parlance, people tend to mean ‘involvement with the paranormal’. But interpretation of occultism and its concepts can be found in the belief structures of philosophies and religions such as Gnosticism, Hermeticism, Theosophy, Wicca, Thelema (the ‘Do what thou wilt’ school of thought developed by Aleister Crowley), Chaos magic (according to The Book of English Magic, the form most appropriate for punks), all the way up to modern paganism.

The occult is often falsely conflated with Satanism, where perhaps we draw our initial mental images from. Though we can see some similarities there – satan just means ‘adversary’ in Hebrew. It’s good to question the status quo, let’s say. Questioning, that search for knowledge, is what the occult should represent.

All of the aforementioned traditions do clearly exist – you can go on Amazon right now and load up on books about all of these subjects and many more. These are ways of looking at the world that in some cases dostretch back through the centuries; some are much more recent concoctions.

The notion of secret knowledge waiting to be tapped in to can be very useful. It helps engage with difficult, troubling or heretical ideas, by demonstrating a) someone else has also had the idea before so you’re not totally alone and b) different ways of thinking are possible. It helps us excavate and unearth trends in our society that may be invisible to the naked eye, useful seams of culture that can help deal with the present and understand the past (and maybe even forge a different future). It is good to remind oneself that there has always been a tradition of people exploring and questioning what these things we call consciousness, and society, actually are, without merely falling into line. Some occultists were deluded, egoists, or chancers. Others were not.

In cultural terms we can talk of an ‘occult heritage’. This is an idea extensively explored by the writer Iain Sinclair, whose occult mapping of London exposes a troubled and complicated relationship between the past and present rather than a reassuring one. This is a heritage comprised of many diverse strands, including the visionary wanderings of William Blake, the psychogeography of the Situationists, the radical antiquarianism of 1960s counterculture, the mystical folklore of Old England (increasingly in vogue in the form of folk horror) and the literary traditions of nineteenth-century Gothic romanticism. Sinclair spearheaded something that gave rise to a glut of sub-par place writing, but also laid the groundwork for such other explorers of occult heritage such as the brilliant Laura Oldfield Ford (go and buy Savage Messiah right now).

Occult heritage relates to fiction itself. We may have a gateway writer who suddenly unlocks the door to a whole host of narratives and ideas that we only knew we craved until we found it. Many of the complex (or just plain weird) ideas found within the various schools of occult thought have found themselves expressed in fiction. This can be no surprise; fiction at it’s best should be an exploration of occult heritage, and freed from the shackles of strict realism, speculative fiction can really take the ideas and run with them.

Many writers of note have dabbled in the occult. Known or alleged members of The Hermetic Order of the Golden Dawn include Algernon Blackwood, Bram Stoker, W.B. Yeats, Sax Rohmer, Arthur Conan Doyle, Gustav Meyrink, Aleister Crowley and importantly, Arthur Machen.

It is possible to see such iconic works as Mary Shelley's Frankenstein in this tradition, dealing as it does with mankind's search for heretical knowledge and it's terrible consequences. Occult ideas later fed into the Modernist movement, with writers like Mary Butts – herself a student of Crowley – exploring themes of nature-spirituality in novels such as Ashe of Rings, where a young woman's attempts to preserve an ancient shrine acts as a redemptive act to counter the horrors of the First World War. Writing at a similar time-period to Butts was Dion Fortune, who wrote extensively about the occult in works such as The Demon Lover, The Goat-Foot God and The Sea Priestess (which became popular within the Wicca movement).

In much literature dealing with occult themes, we find a recurring idea of ‘peering behind the curtain’, breaking down the doors of perception to see reality as it truly is. When this happens, in fiction at least, things tend to end badly. Even writers with an avowed positive interest in the occult tend to present this search for knowledge as perilous – the implication that hidden knowledge can come with a horrific price. On a more practical note, it makes for a better story.

A foundational text of this kind of is Arthur Machen’s The Great God Pan (1894). Rather than the candles-and-incense dark ritual of cliche, this features Victorian brain-surgery in the wilds of Wales, with a doctor hoping to open the mind of humanity to fully experience the spiritual – an experience he calls ‘seeing the great god Pan’. The victim/patient, a young woman, awakens from the experiment in awe, terrified, but quickly becomes ‘a hopeless idiot’. The story doesn’t get any cheerier after that: people are left hysterical and psychologically damaged by the things they discover. It’s also quite debauched by late Victorian standards, which led the Manchester Guardian to declare it ‘an incoherent nightmare of sex’. The Great God Pan is not Machen’s best work – for that I would really recommend the stories ’N’, ‘The White People’, or the short novel The Hill of Dreams – but it introduces a crucial idea: the occult knowledge we crave is there, but we lack the tools to fully comprehend it.

Machen was both a huge influence on H.P. Lovecraft and on the British psychogeographers on the latter part of the 20th century; importantly on Iain Sinclair, who wrote a short book, Our Unknown Everywhere: Arthur Machen as Presence, on how Machen’s notion of the occult bleeds into our understanding of place. ‘He who adventures in London has a foretaste of infinity’, Machen memorably wrote. Occult knowledge bleeds through the cracks in mundane reality. Weird fiction and psychogeography start to blend.

‘Seeing the great god pan’ becomes a shorthand for accessing this knowledge; awe morphs into terror, itself a common theme found in horror and weird fiction (and would go on to lend itself to the American brand of cosmic horror popularised by Lovecraft).

In Peter Straub’s A Dark Matter a small group of friends have shared a terrifying secret for years. Something happened in a field near the University of Wisconsin in Madison in 1966 that left one boy dead and mutilated and another missing. The secret has had terrible effects on each of them. All fairly standard horror-novel fare, but A Dark Matter is an interesting American take on a well-trodden theme; it can be read as a criticism of the ‘radical antiquarianism of 60s counterculture’ that leads the characters into the disastrous ritual in the first place. Their search and desire for knowledge was genuine, and the desire is exploited, badly executed by guru-shysters, with terrible consequences.

Britain’s greatest chain-smoking ex-punk magician, John Constantine, lives with the ramifications of what is known as the ‘Newcastle incident’ in the Hellblazer comics. In a club called the Casanova run by a ‘sex and drugs magician’ called Alex Logue (a clear Crowley analogue), an occult summoning goes disastrously wrong and a young girl is sent to hell. He ends up in an asylum and is haunted by the events for the rest of his life. Constantine himself can be seen as an 80s punked-up reboot of the grand tradition of occult detectives in literature: Algernon Blackwood’s Dr. John Silence, William Hope Hodgson’s Thomas Carnacki, Aleister Crowley’s Simon Iff, and many more.

The first series (the good one) of True Detective is suffused with the sense that things started going wrong for the two protagonists at a specific moment in time: the moment they witnessed the disturbing aftermath of occult sacrifice in a burned out cane-field in Louisiana. It is no coincidence that the series draws heavily on the traditions of weird and occult fiction – Thomas Ligotti, Robert W. Chambers, Ambrose Bierce and Lovecraft – for it’s thematic content, and the series successfully suffuses the action with a very strong sense of place.

One of the greatest treatments of the theme of hidden knowledge, exploring ideas found in the various branches of gnostic thought, is M. John Harrison’s The Course of the Heart. Appropriately the novel is an expansion of a short story titled ‘The Great God Pan’. Back in 2002, China Miéville went as far as to nominate the book as his favourite weird fiction novel, stating ‘this unforgiving story of gnosticism and loneliness worries and worries at me like a dog’. Again the spectre of the ritual-gone-wrong haunts Harrison's characters, defining and ruining their lives.

Many other writers have addressed occult themes in their work, and continue to do so. A list that barely scratches the surface would include Hilary Mantel's Beyond Black, Anna Kavan's Mercury, Peter Ackroyd's The House of Doctor Dee, Lolly Willowes by Sylvia Townsend Warner, Jonathan Strange & Mr Norrell by Susanna Clarke (an epic novel dealing directly with English magick) and Nicola Barker's Darkmans.

In the world of speculative fiction, the revelations promised by the occult schools of thought are real and there to be discovered. There are realities beyond our own, other ways of seeing, perhaps even other ways of being if we could only understand this new and terrifying knowledge. Many of the writers who address these topics in a serious way view it though the lens of horror/dark fantasy, but with a deep and all-pervasive feeling of regret, the sense that perhaps an opportunity was missed. The knowledge comes at at terrible price. After all, what do you do once you know there really is more to life?

Maybe the greatest fear is that we know there is more to the world, to consciousness, than what we perceive in our day-to-day lives. Perhaps the greater tragedy is that we also know that our limited perception of the world prevents us from ever truly comprehending; we can see the great god Pan, but we’ll never understand him. That is the source of existential horror that powers these works – the very thing preventing enlightenment is ourselves.