Daniel Radcliffe in the film adaptation of The Woman in Black, the 1983 book by Susan Hill.
Photograph: Publicity image from film company

It has been supplanted in recent years by vampires, witches and other monsters, but now the good old-fashioned ghost story is back with a bang, with everyone from debut novelists to established literary stars such as David Mitchell and Gillian Flynn hoping to raise the hairs on readers’ necks this Halloween.

Flynn’s The Grownup, the chilly tale of a fraudulent medium and a possibly haunted Victorian home, which won an Edgar award in the US when it was published in George RR Martin’s Rogues anthology last year, will be available as a standalone tale for the first time on 3 November. Meanwhile Mitchell’s haunted house tale, Slade House, which began life on Twitter, arrives next week, as does Little Sister Death, a previously undiscovered ghost story by cult US author William Gay.

They will join Catriona Ward’s debut novel Rawblood, Neil Spring’s The Watchers, Andrew Michael Hurley’s word-of-mouth hit The Loney and Kate Mosse’s The Taxidermist’s Daughter, all of which describe hauntings of one kind or another. The queen of the genre, Susan Hill, also has a new book out, as her spine-tingling stories are published together for the first time, while those who prefer to get their chills the old-fashioned way – read aloud – should head to the website of author Robert Lloyd Parry who, as “Nunkie”, will tour Britain this autumn performing classics of the genre.

Not since the heyday of MR James and WW Jacobs has the ghost story been so in vogue, but why? “We’re definitely seeing a resurgence after horror has held sway for a long time,” says Mosse. “The thing about horror is that it’s not that subtle; it’s a straightforward chase about the terrible thing that’s going to get you. With a ghost story the whole thing is, ‘Is it coming? Is everything in your head?’ Ghost fiction plays on those fears – which is why I describe The Taxidermist’s Daughter as not a whodunnit but a whydunnit.”

Spring’s acclaimed first novel The Ghost Hunters, which takes place in Borley Rectory in Essex – infamously once known as the most haunted house in Britain – has just been optioned for an ITV series starring Rafe Spall. He says: “The thing with a good ghost story is that it’s about ‘What if this happens? What if this is real?’ It’s in those moments that you start to imagine the worst.”

That’s certainly true of Gay’s terrifying lost novel Little Sister Death, which drew on the author’s obsession with America’s infamous Tennessee Bell witch case (also the influence for the indie horror, The Blair Witch Project) to tell a Shining-esque tale of thwarted ambition, a haunted house and a family on the verge of collapse.

“Little Sister Death works because of William’s sense of character and place,” says Angus Cargill, Gay’s editor at Faber.

“You know all these horrible things are going to happen. They’re looming from the get-go, so you end up shouting at the page, ‘don’t do that, don’t take your family there’.”

Cargill attributes the ghost story’s comeback to a growing acceptance of genre fiction: “We’re definitely seeing less of the sort of snobbery there used to be. I love it when writers cross genres, so it’s great to see someone like William, who was known as a literary southern gothic writer, move more towards horror, or Mitchell writing a ghost story.”

Hill, whose The Woman In Black, first published in 1983, continues to terrify as a book, film and a play, agrees. “There has been a sea-change in attitude towards genre writing,” she says. “Literary novelists started turning their hand to the crime novel. Writers such as Hilary Mantel changed our perception of the historical novel. The genres became not only acceptable but writing to which people aspired, whereas they would not have done 20 years before. I took a lot of flak about ‘dumbing down’ when I published Woman in Black and I’m very pleased it has changed – there’s never been any reason to be ashamed of writing genre fiction, but at last people realise that.”

It’s also arguable that as the nights draw in people draw a strange comfort from spooky tales of things going bump in the night. “Queen Victoria in the Doctor Who episode Tooth and Claw puts her finger on it: we think ghost stories chill us but actually they offer comfort,” says Mitchell, whose Slade House, a Halloween-set tale drawing on everything from The Turn of The Screw to Shirley Jackson’s The Haunting of Hill House, is one of the most anticipated novels of the autumn. “A universe with ghosts is a universe with an afterlife, which means the bed in the hospital ward or the view from the hospice window is not the end; we carry on, we’ll get to see the people we love once again. So while ghost stories scare the pants off you, they also defang mortality.”

Catriona Ward, author of the much-praised debut Rawblood, agrees. “I think the thing about ghost stories is that it’s a safe place to enact your darkest fears,” she says. “Reading a ghost story gives you the permission to go to places we actively and rightly avoid in normal life. A good ghost story asks the reader to examine the horror within – but it’s in a safe and contained way.”