We hear a lot about the death of the author, but William Golding is an author who was almost still-born. The man who wrote Lord of the Flies found that no one wanted to publish it. In 1953, his manuscript spent seven months being sniffily perused by publishers, who all promptly returned it. The Curtis Brown agency even declined to represent the would-be author, a dispirited schoolmaster who had written the book during classes and given his pupils, in lieu of an education, the humdrum task of totting up the number of words per page. A dead end seemed to have been reached when the Faber reader, picking through pages that were now yellow and grubby from handling, contemptuously rejected the submission as "absurd & uninteresting … rubbish & dull".

Then Charles Monteith, a former lawyer hired as an editor by Faber only a month before, retrieved the book from the bin and persuaded his colleagues to buy it for the piffling sum of £60. As a set text for schools, Lord of the Flies went on to sell millions of copies, introducing adolescents worldwide to the idea of original sin and the knowledge of their own barbarity.

My childhood reading life began, so far as I can recall, with RM Ballantyne's naively imperialist story The Coral Island; my innocence came to an end when I opened Lord of the Flies, which warps Ballantyne's tale into an allegory about the wickedness of our species and its rightful ejection from the happy garden. The novel, as the critic Lionel Trilling said, marked a mutation in culture: God may have died, but the Devil was flourishing, especially in English public schools.

Yet the man who wrote Lord of the Flies spent the rest of his life regretting that he had done so. Golding considered the book "boring and crude", its language "O-level stuff". Its classic status struck him as "a joke" and he disparaged his income from it as "Monopoly money". And what right had it to overshadow later, better books, like his evolutionary saga, The Inheritors, his medieval fable, The Spire, or his solipsistic tragedy, Pincher Martin?

Towards the end of his life, he refused to reread the manuscript (much revised, on Monteith's orders, before publication): he feared he'd be so dismayed he might do himself a mischief. Golding whispered the truth about these protests in his journal. He abominated Lord of the Flies, he confided, because "basically I despise myself and am anxious not to be discovered, uncovered, detected, rumbled". Discovery, uncovery, detection and rumbling are the appointed tasks of the biographer, about which John Carey, in this authorised life of a man he "admired and respected", evidently feels uncomfortable.

Golding called himself a monster. His imagination lodged a horde of demons, buzzing like flies inside his haunted head, and his dreams rehearsed his guilt in scenarios that read like sketches for incidents in his novels, which they often were. After dark, his mother became a murderous maniac, hurling knives, shards of shattered mirror or metal pots of scalding tea at little William; a girlfriend he had cast off returned as a stiffened corpse, which he watched himself trying to bury in the garden. At his finest, Golding paid traumatised tribute to the pain of other creatures, like the hooked octopus he once saw impaled by the "vulnerable, vulvar sensitive flesh" of its pink, screaming mouth, or a rabbit he shot in Cornwall, which stared at him before it fell with "a combination of astonishment and outrage".

But pity didn't prohibit him from firing the shot. He understood the Nazis, he said, because he was "of that sort by nature". His sexual assault on a 15-year-old girl has been titillatingly leaked to publicise Carey's biography. More generally, his son-in-law testifies that Golding specialised in belittling others – if that is, he recognised them at all. As Carey notes, he chronically misspelt names because he couldn't be bothered with people and their pesky claim to exist.

Carey documents Golding's ogre-like antics, but is reluctant to speculate about their origins. "I do not know," he says, "why he thought he was a monster" and he concludes this long, loyal, conscientious book by admitting there may be a primal scene, a hidden obscenity, that still eludes him – "something I have not discovered". Should a biographer, I wonder, accept defeat with such good grace? Carey prefers to deal with the masks the monster wore in public. At times, Golding impersonated a twinkling Cornish pixie; behind the helms of his boats, he pretended to be Captain Hornblower or perhaps, when the role came closer to caricature, Cap'n Birdseye.

His worst rampages occurred when he was drunk. Once, staying at a friend's house in London, Golding awoke in panic and dismembered a Bob Dylan puppet because he thought it was Satan. Carey nervously makes light of the episode, referring to it as a '"diabolic encounter". Religion and rationality, myth and science, fight it out in Golding's books as they did in his brain; it may be that Carey is too sane or puritanical to comprehend the creative madness of his subject.

He is tactful about Golding's relations with his children, both of whom suffered psychological upsets, or with his put-upon wife, who seems to have had her revenge by interrogating him at public lectures; at a gig in Lisbon, her voice from the darkened auditorium demanded to be told why there weren't more women in his books. Carey, a battle-scarred class warrior whose books include The Intellectuals and the Masses, sympathises with the young Golding's embarrassments at Oxford, where interviewers wrote him off as "not quite a gentleman". He's strangely reticent, however, about the old man's desperation to gain admission to the establishment. Golding pestered well-placed acquaintances to nominate him for a knighthood, which he called "Kultivating my K", and when it was finally doled out he changed the name on his passport with indecent alacrity and began to take pleasure in the sycophancy of hotel managers and head waiters.

The self-contempt that Golding defined as the clue to his character pays dividends for Carey the textual scholar, who here unearths a series of early drafts for published novels or extracts from projects unjustifiably abandoned – a "magnificent" but unfinished work of Homeric science fiction, a memoir that was self-censored because too raw, a film script about a traffic jam that rehearses the Apocalypse, a first version of The Inheritors that "cries out to be published as a novel in its own right" and a segment excised from Darkness Visible that is also "a masterpiece crying out for publication".

I suspect the cry Carey hears is that of unborn infants begging him to deliver them into the light and I hope he will do so. As a biographer, he may not have uncovered Golding's darkest, deepest secrets, but at least his detective work has grubbed up these intriguing, revealing relics. The man who wrote Lord of the Flies indeed wrote better things, some of which the rest of us should be given the chance to read.