We are about to enter the time of superlatives. Although JK Rowling's publisher is understandably cagey about what we should expect today, when Harry Potter and the Half-Blood Prince is released, we know we are on course once more to experience a record-breaking publishing phenomenon. The book will have an initial print run of more than 10m in the United States - the biggest first printing of any book, ever. The previous volume, Harry Potter and the Order of the Phoenix, was the fastest-selling book in history, shifting more than 5m copies in a single day, and Rowling already earns more than any other woman in Britain, bringing in something close to £100m a year.

Although we can measure the size of the phenomenon by crunching numbers, that doesn't help us to understand why Harry Potter stands quite so large in our culture. And Harry is not alone - in a way his success only serves to echo and reinforce the equally unexpected breakthrough of The Lord of the Rings 50 years ago. That too was conceived for children but then came to define, in defiance of critical opinion, what many adult readers were looking for. Indeed, if you look at the results of popular surveys, such as the BBC's The Big Read in 2003, it's pretty clear that for most readers the 20th century was not best expressed in the current of hard realism that ran through it, from Ernest Hemingway to Saul Bellow. Nor was it the age that found its most perfect literary form in the experiments of modernists, or in the softer terms of magic realism. Most readers, as you can see by the way these surveys are always won comfortably by The Lord of the Rings, found what they were looking for in pure fantasy.

Alongside Rowling and Tolkien in the children's sections of bookshops are, of course, many other writers who have created similarly detailed magical universes - from Alan Garner and Susan Cooper a generation ago, to new voices such as Michelle Paver, Garth Nix and Catherine Fisher.

Yet there are only two others, Philip Pullman today and CS Lewis in Tolkien's day, who have managed to break into the imaginations of grown-ups as well as children with irrefutable force.

Their appeal is being underlined, right now, by their re-invention in the cinema. While Peter Jackson's vision of the Lord of the Rings faded out 18 months ago, audiences can expect the Chronicles of Narnia to begin at the end of this year, His Dark Materials to start next year, and Harry Potter to go on and on.

It remains hard for critics to explain the size of this phenomenon without patronising these audiences. Many observers have put the popularity of these books among adults down to mere infantilism. "We like to regress," AS Byatt wrote, explaining the appeal of Harry Potter and other children's literature to adults. "Some of Ms Rowling's adult readers are simply reverting to the child they were when they read the Billy Bunter books, or invested Enid Blyton's pasteboard kids with their own childish desires and hopes." "Culture, it seems," added Mark Lawson, "has become one giant tuck shop."

But it is still worth asking why so many readers should feel this abiding need to crawl into burrows and magical wardrobes. Even when writers do not topple all the way into fantasy lands, you often see novelists nowadays finding a way of sneaking a little magic into their narratives. This kind of magic is not like the unpredictable enchantments of the magic realist writers - it is always reassuring, muffling rather than sharpening the problems of everyday life. One of the attractions of The Da Vinci Code is the idea of an inclusive and gentle religion beyond the doctrines of the church. The Lovely Bones by Alice Sebold comforts readers against the horror of a vicious murder with a narrator speaking from the afterlife, while The Time Traveler's Wife by Audrey Niffenegger, in which the playful magic of a man who can flip through time is used to give a sense of enchantment to a standard boy-meets-girl romance, similarly allows readers to float into a more confortable world.

But none of these books is as reassuring as the grand fantasy worlds, whose writers answer a particular yearning. Above all, these books comfort readers against the harshness of a secular world. Although rationalism has captured minds in the west, it has not captured hearts on quite the same scale. It is tough to live in a world where there is no grand force that cares for your existence, no sense of purpose to the universe, no conscious presence being expressed in the cheerful sprout of flowers or the menacing rumble of thunder. Even if you were brought up on a very high-minded atheism, as I was, you can still find a totally rationalist world a rather dry and quiet place.

Philip Pullman, who describes himself as agnostic and sees "no evidence of a god whatsoever", gives voice to what it is like to feel that god-shaped hole when one of his characters, Mary Malone, describes how she once wanted to be a nun: "What I miss most is the sense of being connected to the whole of the universe. I used to feel I was connected to God like that, and because he was there, I was connected to the whole of his creation."

That isn't to say Pullman's books, or those of a Christian writer such as CS Lewis, give readers a direct substitute for monotheistic religion. Because, while you could say that these books fill a god-shaped hole, you could equally say that they fill a wizard-shaped hole, or a dragon-shaped hole. For in these books the supernatural is not confined to some distant force off in the sky - it is howling and mewling and roaring all around the characters, in talking eagles and stones and rings and wands and swords of power. Even Lewis, the most Christian writer of the lot, celebrates not a monotheistic but an animistic universe in Narnia. The rupture between pagan and Christian Europe was a harsh one, in which pagan believers were violently persecuted and sacred groves destroyed; but in the Narnia books we find a happy continuum between pagan and Christian beliefs, as when the Christ-like Aslan romps with "a youth, dressed only in a fawn-skin, with vine-leaves wreathed in his curly hair. His face would have been almost too pretty for a boy's if it had not looked so extremely wild." Lewis loved the Greek and Norse myths long before he fell for Christianity, and in his so-called Christian allegories he gives us everything that Christianity leaves out, including this sensual dance between the god of wine and Christ.

Gorgeous as all these playful encounters between angels and witches, shamans and talking animals clearly are, these fantasy writers do not just provide the carefree magical pleasures that earlier writers of fantasies for children - such as Lewis Carroll, E Nesbit, or JM Barrie - did. The most popular of these recent fantasy worlds are always driven by the heavy hand of prophecy. Frodo's mission to destroy the ring is enunciated in typically portentous rhetoric: "If I understand aright all that I have heard, I think that this task is appointed for you, Frodo; and that if you do not find a way, no one will." Rowling's comic touch tends to run against such heavyhanded soothsaying, but by the end of her last volume, Dumbledore had replayed the words of the prophecy for a reluctant Harry: "The one with the power to vanquish the Dark Lord approaches ... either must die at the hand of the other for neither can live while the other survives."

This sense of order expressed by such prophecies gives a religious tone to these fantasies. As William James said, "Were one to characterise the life of religion in the broadest and most general terms possible, one might say that it consists of the belief that there is an unseen order, and that our supreme good lies in harmoniously adjusting ourselves thereto." And this leads to a very particular view of heroism. In order to fulfil their destiny and re-assert the order of the universe, Frodo must give up the ring, while Pullman's Will and Lyra must give up their love. And we are gradually learning that Harry Potter's heroism is going in the same direction. "Your mother died to save you," Dumbledore tells him. "To have been loved so deeply, even though the person who loved us is gone, will give us some protection for ever."

With the repetition of this idea that true heroism is about pity and love, it is becoming clearer as we reach this, the penultimate instalment, that the books will only end with some supreme sacrifice from Harry. Will he have to break his wand? Return to the Muggle world? Or is there some other sacrifice being prepared that we cannot yet imagine?

It is hardly new to suggest that art is stepping into the places that God has vacated - Wagner said more than a century ago that "when religion becomes artificial it is for art to salvage the essence of religion". But it is crucial to remember that however far they go in filling spaces that religion once filled, these books do so in the spirit of entertainment, not doctrine. Even Tolkien, who was capable of taking his work extremely seriously, did once remind a correspondent: "The tale is after all in the ultimate analysis a tale, a piece of literature, intended to have literary effect."

Much as we may enjoy them, such tales do not stand as old myths stood. They demand nothing from us. We can create clubs to admire them, but not churches to channel their authority. Even those that lay claim to be religious allegories are tales to comfort, not commandments to live by, as easily sampled by atheists as by believers. You could say that it is infantile of readers to enjoy them, but you could also say that it is very grown-up of those readers to satisfy their infantile desires in children's books, rather than holy books.

· Harry Potter and the Half-Blood Prince is published today by Bloomsbury, price £16.99. To order a copy for £13.99 with free UK p&p call Guardian book service on 0870 836 0875.