What is it about himself that J. M. Coetzee feels compelled to confront? His new novel, Youth, purports to be a work of fiction. It certainly has the compulsion and internal logic of fiction - he creates a believable world and allows autonomous creations to move freely in it. In fact, Youth is less a work of imagination than a stylised memoir, in which Coetzee revisits the humiliation and struggle of his early years as a restless student in London. As such, it's a study in failure which, when read against what we know of his mature achievement - twice winner of the Booker Prize, celebrated essayist and successful career academic - feels like a peculiarly tortuous exercise in intellectual introspection, and not much else.

Youth is written, as was his earlier, less artful memoir, Boyhood: Scenes From Provincial Life (1997), in an evasive third-person, so that the young Coetzee once more becomes an actor in his own drama. The story he tells is resonant if overfamiliar: that of a young colonial's journey, in the early 1960s, from the margins of Empire to the metropolitan centre, a journey undertaken by Orwell, V. S. Naipaul, Dan Jacobson and many others before him. The colonial shares with Coetzee the name 'John', a background in mathematics and linguistics, and a desire to escape from 'an undistinguished, rural family, bad schooling, the Afrikaans language'. He wants to be a writer (what else?) and he also wants to suffer for his art. He takes a cramped room on the Archway Road in north London, and prepares for his emotions to be 'transfigured and turned into poetry'. In the event, he does little more than wither into aimlessness, a self-styled 'wanderer' who, like Orwell's Gordon Comstock and Forster's Leonard Bast before him, discovers how hard it can be for a provincial with literary ambitions in the metropolis.

Coetzee never once leaves the young man's side, pursuing him in thought more than in action. Which means that the reader is locked up inside John's head for the entire book - and what a claustrophobic space it is. Consciousness is here frequently rendered as a stream of rhetorical questions: Why is life so cruel? Why must I suffer? Why am I so unhappy? Oh, all right, I exaggerate - but not by much, because John is a model of romantic gloom and willed turmoil, the kind of obsessive self-watcher who never misses the opportunity to steal a glance at himself as he passes a shop window.

Much happens to John, in Cape Town and then in London: he has several lovers, one of whom becomes pregnant; he works as a computer programmer at IBM, where he meets a regiment of blandly untroubled bureaucrats; and he writes an awful lot of verse, scarcely any of which we ever see. In fact, very little about him is shown. Speech is mostly reported, and everything is told at the same cool, measured distance, relayed through the thick filter of his thoughts.

In his wonderful comic novel Life is Elsewhere, Milan Kundera ruthlessly satirised the aspirations of the poet manqué, in the form of a young man called Jaromil who, like Coetzee's student, is sustained by immortal longings but for whom truth and beauty are always tantalisingly elsewhere. Jaromil is a clown who thinks he's a great poet, but we love him for this, even as we laugh at him, because he is one of life's holy innocents and his consequent artistic strivings have true pathos.

But it's hard similarly to be moved by John, even as he drifts impecuniously from one disappointment to another.

At best, he's a monster of self-absorption; to him, other people are nothing but the means to an end of poetic self-fulfilment, hence the impossibility of conversation. Later in the novel, he takes to wandering the winter streets of London. Of course, he is a listless guide, making no attempt to animate the city through which he moves. Instead, he looks mostly at his feet.

Coetzee is a writer of deep intelligence, drawn to symbol and allegory. He has perfected a kind of prison literature: his lonely characters operate in societies without any recognisable moral centre, often afflicted by a nameless menace, guilty of no sin except that of being alive. So life is a prison sentence; birth is a crime. His previous novel, Disgrace, a parable of the social dislocation of the new South Africa, deservedly won the Booker. In that novel - and again in Youth - his prose is stripped of all superfluous ornamentation. The more he writes, it seems, the more concise and wintry becomes his tone and style - the inevitable slow glide towards silence of the Beckettian that he is.

Youth has none of the urgency and contemporary relevance of Disgrace, in which every sentence carried an authentic charge of intrigue. It's a book of great sorrow and regret. Coetzee is very hard on himself in creating a character - if indeed it is himself - so wretched and dislikeable. Perhaps that's his point: that he had to remind himself how much he once suffered in order to remake himself, to become the person he wanted to be, the writer he is, free from the taint of family and of the past.

Youth borrows its title from Conrad's great novella of romantic wonder that ends with a young Western sailor's vision of the Orient, of the 'East', as something mysterious and dreamlike: 'Only for a moment; a moment of strength, of romance, of glamour - of youth.' But there all similarities end. There's nothing transcendent in this resolutely earth-bound book.

Coetzee once said of Robinson Crusoe, a novel he rewrote in his own Foe (1986), that the idea of a man being marooned alone on an island is perhaps the 'only story'. He has once again written about shipwreck and ontological isolation in Youth, a work of unremitting despair.

One hopes that through returning to the primal scene of his early unhappiness in London, Coetzee has at last achieved a form of catharsis. From here, he can move on.