Elizabeth Costello doesn't call itself a novel, but "Eight Lessons", with a postscript. Some of the lessons have been published before, two of them as the slim volume, The Lives of Animals.

In that book, an Australian novelist called Elizabeth Costello is visiting her son and his wife at an American university, where she gives two lectures. The first, "The Philosophers and the Animals", takes Kafka's "Report to an Academy", in which a highly educated ape tells a learned society the story of his ascent from beast to near-man, as the basis for a harrowing indictment of our treatment of animals, using the Nazi death-camps as a shocking parallel.

The horror of the camps was that "the killers refused to think themselves into the place of the victims"; by analogy, we persecute animals by dint of not imagining ourselves into their lives, even though "an animal - and we are all animals - is an embodied soul". This lecture, or lesson, is greeted in the small-town university with a mixture of scepticism, exasperation, cautious academic politeness, and outrage on the part of a Jewish poet who is appalled by Costello's analogy between murdered Jews and slaughtered cattle.

The second "lesson", "The Poets and the Animals", pursues the issue of animal rights through the ways in which writers have used animals as metaphors, as tools of argument, or in attempts to "embody" animal life in poetry, "to return the living, electric being to language". In the question-and-answer session that follows her talk, Costello is challenged (by voices standing in for many of Coetzee's readers) about the "western cultural arrogance" of the animal rights movement, its sentimental utopianism, and the philosophical shortcomings in claims for equality between animal and human understanding. She answers all these points firmly, but later, with her son, breaks down in distress at the disparity between her outrage over "the animal business" and other people's equanimity. "Everyone else comes to terms with it, why can't you?"

The Lives of Animals, which was published between Disgrace and Youth, was received with nervous respect. It was clearly linked to the scenes in the animal-welfare clinic in Disgrace and the appalling things done to the many dogs in that novel, in which the main character discovers he is going to have to start his life again, "with nothing", "like a dog". The lesson David Lurie learns passes judgment, by implication, on a society in which groups of human beings have been oppressed and treated "like animals".

While writing Disgrace, and his ambiguous, quasi-fictional, third-person memoirs, Boyhood and Youth, Coetzee - notoriously wary of confessions, public author-appearances, tribunals and interviewers - has been giving Elizabeth Costello's lessons as lectures, guarding his own voice inside her "beliefs" and arguments. (On the occasion I heard him, questions were not invited.) Now, as if doubling the point, he has reprinted The Lives of Animals inside Elizabeth Costello, alongside other episodes in which she has to perform, or state her case, or answer for her beliefs.

In the first section, she wins a big American prize and gives a lecture called "What is Realism?", which again makes use of Kafka's ape. On a cruise ship, lecturing alongside a self-promotingly "exotic" Nigerian writer, she talks about "The Future of the Novel". Visiting the sister she hasn't seen for years, who works in an African mission for the terminally ill, Costello argues with her over the rival belief-systems of classical humanism and Catholicism. She tells a startling story of a sexual consolation she once offered to an old dying man, pitting the Greek celebration of human beauty against the Gothic Christian insistence on human suffering.

At a conference in Amsterdam, speaking on "Witness, Silence and Censorship", she confronts a novelist who has written a vivid account of the execution of Hitler's would-be assassins. She argues that the evil of that history has been transferred to the writer and thence to the reader, and that there are some things that are better not written about. Her own secret example is a violation she suffered in her youth (another of the shockingly physical moments irrupting into this book of arguments), the memory of which she has buried silently inside her like "an egg of stone".

Responding to a book on the legend of Eros and Psyche, she asks, in sensual detail, whether we can imagine what it would be like to be made love to by a god. Finally, in a deliberately (but none the less dispiritingly) clichéd and Kafkaesque allegory of purgatory, a tribunal requires her to state her beliefs before she can go through to another world. The writer's job, she has always thought, is "believing whatever has to be believed in order to get the job done". Now she is beginning to think that the writer must have, if not "beliefs", then at least "fidelities".

In this fragmentary and inconclusive book, more like a collection of propositions about belief, writing and humanity than a novel, it is clear that animal rights is not the only issue. The creature in the zoo is also the novelist herself, and part of the book's driving force is an impatience with the way famous writers are required to perform like rock-stars, or to provide confessions or state their beliefs. Costello isn't even very good at public speaking, and keeps telling herself not to do it. There is distaste and scorn throughout, too, for interviewers and thesis-writers, goldfish "circling the dying whale, waiting their chance to dart in and take a quick mouthful".

But Costello's performances, however reluctant, have serious implications for the writer's role. The "world mirror" of realism is broken; the words on the page no longer have authoritative meanings; we can no longer say with confidence who we are; "the gods" seem no longer to be watching or listening - these are all Costello's contentions. Yet there is still some idea of divine inspiration around the figure of the writer - why else would people cluster to hear and see them? The "humanities", however secularised, still have something to do with "a quest for salvation". Good and evil still have some force.

But these notions must somehow be embodied in order to be expressed. Costello (and presumably Coetzee) opposes "embodiment" - fullness, the sensation of being - against mechanical, abstract, rational cogitation. If you are to imagine yourself as the other (a person imagining what an animal feels like, a male writer impersonating a woman, a human imagining intercourse with the divine), then "embodiment", "inwardness", is essential. The proposition that "all humankind is one" needs to be understood not as an abstract concept but as a felt reality. Every episode in the novel acts out this opposition between "embodiment" and "reason". Coetzee puts Costello in the almost untenable position of mounting a reasoned attack on reason.

Some recantation appears to be going on here (recantation is one of Coetzee's themes): the arguments about the indivisibility of the soul and body are a far cry from the prissy protagonist of Youth wanting to study at university in a "Department of Pure Thought". Coetzee has always divided his fictions between eloquent, enquiring rationalists and stone-dumb characters, their souls locked in their bodies like animals, such as the girl in Waiting for the Barbarians, Michael K, or Friday in Foe. Should the novelist, too, move towards the reticence or silence of such characters? Elizabeth Costello "no longer believes that storytelling is good in itself". The ultimate recantation would be to stop writing fiction altogether.

The postscript to Elizabeth Costello is a sequel to Hofmannsthal's letter of 1902 from "Lord Chandos" to Francis Bacon. (Characteristically austere and allusive, Coetzee points us towards the original letter without explaining it.) Hofmannsthal's imaginary 17th-century writer is protesting against scientific abstractions, and asking what place there is for poetry in a world of science. He argues that there is a need for a new language, closer to nature. As yet he can find no language for the revelations he gets from ordinary things and animals.

This anti-Enlightenment document, a founding text for modernism, inspires Coetzee to write another invented letter, from Chandos's wife to Bacon, speaking of the extreme states of mind of those who feel themselves "interpenetrated by fellow creatures by the thousands". There is no language available for this kind of "embodiment".

Perhaps the only solution, then, as Elizabeth Costello has suggested, is to stop writing fiction. Perhaps Coetzee is now going to give up story-telling forever, and write philosophical essays instead. Judging by this difficult and unforgiving book, that would be a diminishment. But he is impossible to predict.