Novelists have increasingly faced the challenge of trying to compete with a culture that is a step ahead of them.

Just before Christmas five years ago, I spent an afternoon in the company of the novelist Zadie Smith and the literary critic James Wood (then of the New Republic, now of the New Yorker). I'd been asked by another magazine to oversee a conversation between Smith and Wood on "the future of the novel and the function of criticism". The idea was that they would continue a public colloquy that had begun in 2002, after Wood wrote a withering and pitiless review of Smith's second novel, The Autograph Man.

Wood had found the novel to be little more than a tissue of "smirking epigraphs" fatally in thrall to the example of American writers such as Dave Eggers and the late David Foster Wallace, of whom he mostly disapproved. Eggers and Wallace were practitioners of something he called "hysterical realism" and their novels burned brightly with an unnourishing sub-Dickensian dazzle. These were smart guys writing big, ambitious books that tried to do nothing less than pin down and analyse an entire culture. And while they were busy practising cultural theory by fictional means, the novel's traditional quarries of character and consciousness got left behind. (In fact, Wallace's case was much more complicated than Wood tended to make it seem, and he actually shared many of the critic's misgivings about the moral and aesthetic legacy of postmodernism, of which hysterical realism could be said to be a variant or tributary.)

By Wood's account, the "hysterical realist" novel - the novel of "information" which can't decide if its job is simply to reflect the cognitive superabundance of life under late capitalism or, as they say in seminar rooms from Berkeley to Bloomsbury, to critique it - had, by the early 2000s, become one of, if not the, dominant mode in British and American fiction. And The Autograph Man, whose protagonist is a half-Chinese, half-Jewish dealer in the signatures of dead celebrities, faithfully mimics its most distinctive narrative tics - Smith is always pointing out, for instance, "that her characters, on the brink of a momentous access of feeling, are undermined by their sense that they are not ­being original, that TV has preceded them". An observation, Wood suggested, that it "may be time to retire".

That same afternoon, Smith told me that she had taken Wood's review "to heart" - and, indeed, you could see signs of this in several of the critical essays she wrote during this period for the Guardian and the New York Review of Books. These were much more likely to cite E M Forster than David Foster Wallace. There were further indications of this shift in her third novel, On Beauty, published in 2005, which was altogether more decorous than either The Autograph Man or her debut, White Teeth, and which she described as a "homage" to Forster (the book borrows its structure explicitly from Howards End). Now she and Wood were in agreement: "the culture [was] doing strange things to novels". Smith confessed that she found the "idea that you can't write a book without it being put through the processing machine of culture really quite frightening".

So, this was the sound of a generation dis­covering for itself a predicament described by Philip Roth in a celebrated essay published more than 40 years earlier, where he'd written that "the actuality is continually outdoing our talents, and the culture tosses up figures almost daily that are the envy of any novelist". For both Smith and Wood, none of their contemporaries had come closer to properly articulating these anxieties for the early 21st century than the American writer Jonathan Franzen. His sprawling third novel, The Corrections, published in 2001, was in part the product of several years' worth of agonised reflections on the place of fiction in a culture that was increasingly and aggressively indifferent to it.

In 1996, Franzen had written an essay for Harper's magazine, "Perchance to Dream", the arguments of which continued to reverberate in a certain stratum of the literary intelligentsia on both sides of the Atlantic in the early years of the new century. The "Harper's essay", as it became known, was both a 20-page howl of despair at the decline of the big, ambitious "social novel" that connects the personal with the societal and a kind of renunciation, in which Franzen declared that in fact the very idea of writing fiction which sought to "engage" with the culture should be given up, now that there are technologies - film and television, principally - that "do a much better job of social instruction".

That culture is so grossly productive of novelties that to engage with it, Franzen concluded, was to "risk writing fiction that [made] the same point over and over: technological consumerism is an infernal machine, technological consumerism is an infernal machine . . ." If the improving ­mission of the novel of social instruction was at an end, what was left was the solace of "sentences of such authenticity that refuge can be taken in them".

The Harper's essay wasn't merely programmatic, however. Much of its considerable interest lay in its account of the genesis of The Corrections (indeed, few reviewers were able to resist using the piece as a lens through which to view the novel). Franzen recalled being "para­lysed" with what would become The Corrections. "I was torturing the story, stretching it to accommodate ever more of those things-in-the-world that impinge on the enterprise of fiction writing." He found that he couldn't help bulking up his "story" until it became "bloated with issues". Liberation, he implied, arrived once he realised he wasn't obliged to dramatise the "important issues of the day".

But The Corrections is not wholly successful in extricating itself from the horns of this di­lemma. Franzen found that it was much harder to give up the impulse to anatomise the culture than the Harper's essay had implied. And his failure to do so was symptomatic. "There are certain places in that novel," Smith said, "and I know I've written them myself in my novels, where the engagement is not with the novel as an organic form, with the characters, with the story, but is a matter of coming straight up to face the writer. It's not the novel I want to write and it's maybe not the novel a lot of people want to read any more. If the novel is going to stake its claim to being a separate part of the culture, then it needs not to be direct commentary."

It is tempting, in retrospect, to read those remarks of Smith's as setting out a programme - one that comes to fruition in "Two Directions for the Novel", an essay included in her most recent book, Changing My Mind. Here she describes Tom McCarthy's intricate, allegorical novel Remainder as an attempt to answer the question of how fiction might stake its claim to being a "separate part of the culture". But it is not clear from this how far she, and we, have travelled, because the dichotomy Smith presents - between the realist novel and the self-enclosed allegory - is pretty much the same one that Franzen was trying to think his way out of at the start of the decade.