“Well, the comic novel, I feel, is perhaps the most difficult form a writer can attempt. I can think of only three or four successful ones – Cakes and Ale, Count Bruga, and Lucky Jim.” That’s SJ Perelman talking to the Paris Review in 1963. Ben Hecht’s Count Bruga has evaporated in the meantime; so has much of Perelman’s reputation – the New Yorker humorist is now best known for his influence on Woody Allen. Still, what can we add to his miserly list? It feels unfair to Stella Gibbons’ Cold Comfort Farm and Jerome K Jerome’s Three Men in a Boat. In 1963 Philip Roth hadn’t written Portnoy’s Complaint yet, and Joseph Heller’s Catch-22 was only two years old. But 50 years on, the literary comic novel can still feel like a rare and thankless achievement. When Howard Jacobson’s The Finkler Question won the Booker Prize in 2010, it was pronounced the only funny novel to win ever, or the only comic novel to win since Kingsley Amis’ The Old Devils in 1986.

Or … hang on, didn’t DBC Pierre win for Vernon God Little in 2003? Don’t Roddy Doyle’s or Salman Rushdie’s efforts qualify? Pull at the loose threads of this tight little definition of “successful comic novel”, stretch it beyond the confines of middle-aged male academics/historians/novelists having sex crises, and it begins to unravel completely. Perhaps the ascendant style of modern novel in English, what James Wood calls “hysterical realism”, is predominantly comic. Don DeLillo, Thomas Pynchon, Zadie Smith and David Foster Wallace might not have much else in common, but we know what Smith means when she jokes about a “700-page generational saga set on an incorporated McDonald’s island north of Tonga”.

What characters would populate that island? We can imagine, say, an architect who designs workers’ housing by day but a vast model of Christopher Wren’s London by night, or an inventor obsessed with inflatable pants for a sedentary civilisation, or a man who puts on a beard to become “the Complete Man” and pick up women. You might recognise all those people from Antic Hay, which Aldous Huxley wrote more than 90 years ago. Even at its most agitated, the new-school comic novel in fact has a long and embarrassingly worthy lineage.

Above all, it comes from Menippean satire, the kind Northrop Frye wrote up as dealing “less with people as such than with mental attitudes. Pedants, bigots, cranks, parvenus, virtuosi, enthusiasts, rapacious and incompetent professional men of all kinds, are handled in terms of their occupational approach to life as distinct from their social behaviour.” It’s not surprising this treatment of these “mental attitudes” is again pervasive, at a time when the professional is replacing the personal, even in private.

Aldous Huxley isn’t the only unlikely ancestor that can be claimed by the forces of comedy. How about George Orwell (Keep the Aspidistra Flying) or Vladimir Nabokov (Pnin)? How about CS Lewis (The Screwtape Letters, David Foster Wallace’s favourite book) or Gustave Flaubert (Bouvard and Pécuchet)? Even crime and horror writers moonlight as comics or daylight as black humorists. Isn’t Patricia Highsmith really a comedic writer? Isn’t Flannery O’Connor? What about Edgar Allan Poe’s satires, those now almost intelligible stories that HP Lovecraft called “blundering ventures in stilted and laboured pseudo-humour”? Both Kafka and his listeners laughed “quite immoderately” when he read The Trial aloud, a phenomenon explored by more than one hysterical realist.

Comedy flourishes even at the wellspring of the maudlin 19th-century novel, like those bacteria that somehow live in a volcano. Let’s take up with the bored and lovesick Dick in Under the Greenwood Tree:

Dick fidgeted about, yawned privately; counted the knots in the table, yawned publicly; counted the flies on the ceiling, yawned horribly; went into the kitchen and scullery, and so thoroughly studied the principle upon which the pump was constructed that he could have delivered a lecture on the subject.

Here in 1872 is Thomas Hardy, of all people, producing recognisably modern comedy, not just in its sympathies but in its construction. Timing, sympathy, repetition, exaggeration and analogue still combine smoothly enough to make a reader laugh today. Time changes complexions of humour quickly, but these jokes are now more familiar to us than most of the author’s melodramas.

The comedy of Under the Greenwood Tree is also modern in the sense identified by James Wood in his (slightly grudging) study of humorous fiction The Irresponsible Self: On laughter and the novel. It’s comedic in a “forgiving” register as distinct from the “corrective”, essentially religious, satire that preceded it. Dick is not a caricature or a choleric or a puppet in a morality play, but a fully realised and humane personality.

Wood also identifies the absence of the corporeally grotesque in the newer “forgiving” comedy. The corrupted, ugly and cruelled body feels inhumane and medieval, the stuff of the giant codpiece or the vaudeville drunk act, too crude and broad for now. He finds Don Quixote and Sancho Panza vomiting in each other’s faces distant, for example. But these vile bodies have a special significance for the old satirists like Jonathan Swift and François Rabelais. They’re a sign that, despite pretending otherwise, humans are still animals. That might make them uncorrectable, and perhaps even unforgivable – we are hypocritical partly because we are quite literally full of shit.

Wood has been too hasty, however, to announce the death of corporeal comedy. If the professionalisation of the 20th century brought back Menippean satire, new tortures (medical, technological, military) have now restored the grotesque body to the comic novel. When in 1710 Swift described a river (the Thames) flowing with “dung, guts, and blood, / drowned puppies, stinking sprats, all drenched in mud, / dead cats and turnip-tops come tumbling down the flood”, it was not only with disgust but also with wonder. It’s a passage that would fit comfortably into a modern novel’s cavalcade of filth, say, Louis-Ferdinand Céline’s Death on the Instalment Plan, with its Parisian fountains turned into craters “all groaning with abysses and drunks”. Even more prescient is Swift’s treatment in Gulliver’s Travels of the Struldbrugs, the immortal race of men whose longevity is a curse, not a blessing, because it comes without physical preservation. That’s the inevitable punchline, and now technology is providing the set-up.

“Real irony is an expression of suffering, and the greatest ironist was the one who suffered the most” was TS Eliot’s take on Swift. “Real irony as expression of suffering” would do as a motto for Steve Toltz’s writing as well. The Australian novelist has followed up his 2008 debut A Fraction of the Whole with Quicksand (Hamish Hamilton; $32.99). And like a Struldbrug, the central character of Quicksand seems to be suffering from chronic immortality without the able body needed to pull it off.

Sometime jailbird and full-time paraplegic Aldo Benjamin is also suffering from clinical frustration, and a surfeit of money-making ideas that don’t make any money. Stunted, sickly and paralysed, he’s also the most bodily afflicted central comic character since Ignatius J Reilly in John Kennedy Toole’s A Confederacy of Dunces. Ignatius, with his malfunctioning pyloric valve and mattress wanks, feels like a relative of Aldo, but Aldo could never be accused of laziness. His philosophy is also more modern, perhaps the most modern: self-help. His self is really beyond help, but so irrepressible it turns him into a holy fool.

Aldo shares Quicksand’s narration duties with his friend Liam, a failed novelist turned policeman, but both of them speak as avatars of Steve Toltz. As does everyone else. Sentences make fishtail turns into gags: “history isn’t a litany of peoples and civilisations, it’s a series of clinical trials”, “he is wearing his heart of darkness on his sleeve”, “as breadwinners we’re fucking losers”, “the creation of the universe was a motiveless crime – though not a victimless one, obviously”. These are the thoughts and words of four different characters. The last one is God.

Few novelists have made one-liners so central to their work – we have to stretch for names like Stanley Elkin or the Spanish surrealist Ramón Gómez de la Serna with his aphoristic greguerías (“Doors get angry with the wind”). Comedians are closer, and Woody Allen’s prose closest of all. But this isn’t a short story, and Toltz takes on much darker subjects than hat blocking. Over the course of hundreds of pages, these thousands of addled aphorisms start to feel closer to the gruelling anti-comedy of someone like Neil Hamburger. Like him, they plead, “But that’s my life!”

Aldo is a muse for everyone around him (Liam, his ex-wife, a former teacher who launches a one-subject art movement called “Aldoism”), but is also riddled with suffering. Murder, suicide, the death of a child (twice) and prison rape (more than twice) are touchy subjects to crack wise about. Toltz has acknowledged Céline’s influence, and sometimes reading him produces that same uncomfortable feeling of reading Céline: that the person producing the gallows humour is closer to the hangman than the condemned.

Céline sees humanity’s suffering like a writhing bacillus through a microscope. His version of immortality is essentially genetic, life as a sex tragedy:

We keep going, we fuel and refuel, we pass on our life to a biped of the next century, with frenzy, at any cost, as if it were the greatest of pleasures to perpetuate ourselves, as if, when all’s said and done, it would make us immortal. One way or another, kissing is as indispensable as scratching.

But Toltz has a fundamentally different use for human misery. The jokes have a more mystical quality, like that zen koan about the boy who achieves enlightenment after a monk cuts off his finger. Aldo is degraded for the same reason the righteous Job is cursed in the Bible, it’s just that getting covered in sores doesn’t cut it any more (although Aldo is also covered in sores). All this vileness isn’t because we’re doomed (although we are), but because nothing human can ever be irredeemably vile. “There’s nothing more depressing than a triumph of the human spirit,” says Aldo, but a degradation of the human spirit, and the human body attached to it, ends up being strangely uplifting. Like Aldo’s ramshackle schemes, we might finally pull this damn thing off.

“Well, the comic novel, I feel, is perhaps the most difficult form a writer can attempt. I can think of only three or four successful ones – Cakes and Ale, Count Bruga, and Lucky Jim.” That’s SJ Perelman talking to the Paris Review in 1963. Ben Hecht’s Count Bruga has evaporated in the meantime; so has much of Perelman’s reputation – the New Yorker humorist is now best known for his influence on Woody Allen. Still, what can we add to his miserly list? It feels unfair to Stella Gibbons’ Cold Comfort Farm and Jerome K Jerome’s Three Men in a Boat. In 1963 Philip Roth hadn’t written Portnoy’s Complaint yet, and Joseph Heller’s Catch-22 was only two years old. But 50 years on, the literary comic novel can still feel like a rare and thankless achievement. When Howard Jacobson’s The Finkler...