One London lunchtime many years ago, the late poet and editor Ian Hamilton was sitting at his usual table in the Soho pub, the Pillars of Hercules. This was where much of the business of Hamilton's literary journal, The New Review, was conducted. It was sickeningly early - not to be at work, but to be at drink. A haggard poet entered, and Hamilton offered him a chair and a glass of something. "Oh no, I just can't keep drinking," said the weakened poet. "I must give it up. It's doing terrible things to me. It's not even giving me any pleasure any longer." But Hamilton, narrowing his eyes, responded to this feebleness in a tone of weary stoicism, and said in a quiet, hard voice: "Well, none of us likes it."

So why is this funny? There is comedy in the inversion of the usual idea that drinking is fun and voluntary. In Hamilton's reply, drinking has become unpleasant but unavoidable, one of life's burdens. The cynical stress on "likes" gives the reply a sense of weary déjà-vu: it sounds as if Hamilton is so obviously citing a truism that it is barely worth saying it aloud. It is always funny when singular novelty is passed off as general wisdom, especially when it is almost the opposite of the truth.

The joke simultaneously plays on the inversion of drinking as good fun while playing off the grim truth of alcoholism, which of course is indeed a state in which drinkers may not much like alcohol but cannot release themselves from it. Against those two worlds - the world of ordinary, pleasant, voluntary drinking, and involuntary alcoholic enslavement - Hamilton's reply proposes a stoical tragi-comic world, populated by cheerful but stubborn drinkers doing their not very pleasant duty. The joke seems to me to open, in a moment, a picture at once funny and sad.

I like Hamilton's joke, too, because it arises gently from its context, out of a natural exchange, and in so doing offers us access, albeit fleeting, to the character of the man who made it. It is unflashy; it is not an obviously great or crushing mot. It represents the opposite of those forced moments when someone says "Do you want to hear a joke?", at which point most of us freeze, alarmed that we won't get the punchline, and nervously aware that we are now inhabiting a "comic moment".

In literature, there are novels that have the feel of Hamilton's quip - novels in which a mild tragi-comedy arises naturally out of context and situation, novels which are softly witty but which may never elicit an actual laugh; and there are also "comic novels", which correspond to the man who comes up to you and says, "have you heard the one about...?", novels obviously very busy at the business of being comic. Tristram Shandy, for instance, is in multifarious ways a marvellous book, but it is written in a tone of such constant high-pitched zaniness, such deliberate "liveliness", that one finds oneself screaming at it to calm down a bit. The "hysterical realism" of such contemporary writers as Pynchon and Rushdie is the modern version of Sterne's perpetual excitements and digressions.

There is a kind of tragi-comic stoicism which might best be called the comedy of forgiveness. This comedy can be distinguished - if a little roughly - from the comedy of correction. The latter is a way of laughing at; the former a way of laughing with. Or put it like this: at one extreme of comedy, there is Momus, the ancient personification of fault-finding, reprehension and correction, who appears in Hesiod and Lucian. And at the other extreme of comedy, in the area now called tragi-comedy, is "the irresponsible self". Not necessarily funny himself, Momus roots out absurdity and foolishness. He sees through you; he truffles for folly. Momus, you might say, is the patron saint of satirists.

The comedy of correction, which might include the Aristophanes of The Clouds and The Wasps, Alberti's allegorical comic tale Momus (written in the 1440s), Erasmus, Rabelais, some elements of Cervantes (though Don Quixote amiably contains many comic modes), Swift, Molière and Flaubert's Bouvard and Pécuchet, is satirical in impulse, frequently violent and farcical, keen to see through the weaknesses of mankind, and essentially pre-novelistic. Bouvard and Pécuchet, though written in the heyday of the novel, is the exception that proves the rule: it is actually much less like a novel than a treatise, written to prove how repetitively stupid we all are, by a writer who was obsessed with the folly and vices of bourgeois idiocy, and who complained in a letter that he found the characters in Madame Bovary "deeply repulsive".

Flaubert was really a religious writer who transferred his devotion to aesthetics. He had the old religious impulse to scourge and check his characters. In fact, the comedy of correction might be called religious comedy, since the ambition of total transparency, the desire to put a window in the human heart, strikes one as essentially religious. That transparency received its memorably terrifying formulation when Jesus - who weeps but who never laughs in the Gospels - admonished us that to imagine adultery is to commit it; we are known, through and through. The few references to Yahweh's laughter in the Old Testament are all examples of laughing at, not laughing with: in Psalm 2, we are told that God will "laugh at" the heathen and "have them in derision".

Most comedy before the rise of the novel is Aristotelian in nature. Aristotle argues in the Poetics that comedy arises from a perceived defect or ugliness that should not be so painful that we feel compassion, since compassion is the enemy of laughter. The Renaissance theorist of laughter, Laurent Joubert, in his Traité du ris (1579), expanded on Aristotle by arguing that ugliness and the lack of strong emotion were crucial to comedy. In order for comedy to work we must in the end feel a pleasure at the lack of our compassion. Thus, when a man is stripped of his clothes, the sight of his genitals is shameful and ugly, and is yet "unworthy of pity", so we laugh.

Secular or modern tragi-comedy, the comedy of forgiveness, could not be more different: it is almost the inversion of the Aristotelian idea. And it is almost entirely the creation of the modern novel - with the huge exception of Shakespeare, whose role in the creation of the modern novelistic art of combined pathos and comedy cannot be overstated.

If religious comedy is punishment for those who deserve it, secular comedy is forgiveness for those who don't. If correction implies transparency, then forgiveness - at least, secular forgiveness - implies deliberate opacity, the drawing of a veil, a willingness to let obscurity go free.

In Pride and Prejudice, Elizabeth learns that laughing at is cruel (it is what her irresponsible father is always doing, not to mention the rebarbative Bingley sisters). Instead, she will laugh with Darcy, which entails being laughed at by him. For Jane Austen, getting married - or rather, falling in love - is the conversion of laughing-at into laughing-with, since each lover, balancing the other, laughs equally at the other, and creates a new form of laughter, a kind of equal laughter. Laughing with Darcy, and loving him, involves Elizabeth in realising that she was wrong to judge him as harshly as she did, that she may take many years to get to know him properly. It involves her in the deeply secular concession that, as Philip Roth has it in American Pastoral, "getting people right is not what living is all about. It's getting them wrong that is living."

Austen is an example of a writer who bridges the transition between the comedy of correction and a newer comedy of forgiveness. In her work, broadly put, there are the minor characters, who seem to belong to the theatre, and who are theatrically mocked and "corrected" by the author in her old 18th-century satiric mode; and there are the great heroines of the books, the sole possessors of interior consciousness, heroic because they exercise their consciousness, who seem to belong to the newer world of the novel and not of the theatre, and who are not mocked but gradually comprehended and finally forgiven.

If you think about it, the novelistic idea that we have bottomless interiors which may only be partially disclosed to us must create a new form of comedy, based on the management of our incomprehension rather than on the victory of our complete knowledge: Svevo's Zeno is the cardinal example; Henry Green's butler, Raunce, is a softer representative. This kind of comedy is also found in Chekhov, Verga, Hrabal, James, Bellow, JF Powers, Nabokov, Tolstoy, Naipaul and Gogol.

One way of looking at the "irresponsible self" in comedy is to examine the difference between reliably unreliable narration and unreliably unreliable narration. Generally, we know when an unreliable narrator is being unreliable because the author is alerting us, reliably, to that narrator's unreliability. (Swift works likes this.) But the modern novel brings us that wonderful character, the unreliably unreliable narrator, manipulated so brilliantly by Svevo and Hrabal and Nabokov.

This category of storytelling can only work if we think initially that we know more about a character than he knows himself - thus we are lulled at first into the comedy of correction - only to be taught that we finally know less about that character than we thought we knew at the outset: thus we are lulled into the comedy of forgiveness. Reliably unreliable narrators are often funny, playful, witty; but they don't move us as deeply as unreliably unreliable narrators.

English comedy has great strengths but has been rather deficient in sympathy; it is the European and American and Irish novel that has pioneered tragi-comedy - an historical oddity, given Shakespeare's mastery of the tragi-comic in plays such as Hamlet and King Lear. Evelyn Waugh, alas, still represents the great image of English comedy in the 20th century, rather than his subtler and gentler contemporary, Henry Green. The famous passage in Decline and Fall , about the Welsh brass band, is still praised as a characteristic piece of English literary fun:

"Ten men of revolting appearance were approaching from the drive. They were low of brow, crafty of eye and crooked of limb. They advanced huddled together with the loping tread of wolves, peering about them furtively as they came, as though in constant terror of ambush; they slavered at their mouths, which hung loosely over their receding chins, while each clutched under his ape-like arm a burden of curious and unaccountable shape."

Imagine what Chekhov or Svevo or James or Bellow would think of this crudity. First, that adjective, "revolting", is clumsy because it so brutally reveals the author's cards, and sets up the passage in such undergraduate terms. And then there is the way Waugh lumpily insists on the men's animality, telling us first that they were "wolves", and then reminding us that they "slavered" and were "ape-like". And finally there is the foundation of the comedy itself - the idea that humour is skin-deep, for one thing; and that the Welsh are inherently funny, for another.

Perhaps the right word for tragi-comedy is "humour". Freud distinguished humour from comedy and the joke. He was particularly interested in "broken humour", which he defined as "the humour that smiles through tears". He argued that this kind of humorous pleasure arises from the prevention of an emotion. A sympathy that the reader has prepared is blocked by a comic occurrence, and transferred on to a matter of secondary importance. What Freud leaves dangling, and what one would want to add to his analysis, is that just because one's sympathy is blocked and transferred - Freud sounds rather Aristotelian here - it does not cease to be sympathy. On the contrary, sympathy is intensified by its blockage. Verga's stories act like this, as do Chekhov's.

This kind of comedy seems to me the creation of modern fiction (and by that I mean late-19th- and 20th-century fiction) because it exchanges typology for the examination of the individual, and the religious dream of complete or stable knowledge for the uncertainty of incomplete knowledge. Compare Joubert and Pirandello. Joubert, in his Traité du ris, argues that "empty promises of visual pleasure cause laughter". If it is announced that we are to see a beautiful young maiden, and are introduced instead to a withered old hag, we laugh, especially if she is "dirty, smelly, drooling, toothless... and more deformed than ugliness itself".

Pirandello, without explicitly referring to Joubert, seems to reply to him in his essay, "L'Umorismo". Suppose, says Pirandello, we see an old woman, heavily made up and inappropriately dressed, in fashions more befitting a woman half her age. She at first seems comic, on the traditional Joubertian principle that incongruity and self-delusion are funny. But if we begin to try to enter the woman's head, if we try to merge with her - while acknowledging that we cannot utterly know her motives - our laughter turns to pity. We wonder if the woman is not herself distressed by her appearance and all the yearning to be young it represents. This mingled amusement and pity Pirandello calls humour. Pirandello saw humour as a modern invention, an enlargement of the old comic tradition.

Modern tragi-comic fiction does not offer a guarantee of reliable knowledge; yet paradoxically it continues to believe that the attempt to know a character is worthwhile, even if the attempt is beautifully frustrated. Henry James, reviewing Middlemarch, argued, a bit unfairly, that George Eliot hemmed in her characters with too much authorial essayism - that she wanted to know them too well, in effect. James's own characters are free to contradict themelves without being corrected by the author, are free to make mistakes without fearing authorial judgment; they are, "like people in real life, to be inferred by the reader", as Coleridge described Shakespeare's characters.

In such fiction, the reader is not to be overly helped by the author, as Waugh so blatantly tells us what to think of the Welsh brass band. We must find out for ourselves how much we know of a character, and we may find that what we know is that we do not know enough (as we do not "know" why Isabel Archer returns to Gilbert Osmond).

The philosopher Henri Bergson said that one definition of comedy was watching people dancing to music through a window, without our being able to hear their music. Bergson's idea of comedy belongs somewhat to the "corrective" school, to the world of Molière, of satire, and of mechanical farce. In the Bergsonian vision, the watcher has an advantage over the dancers. He comprehends them, sees how foolish they look and knows why they are dancing. He comprehends them because he is deprived of their music. His deprivation is his strength. But what if his deprivation was his weakness? What if that watcher did not know that the dancers were dancing to music? What if he had no idea why they were dancing? What if he felt no advantage over them, but felt, with mingled laughter and pity, that he was watching some awful dance of death, in which he too was obscurely implicated?

This alternative picture comes closest to my notion of the modern novel's unreliability or irresponsibility, a state in which the reader may not always know why a character does something or may not know how to "read" a passage, and feels that in order to find these things out, he must try to merge with the characters in their uncertainty.

Such a person is no longer the cruelly laughing Yahweh, and no longer the correctively laughing theatregoer, but simply the modern reader, gloriously thrown into the same mixed and free dimension as the novel's characters.

· Adapted from the introduction to The Irresponsible Self: Humour and the Novel by James Wood