To mark the 25th anniversary of the Booker prize in 1993, three former chairs of the judges - Malcolm Bradbury, David Holloway and WL Webb - gathered in literary confab and re-emerged to announce that Salman Rushdie's Midnight's Children had been duly chosen as the "Booker of Bookers" - the best novel out of all the winners. It was a verdict that, at the time, was as little disputed as it was mysteriously decided. In 2008, to mark the 40th year of the prize, the public has chosen, from a shortlist of six, the best from a now longer list of winners. And the voters have come to exactly the same decision as those critical mandarins 15 years ago. It's Midnight's Children again.

As one of the three judges who sweated over the choice of the shortlist on which the public was invited to vote, I found myself unfussed by the outcome of the poll. Midnight's Children is still a wonderful, buoyantly self-delighted novel, where east and west - Sheherazade and Tristram Shandy - meet as they never met before. But it hardly needs this poll to tell us of its qualities or its influence. The point of the "Booker of Bookers" vote was not this result, but the invitation to look back down the line of sometimes forgotten winners, to see which seem to have outlived the literary fashions that they might once have exemplified.

Rushdie's win in 1981 does seem a watershed, and not only because it was the first year in which the prize ceremony was televised. Midnight's Children (a novel by a then unknown writer) brought a new energy to fiction in English and it soon seemed to have announced a fertile decade for British fiction in particular. Literary fiction discovered a delight in formal experiment: the common reader was soon well acquainted with multiple narrators, and unreliable narrators, and even downright inarticulate narrators (The Remains of the Day, Last Orders, True History of the Kelly Gang). Yet look back at the winners before 1980 and the landscape seems different. Guiltily, I realised that I had read most of the winning novels since 1980, but very few from before. For me, deciding on a shortlist would first involve a route march through the Booker canon of the 1970s.

The prize was first given in 1969 to PH Newby's Something to Answer For, the only winning novel not currently in print. When he wasn't writing novels, Newby was running BBC Radio's Third Programme. You might imagine a dilettante author, whiling away some civilised evenings. Yet the novel in question is altogether more cunning than you would expect. Set during the 1956 Suez crisis, it tells the story of an Englishman, Townrow, who returns to Egypt to settle the affairs of a dead friend. He is beaten and robbed, and we follow his subsequent progress in imitation of his own "never-ending daze". It is like Graham Greene on hallucinogens, a narrative enactment of post-imperial and post-marital unease. It is often very funny. I listened recently to an old interview with Newby, recovered from the vaults for a "Best of Booker" feature on Radio 4. Newby spoke of his pleasure in writing fiction and, with apparent equanimity, of the certainty that his novels would be forgotten after his death. Those first Booker judges were right to think his fiction deserved better.

Some early winners had qualities that are no less admirable for seeming dated. In 1976, three judges, who included Mary Wilson, writer of poetry and wife of the prime minister, chose David Storey's Saville from an all-male shortlist. Resolutely not a crowd-pleaser, it is a long, sombre, minutely realist account of a working-class boy's childhood and adolescence in an unnamed northern mining village. Though distant in time - it begins in the late 1930s and ends with the protagonist in his early 20s - it is uninterested in history or the trappings of period. Everything is concentrated in family, locality and school. It is a world where grammar school is the only way up, but at the cost of subjugation to educational tyranny and eventual estrangement from the family that was once so "proud" of your achievements. Storey is a true bard of classroom humiliation. The needling, daily running joke that one supercilious grammar school master makes out of the boy's father's casual way of spelling his own surname is a classic of quotidian sadism.

The novel is a chronicle as much as a narrative, accumulating material rather than forcing it into shape. (Though there is a covert David Copperfield plot: Colin is befriended by a charming, confident, middle-class boy who eventually steals the girl he loves away from him.) "We don't teach poetry here. Just matter-of-fact English," Colin is told by the head of the school where he has his first teaching experience. Storey seems to be giving the warning himself. The style is deliberately neutral: exact, but uninflected by feeling. The point is apparently to mimic the blankness of the boy, who in order to survive must withdraw into himself. The whole story is told from Colin's point of view, yet a brief prelude narrates his parents' married life before he was born, and the death of a first son, Andrew, whom he never knew. It is characteristic of Storey's method that the parents' grief is dealt with in two short paragraphs, and the dead son is scarcely mentioned for nearly 500 pages - until a chance finding of some of his childhood drawings allows Colin, in his late teens, to glimpse his mother's abiding sadness.

Perhaps I would never have read Saville if I had not been one of the Best of Booker judges. It may be a novel that looks backward, in particular to early DH Lawrence, but it is a reminder of qualities that the contemporary novel has largely forgotten. Not all the early winners come back to life so vividly. The second winner of the Booker prize was Bernice Rubens's The Elected Member in 1970, and it hardly seems a forgotten classic. A grim comedy of Jewish family life, which shares its attention between a widowed rabbi and his clinically deluded son, it is more grim than comic. It is an honourable and uncalculating novel, but devoid of formal interest. John Berger's G, the 1972 winner, is beached by time for the opposite reason: it is more like a thesis about novel-writing than a living narrative. Chosen, rather surprisingly, in the year in which Cyril Connolly was chair of the judges, it has an ideological forcefulness that has not weathered well, while its "experimental" narrative method now seems ostentatious rather than purposeful.

If any of the early winners deserves to be pulled back into the light, it is surely JG Farrell's The Siege of Krishnapur, which won in 1973. Its subject matter is hardly unfashionable nowadays. Farrell was preoccupied with the demise of imperial fantasies; he also wrote novels about the 1916 rebellion in Ireland and the fall of Singapore to the Japanese in 1942. Though he could hardly have foreseen the coming trend, his sardonic version of colonial self-delusion would happily find a place on BA courses in post-colonial literature. His interest in Victorian mores also seems to foreshadow the reconstructions of Victorian culture achieved in AS Byatt's Possession or Michel Faber's The Crimson Petal and the White. Farrell's brilliantly imagined novel is about the shredding of Victorian self-belief. Science itself has been found wanting. The desperate fighting is paralleled by an intense conflict between the garrison's two doctors, who disagree radically about the causes and proper treatment of every potentially fatal affliction. Their dazed potential patients take to carrying round cards specifying which doctor they should be taken to if struck down. "And you might even find the names of both doctors scratched out and substituted more than once, such was the atmosphere of indecision which gripped the enclave."

Farrell's narrative method is both sophisticated and old-fashioned. His omniscient narrator moves confidently between his characters. Like some latterday Thackeray or Fielding, he watches their follies. It is a method suitable to the cruel black comedy of the book. Sometimes the reader feels a little like the local Indian population, who take up comfortable positions on an adjacent hill to watch the unfolding drama, with all the relish of "gentlemen returning to their seats in the theatre after the interval". They comment with amusement, but no particular malice, on the indignities being suffered by some of those officials who have previously lorded it over them. Their delicious-looking picnics are more torture to those on the ramparts who have telescopes.

The other novel we chose from the 1970s for the shortlist, Nadine Gordimer's The Conservationist, has been neglected in an odder way, given that its author is a Nobel prizewinner. Even those who know and admire her short stories, or novels such as Burger's Daughter or July's People, are unlikely to have read the work that won her the Booker. Narrated largely through the consciousness of its calculatedly unsympathetic protagonist, it is a story too committed to its narrative technique to make any concessions to the reader who is not willing to concentrate. It is a brilliantly unsettling narrative, but by design rebarbative. Even the wonderfully exact natural description is somehow tainted, as it is seen through the eyes of a man who finds it easier to love landscapes than people.

Gordimer shared the prize in 1974 with a novelist who provides further proof that winning the Booker was not always a shortcut to fame and fortune. Stanley Middleton is 90 next year, and has published more than 40 novels. Middleton's Holiday is about a middle-aged man, recently separated from his wife, who returns to a seaside town he knew in his childhood. He thinks about his past, talks to strangers and keeps bumping into his in-laws, who, as if from malign purpose, are holidaying in the same shabby resort. Though designed to be contemporary, it feels as though it belongs to the 1950s. It is a world in which a woman might decline an invitation to the cinema because "I'm not dressed for it", and even calls its protagonist by his surname.

Fisher is provincial and middle-class, a lecturer in education (Middleton himself was for many years a schoolteacher). It is as if the protagonist of Lucky Jim had aged and failed to escape his fate. Indeed, Middleton's women are reminiscent of Kingsley Amis's: they look utterly repressed in their twin-sets and hair-dos, but they say the most waspish and unsentimental things, and seem privately to scorn the respectability they act out. Holiday captures rather brutally the private condescension of its main character to the fleshy holidaymakers who surround him, "Lancashire girls in their innocent bikinis" and their flabby beaux, the prematurely ageing wives, the smirking husbands with their dirty talk. He renders perfectly the "inquisitorial civility" of his landlady - "People without culture or subtlety". The book is as observant and morose as its protagonist. "The whole world he saw in a dazed disproportion." Gruesomely exact about the small talk, the diet and the prejudices of these characters, it is a novel rooted in the shabbiness of ordinary England. The protagonist contemplates adultery, but returns to his intelligent, unaffectionate wife. "Risk did not appeal." On the last page, though he is thinking about sex, he and she are having a cup of tea with her parents. "Nobody knew anything, Fisher decided."

Defiantly provincial, it is a novel that would now look peculiar on a Man Booker shortlist. Yet it is still there for curious readers partly because of the prize. Everyone notices the sparkle that the Man Booker prize gives (and plenty complain about it). Its vindication is perhaps, however, that it enables good books to, as Laurence Sterne put it, "swim down the gutter of Time". Since the announcement of the Best of Booker shortlist, the bestselling novel of the six at Foyle's bookshop in London has been Farrell's The Siege of Krishnapur. If the publicity has encouraged readers to discover this neglected modern classic, the exercise has been worthwhile.