If there is debate about who is the greatest literary detective, there can be no dispute as to who is the most recognisable. An IMDb search for adaptations of GK Chesterton's Father Brown stories yields one 1950s film, two television series (one of which has just started on BBC1), a clutch of American Masterpiece Mystery shows, and an American TV movie. A similar hunt for Arthur Conan Doyle's Sherlock Holmes turns up an immediate 200 hits, with many more following: there's Young Sherlock, Robert Downey Jr, Benedict Cumberbatch, Jeremy Brett, Basil Rathbone and Billy Wilder, as well as the Baker Street Irregulars, all the spoofs, the debunkings, the modernisations, and so on and on. Even Tom and Jerry have met Holmes.

The reasons for Holmes's on-screen triumph over the diminutive cleric are not hard to find. Holmes has benefited both from the attractiveness of his milieu (that foggy, gas-lit, late Victorian London) and the fact that, as a myth, he transcends it. Holmes can fight Nazis, have an internet addiction, or live in Los Angeles, and remain recognisably Holmes. Father Brown is both more linked to a specific early-20th-century Britain and hampered by the very variety of the tales in which he appeared; there is no one typical Father Brown setting, no single atmosphere to which they belong. The new BBC series has plumped for one world, embedding the stories in a Midsomer-style 1950s English village, that mystical land of Victoria sponge and murder guaranteed to bring succour to the heart of the afternoon TV viewer. And to my heart, too, I should confess, although nostalgia is one element absent from Chesterton's stories themselves, even if consolation is their central purpose.

Besides, where Sherlock Holmes is the Bohemian Übermensch who dominates every tale, and (other than the Boswell-like Watson) often the only character any reader of the stories remembers, Father Brown is comically unobtrusive. Indeed it seems that Chesterton was at first occupied with making a joke in which he wrote detective stories where the aim was to puzzle out who the real detective is. Often Brown makes his first appearance in an aside or as one item in a list, edging sideways into the story. His most conspicuous feature is his inconspicuousness. Neither film nor TV is a medium built for the celebration of humility. Its great actors like, naturally enough, to be at the centre, for film favours the show-off, the celebrity. To base a show on a man who quietly observes others is a challenge the new BBC series perhaps rightly does not attempt to solve.

The 1974 ATV series with Kenneth More as Father Brown was happier with this aesthetic of displacement, and the hero's entrance was sometimes delayed for more than a third of a programme's length. It's good stuff, but still More could not help but exude a saloon-bar chumminess, an unshowy, no-nonsense bonhomie. As a result he can appear uncomfortable just looking on, resisting his default setting of appearing charmingly pleased with himself. Yet, in Robert Hamer's 1954 film, Alec Guinness was perfect casting, a master of self-effacement playing another, the kind of actor who could dominate a film while somehow simultaneously vanishing from it.

Where Holmes is dashing, hawk-like, and a boxer and swordsman as much as an amateur chemist and logician, Brown is dowdy, baby-faced, plump, a parish priest and an "Essex turnip". Holmes is not just a great detective, he has a great profile. His methods may be analytical, but his persona is pure theatre, the dressing-gowned matinée idol of rational detection. It's perhaps not surprising that stories about a short, nondescript priest should lack sex-appeal. And on-screen, sex sells: Holmes has become sexier and sexier; for the current BBC series of Father Brown, Mark Williams is a good character actor, but one perhaps unlikely to set hearts a-flutter. (Other than The Fast Show, I suppose he's best known for playing Ron Weasley's dad.) If Sherlock Holmes leads directly to Dr Who, then it's true that the only Dr Who we could also easily imagine playing Father Brown is Sylvester McCoy, that series' whimsical nemesis.

Holmes disperses mystery, turning the disgust and mayhem of murder into something ultimately comforting; all is in order, so long as Holmes is there to disentangle the strangeness. He is a Tory detective, inscribing with bullet holes the initials of Victoria Regina on the wall of his Baker Street flat. The more radical Father Brown shifts our sense of mystery elsewhere; a crime is solved, but the solution comes in such a way as to unsettle our sense of what we know, or think we know. That the unsettling points to something larger than our first thoughts is these tales' great lesson. The world is weirder than we previously believed, but also much better, richer and stranger. Chesterton remarked that the "detective story differs from every story in this: that the reader is only happy if he feels a fool".

Chesterton's tales, in their different way, offer something of the solace found in Holmes, and in all detective stories, that notion central to the form that the mind fits the world: "'What we all dread most,'" said the priest in a low voice, 'is a maze with no centre.'" John Gross has suggested that more than any other Edwardian writer Chesterton was caught up in escaping the moribund pessimism of 1890s decadence. It is a favourite paradox for critics writing on Chesterton that despite his contempt for the fin de siècle, he was imbued with the spirit of the decade that he professed to hate. In his most purple passages, he writes like Oscar Wilde on steroids; there's sometimes a Paterian luxuriance in the prose, a specifying gorgeousness. Despite these traces, hemmed in by the decadent jadedness, Chesterton plotted ways to escape. He embraced a view of the world based on an essential excitement that it should exist at all. Chesterton declared that "the aim of life is appreciation", his version of DH Lawrence's dictum that there is no point living with a bad grace.

It may seem anomalous that the literary form that gave the best opportunity for displaying this enraptured gratitude was the detective story, filled as it is with murder, guilt and corpses. Yet Chesterton was given over to paradox, and his imagination was ideally fitted to a form that turned on the revelation of small surprises. Of course every surprise must also seem a solution. It must persuade us that it fits the facts, and so unmask the mystery that preceded it as having a hitherto undetected coherence. In this, his religion aided him, for it meant in his eyes that the truth was what hardly anyone else believed. Surprise was as intrinsic to Chesterton's thought as it was to his method. Each unexpected twist reveals a world that is both endlessly strange yet undeniably apt.

The early Father Brown stories begin with a picture; the early Holmes short stories begin with a debate about the nature of the stories themselves, a consideration of methodology. Every great detective was meant to have his distinguishing quirks, but far more importantly, he was to have his "method". At times, Brown's "method" can look like a matter of inspiration, not deduction; as such, it would leave the reader looking on admiringly, but seldom able to participate in the story's puzzle-like aspects.

For Father Brown does have a method and, late on in the stories, in "The Secret of Father Brown", he reveals it. In an earlier story, Brown makes a distinction between approval and sympathy, a distinction that is the soul of his approach. Empathy, a leap into another's point of view, is his tactic, just as it is also film's ultimate goal. Brown puts himself imaginatively in the minds of the killers, identifying himself with their desires, their way of seeing, their wants, their limitations, their falling for crime. But in doing so, he discovers simultaneously his own capacity for murder; distinct from them, he shares their humanity and, in potential at least, their sin. Like Edgar Allan Poe's Dupin, Father Brown doubles himself with the killer, but does so in a spirit of humility and charity. The criminal comes close; indeed Brown places himself inside him.

In presenting his approach to crime in this way, Chesterton was giving not so much a method for detection as his creed as a writer. Chesterton was emphatically an artist, and in the Father Brown stories he sought to investigate the foundations of his art. The writer becomes a kind of actor, playing the part of another, and like Brown, he too was "a sort of understudy; always in a state of being ready to act the assassin". Chesterton favoured many writers – Chaucer, Shakespeare, Browning, Dickens – and there is something of a dark version of their abundance in these stories, in the figure of a man who imagines himself inside 100 murderers. And at odd moments, Brown himself can seem an uncanny figure, a goblin, a blunderer making the blackest jokes, a horned and Satanic figure caught in a glance in a distant mirror.

Chesterton's conscious aim in writing the Father Brown stories was to subvert the misconception that priests must be unworldly innocents ignorant of the world. Instead it turns out that Brown (like his "real-life model", Father John O'Connor) is shockingly well informed about the most surprising of sins. Brown's life as a priest brings into the tales enigmas that a detective might not be expected to solve: not just whodunit, but the ultimate mystery of life and death.

Chesterton inherited a form naturally given to morbidity and infused it with a reckless joy. He also took over a form preoccupied with facts, and turned it rather in the direction of "atmospheres". In a late story, it is said that Father Brown dislikes the telephone, as he prefers "to watch people's faces, and feel social atmospheres". In The Club of Queer Trades, another of his detectives, Basil Grant, suspects facts, but finds a solution through his awareness of the "atmosphere" of a letter: "how facts obscure the truth … Facts point in all directions, like the thousand twigs on a tree. It's only the life of a tree that has unity and goes up – only the green blood that springs, like a fountain, at the stars." After reading the Father Brown stories, it's their atmospheres that linger, long after the details of the cases themselves are forgotten. It's the seaside town in winter, the walled garden, the crazily stacked apartment blocks, the occultist's tower, the cursed book, the aesthete's hidden island lost among the fens that remain. As a polemicist, Chesterton was perhaps illogical, but here his belief in a different standard of "proof" – the feel of a thing – comes to marvellous fruition. They are undoubtedly the greatest short detective stories yet written.