Woody Allen is a man of habits. For over half a century, the stand-up comedian, writer and filmmaker has been true to his nerd glasses and his fashion-resistant wardrobe of khaki trousers. He has stayed loyal to his rigorous work habits, exotic phobias, piquant neuroses, distinctive Brooklyn accent and the clarinet he faithfully plays in a New Orleans-style jazz band on Monday nights in Manhattan. Because of such irregularities, Allen is instantly recognisable around the world for being… Woody Allen. He has made the existential fears of a neurotic New Yorker integral to his odd, enduring appeal.

That’s quite a feat for a small, skinny guy from Brooklyn born Allan Stewart Konigsberg – and all the more impressive to anyone watching any actor who isn’t Woody Allen attempt an impersonation when cast as the “Woody Allen character” in a Woody Allen movie. Remember Jason Biggs mimicking his iconic stutter in Anything Else? Will Ferrell's attempt in Melinda and Melinda? The agonies of Kenneth Branagh’s failed imitation in Celebrity? It turns out playing Woody Allen is no easy job. Mannerisms alone do not make the man.

Allen may be a world-class worrier, but it so happens that, at the age of 78, he has made 48 movies, and a 49th, Magic in the Moonlight, is due out this summer. (With good reason journalist Peter Biskind memorably referred to the prolific filmmaker as “The Joyce Carol Oates of the movies”.) In addition to his regular workload, Allen now plays an unlikely pimp named Murray – by day, Murray is a Manhattan dermatologist – in Fading Gigolo, a little romantic drama not of his making, but of writer-director-star John Turturro. And with the recent opening of Bullets Over Broadway in New York, Allen has become a theatre muse of sorts, having adapted the screenplay of his 1994 movie into the book of a brassy, singing-and-dancing romp.

I am not ignoring the complications of Allen’s links to scandal, dating back to 1992 and his involvement with (and later marriage to) Mia Farrow’s adopted daughter, Soon-Yi Previn, 34 years his junior. More recently, he has been in the news again with the resurfacing of 1993 allegations of molestation made by his adopted daughter, Dylan Farrow.

No one will ever claim that celebrating Woody Allen in all his complexities and disturbances is a simple proposition even for his biggest fans. Some, indeed, may decide they are too dismayed by the man they think Allen to be from these allegations to accommodate his art. But I am not one of those people. I am one of those people who finds it a hilarious, endearing and unlikely quirk of the universe that the Woody Allen persona – intellectual, obsessive, neurotic, fatalist and lifelong urbanite – is as firmly fixed in the comedy universe as the characters of Charlie Chaplin, Buster Keaton or Jacques Tati.

Take the funny and run

No matter how Allen grows or experiments, fails or succeeds, he is forever the amusing worrier, the reasonable pessimist. To the vast population of readers and moviegoers beyond the self-contained island of Manhattan, he is the distillation of a certain kind of New Yorker: a product of Jewish existential anxiety, a philosopher-king of comedy. The man himself may sometimes chafe at the reputation, exasperated by those fans who prefer ‘funny Woody Allen’ to the more serious, literary or Ingmar Bergman-obsessed cinephile he sometimes prefers to be.

But I expect Allen doesn’t really care what the fans think. And that, too, is part of his appeal. This man of habits – planning his next movie, writing his next humour piece for The New Yorker and practicing the next Dixieland rag on the band’s music set-list – continues to do his work and live in a way that makes him comfortable, or at least as comfortable as a deeply, honestly and intractably anxiety-ridden person can be. One of the psychoanalytic profession’s most famous self-avowed patients, he is also a useful example of the limits of psychoanalytic insight: all those sessions on the couch and still, as Allen said in an interview with an assist from Emily Dickinson, “the heart wants what it wants.”

Allen’s characters seek security and beauty. They enjoy the comforts of living quarters with good furniture and they celebrate the invention of taxis. They tend to have money rather than need money – Cate Blanchett’s poor Jasmine goes mad for the want of it. They have professions, appreciate the arts (especially literature, as there is no greater calling than being a writer in an Allen movie), and they know how to behave well in restaurants. Unsurprisingly, they’re great conversationalists.

Surprisingly, though, they are vulnerable. And that may be why the Woody Allen character endures after all. He gets the girl, or tells the joke, or enacts the comedy premise and still he knows that Death awaits him in the end. And what then? Our favourite pragmatist says, “I don’t believe in the afterlife, although I am bringing a change of underwear.”

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.