Barry Sonnenfeld (in headphones) on the set of Men in Black.
Photograph: Melinda Sue Gordon/Colombia Tristar

How could it happen? How could the movie industry’s most reliable maker of blockbusters be a short, bearded 47-year-old who hates Los Angeles; a quick-to-tears nice Jewish boy who suffered the public humiliation of his mother paging him at a New York Knicks ball game (“Mr Sonnenfeld, please call your mom urgently”); and a renowned kidder whose ball-busting methods have levered him into the upper branches of Hollywood’s power tree? Mind you, short Jewish guys have been a fixture in Hollywood for decades, but few have shed so many tears in the cause of cinematic perfection, or experienced so much misery in the midst of massive success.

Barry Sonnenfeld is all this and more: perched high on the cash mountain that is Wild Wild West, with his 15-year film-making career having taken him from photographing the Coen brothers’ debut, Blood Simple, to the Hollywood hit Men in Black. Sonnenfeld is a curious blend of garment-district smarts and self-deprecating enthusiasm. Although he looks like sometime mentor Steven Spielberg, and wears a gaudy orange tie decorated with a string of lips, he also bubbles over with cheerful, guileless good humour as he spins one story after another.

The time when he faced down a roomful of MGM executives over a $250,000 cut in a movie budget; the time when he was reduced to tears as he and the Coens drove for hours around upstate New York looking for rainstorms to film the opening of Blood Simple; the time when he bumped into Martin Scorsese and didn’t recognise him in a spooky darkened hallway.

And then there’s Will Smith. Sonnenfeld loves his star - so much so that you can’t tell when the kidding stops and the adoration begins. “Will is truly unlike anyone you’ve ever met,” he says, “because what you see on screen is really Will - that love of life, that energy, that refusal to fail. He’s very moral and weirdly Zen-like in trusting where the arrow will go. He’s a spiritually perfect person, and you want to be with him all the time. He’s like a religious figure to me. For some reason we get along really well: he trusts me and I trust him. There’s a little father-son relationship there, but Will’s the father and I’m the son.” Sonnenfeld owes a lot to Smith; and while the rapper-turned-actor surely knows what he owes the man who crafted two of his star-making hits, it’s Sonnenfeld who’s got the most to be thankful for.

It’s because of Smith, for example, that Sonnenfeld will get to direct the much-slavered-over biopic of Muhammad Ali. Having cast Smith in Men in Black despite executive producer Spielberg’s initial reservations (he wanted blue-eyed-boy Chris O’Donnell), Sonnenfeld knows how to use his special relationship with the planet’s hottest star. “When Wild Wild West came along, I said to the producers, ‘I will read the script and, if I like it, I will ask Will to be Jim West; and if Will says no, then there’s no movie.’ And I knew if Will said no, I wouldn’t want to direct it - I’d tell them, I wanted Will, he’s not available, sorry.”

“Here’s the other weird thing about being a director,” he continues, eyes lighting up. “You never know what tiny thing it is that’s going to attract you to a project. You think, why exactly did Stanley Kubrick do Eyes Wide Shut? He could do anything. In the case of Men in Black, it was one line of dialogue. The original script wasn’t very good, but there was a line of dialogue for the Will Smith character - who wasn’t Will yet, it hadn’t been written for a black guy - when they’re standing in front of the surveillance screen. The Tommy Lee Jones character says, ‘I guess you’re pretty amazed by all this’; Will says: ‘Not really, because I had a third-grade teacher who I thought was from Mars’; and Tommy’s character says: ‘Mrs Edelsten. Venus actually - well, one of the moons.’ That gem was surrounded by 100 pages of some of the worst writing ever. So many directors had passed on it; but I saw something in there that said, there’s a movie for me.”

It’s principally the nursing of this backwater, passed-over project into a $545m hit that made Sonnenfeld’s reputation. Satirising The X-Files, Ghostbusters, Reservoir Dogs and Independence Day - themselves no slouches in the tongue-in-cheek department - Men in Black scored heavily for injecting a big-budget effects movie with an acid-laced wit typical of Sonnenfeld’s earlier films, Get Shorty and The Addams Family. “I used to say to Steven: ‘I gotta warn you, whatever anyone is saying, we are making a tiny buddy film. Don’t think of it as an event movie.’ Get Shorty, Men in Black and The Addams Family are autobiographical: they all are ultimately about my own sensibilities; I just can’t do anything in a straight ahead way. It drives my wife crazy.”

Nebbish and mummy’s boy that he is, Sonnenfeld’s sturdy marriage - to Susan Ringo, aka Sweetie - is a key plank of a personality that’s unconventional in its conventionality. (“Sweetie and I read scripts together, usually in bed.”) Sonnenfeld actually came into film-making via the graduate film-school route: after a political science degree at New York University, he drifted into NYU’s not-yet-legendary film programme in the late-70s. “Even though I was the only child of very protective Jewish parents, they didn’t necessarily want me to be a lawyer or anything. My father always said to me: ‘Just figure out what would make you happy.’”

At NYU, Sonnenfeld found his vocation as a cinematographer, where he became friends with another lighting cameraman, Bill Pope (The Matrix, Clueless, Army Of Darkness). And it was at a student party that Sonnenfeld made another valuable connection: a gangling Minnesotan called Joel Coen, who was helping another film-making tyro, Sam Raimi, edit a movie called The Evil Dead. Coen eventually hired Sonnenfeld to shoot his and his brother’s own debut, a gore-splattered film noir, Blood Simple.

“To tell you the truth, we sort of learned together. Because we’d never worked on a feature before, we went down to Austin and we had nothing to lose. If we messed it up, no one would even know we had ever existed. We worked out for ourselves that we wanted the camera to be a participant in the movie. I taught Joel and Ethan about the wide-angle lens: when you move a camera with a wide lens, it has a tremendous amount of energy.”

Before Blood Simple, Sonnenfeld had stooped to shooting porn films to earn a dollar; after it, he was a fully-fledged industry pro with an exceptional entry on his showreel. Sonnenfeld crossed over to Hollywood with the Coens, when 20th Century Fox took on their second film, Raising Arizona - another eyecatching showcase with crackerjack camerawork, as was his third and last Coen movie, the gangster pastiche Miller’s Crossing. But he branched out too and learned about the demands of mainstream studio film-making: he shot Danny DeVito’s Throw Momma from the Train, the Tom Hanks comedy Big, and a pair of movies for Rob Reiner, When Harry met Sally and Misery. Sonnenfeld also studied at the feet of the master when he took over the final week’s photography of Martin Scorsese’s GoodFellas; original cinematographer Michael Ballhaus had left to start another film.

“I learned the most about making movies from the Coens, because of the way they do pre-production. The only way to make a movie come out on time, and on budget, is with a lot of pre-production. It’s where you can make all your decisions. There’s no pressure - it’s free. The worst time to make decisions is when the crew is playing football because you can’t figure out where to put the camera. You’re not going to make good decisions when it’s costing $10,000 an hour.

“The other man I learned a lot from was Scott Rudin, who I think is the single best producer around. The thing that Scott taught me was that almost everyone in Hollywood - almost everyone on the planet but most importantly in Hollywood - is afraid. They’re afraid of failure, they’re afraid of making their own decisions and having to stick by them. And if you’re willing to grab the person who’s saying no by the hand and say, ‘You know what? Let’s both commit suicide. Let’s run and jump off this cliff, because neither of us deserve to make this movie, or indeed live,’ then 999 times out of a 1,000 they will let go before you have to let go and say: ‘OK, I was just bluffing!’”

Rudin was the man who spotted Sonnenfeld’s potential, and gambled on him with The Addams Family. “I was a cinematographer; I had no interest in directing. He tried to get Tim Burton, then tried to get Terry Gilliam; they both said no. Rudin’s idea was, if you can’t get the guys you really want, don’t go for some middle-of-the-road choice; take a chance and either fail or succeed. So he said, ‘Let’s get Sonnenfeld, who’s a visual stylist.’ He just thought I was good, and I thought he was stupid for thinking I knew how to direct.”

As if to fulfil the New York Jewish liberal stereotype, Sonnenfeld waxes lyrical over the Charles Addams drawings in the New Yorker magazine (“They were always my source material - never the TV show”), before enthusing over Christina Ricci’s portrayal of sombre pug Wednesday Addams (“She was basically Barry Sonnenfeld: although everyone else in the movie was hamming it up, her whole role was just to state the truth. No spin.”)

The Addams Family was an unexpected hit, exuding a confidence of tone as it pulled off the difficult feat of blending a macabre comic sensibility with hi-tech special effects trickery. (“We spent more time trying to light Cousin It’s hair than we did on Anjelica Huston. We’d got some weird hair, made out of a reflective material, and if the light wasn’t exactly at 45 degrees to the lens, it messed up.”) After stumbling with the Michael J Fox vehicle For Love or Money (released briefly in the UK as The Concierge), Sonnenfeld was hired to direct another non-threatening effects movie, Forrest Gump, by producer Wendy Finerman; but, after persuading Tom Hanks to take the lead role, Sonnenfeld dropped out of the long-in-gestation project to direct Addams Family Values.

Between Addams Family Values and Men in Black came Get Shorty, the first of the wave of Elmore Leonard adaptations. (Sonnenfeld also served as executive producer on another one, Out of Sight, and turned Leonard’s novel Maximum Bob into a short-lived but cultish TV series.) Get Shorty is the least flamboyant of Sonnenfeld’s hits, but impressed audiences and critics with its savvy, knowing style.

Now, with Wild Wild West, Sonnenfeld no longer has to prove himself to anybody. He’s mastered his craft and, whatever the reviews of his latest film, the box-office returns tell their own story. If nothing else, Wild Wild West showcases once again his love of gadgetry, of trickery, of the magic of special effects cinema. “I must be some kind of masochist,” he ruminates, “everything in Addams Family to Wild Wild West is just a nightmare for the director. I hate shooting all that blue screen and wires stuff; but I guess I’m a whore - a masochist and a whore - because that stuff seems to work so well with the audience and it’s always good for a laugh. I go for it in spite of the personal pain it causes me.”