Charles Starkweather was a 19-year-old warehouse worker with a James Dean obsession and an unpredictable temper. In the early hours of December 1 1957, Starkweather murdered a gas-station attendant who refused to let him buy a toy puppy on credit. The media christened him the Mad Dog Killer, and within two months, he had shot dead 10 more people, and fled across the midwest with his 14-year-old girlfriend, Caril Ann Fugate. The couple was apprehended in February 1958; Fugate sentenced to life, while Starkweather went to the electric chair the following year.

Terrence Malick began writing his screenplay Badlands, based on the pair's bloodthirsty road trip, in 1970, when he was 27. The story had already been loosely dramatised in the 1963 film The Sadist, though hardly anyone remembers that one now. Nor do the names Starkweather and Fugate ring too many bells. But time has been kinder to Badlands. Malick's transformation from dazzling young director to enigmatic recluse was never going to harm the reputation of one of the few films he actually got around to finishing. In fact, no film-maker apart from Jean Vigo has attracted such intense reverence with such a slim output. Badlands explains why. It is eloquent about the intersection of crime, romanticism and myth-making in America, and innovative in its use of colour, editing and voiceover. It's also a miracle that this mighty work could have emerged from such an apparently shambolic production.

Malick, the son of an oil company executive, came relatively late to film-making. He worked as a philosophy lecturer and journalist before arriving in Los Angeles in autumn 1969 to study at the American Film Institute. "I'd always liked movies in a kind of naive way," he explained in 1975. "They seemed no less improbable a career than anything else." Lanton Mills, the short film he made at the AFI, landed him some lucrative rewriting jobs: he worked on Jack Nicholson's Drive, He Said, and wrote an early draft of Dirty Harry when Marlon Brando was attached to star.

Badlands was the first feature he wrote for himself to direct. Executive producer Edward R Pressman raised half the budget, while Malick put in $25,000 of his own and harvested the rest from professionals outside the industry - lawyers, doctors, dentists. "I developed a kind of sales kit with slides and videotapes of actors, all with a view to presenting investors with something that would look ready to shoot. To my surprise, they didn't pay too much attention to it; they invested on faith."

Martin Sheen, who was cast as Kit, the nonchalant psychopath based on Starkweather, thought Badlands was the best script he had ever read. "Still is," he told this paper in 1999. "It was mesmerising. It disarmed you. It was a period piece, and yet of all time. It was extremely American, it caught the spirit of the people, of the culture, in a way that was immediately identifiable." Malick, known for his opaque direction, reputedly told the actor on set: "Think of the gun in your hand as a magic wand." Sissy Spacek played Holly, the baton-twirling schoolgirl who elopes with Kit after he kills her father (Warren Oates). And the photo-shy Malick took the small role of an impromptu caller when the actor he had cast failed to show. ("I prefer working behind the camera," he admitted later.)

Shooting began in July 1972. The production designer Jack Fisk, who was to work on all Malick's films, was later praised for his Edward Hopper-esque recreations of small-town America. "Whatever you see in Badlands is Terry's style, not mine," he said in 1982. "He's very strong visually. He was always willing, eager to change to things. He'd see something in the yard and say, 'Let's put that in the bedroom.' That's one thing I learned from him: spontaneity."

But that impulsiveness came close to wrecking the film. "The shoot went on for ever because the crew kept quitting," Spacek told me in 1999. "They were completely brutalised. They'd be setting up one shot over here, then Terry would look over in the other direction where the moon was rising up and he'd go, 'Let's shoot over there!' I have these memories of everyone tearing off across the desert in pursuit of one sunset or another." Fisk was one of the few who didn't quit. "I had a vested interest. I'd fallen in love with Sissy, so that also kept me going." The couple married two years later.

British cinematographer Brian Probyn established the dreamy texture of the picture before being taken ill, exhausted by the heat, the long hours and Malick's idiosyncrasies. "On several occasions," says Pressman, "I can recall Brian shooting with the slate upside down as a form of protest in a disagreement with Terry about methods of orthodox coverage and matching shots." Tak Fujimoto - who later became Jonathan Demme's regular cinematographer - took over after Probyn's depature, until a new director of photography, Steven Larner, was found. "Amazingly, despite the input of these different hands, the film looks remarkably seamless," says Pressman. "There were three cinematographers, lots of editors, sound men," said Fisk. "Except for the actors, the art department was the only one that completed the film. If the picture survived all those problems, it's because one thing was consistent: Terry Malick's vision."

There were other setbacks. All the cameras were destroyed during the filming of the morbidly beautiful sequence in which Kit burns down Holly's family home. By the end of the shoot, Malick was $35,000 over budget, forcing him to take on more rewriting assignments before the picture could be edited. But he inspired loyalty among those who stayed the course and kept their sanity. "People who've worked with Terry either love him or hate him," said Spacek. "I love him. Badlands was a real turning point for me. We'd spend hours talking about things, and then the next day I'd look at the rewrites, and there'd be all the things I told him."

The film's dislocated emotional effect arises almost entirely from Holly, whose blank-affect narration goes starkly against the grain. Traditionally, a voiceover fills in the gaps for the audience, but Badlands is defined by the contradiction between what we see and what we hear. "Holly seems at times to think of her narration as like what you get in audio-visual courses in high school," explained Malick. "When they're crossing the badlands, instead of telling us what's going on between Kit and herself, she describes what they ate and what it tasted like, as though we might be planning a similar trip."

What's more, her blank reaction when Kit guns down her father makes the slaying more shocking than any amount of hysterical identification. "She isn't indifferent about her father's death," Malick pointed out. "She might have cried buckets of tears, but she wouldn't think of telling you about it. It would not be proper. You should always feel there are large parts of her experience she's not including because she has a strong, if misplaced, sense of propriety." This suggestion that we may not be getting the full story is crucial to appreciating Malick, the master of the ellipsis, who is more likely at a moment of high drama to turn his camera on a quivering blade of grass or a bird wriggling in its nest.

After a 10-month editing period, Badlands was premiered at the New York film festival on October 13 1973. By all accounts, the reception it received overshadowed even Scorsese's Mean Streets, which played at the same festival. Vincent Canby of the New York Times called Malick's debut "cool, sometimes brilliant, always ferociously American". Sheen himself later said: "I will never be better than I was in Badlands." Audiences, however, stayed away. But if ever there was a lesson that box-office success is not synonymous with influence and longevity, Badlands is it.

Malick is now in post-production on Tree of Life, starring Brad Pitt and Sean Penn - only the fifth feature he has directed in 35 years, following Days of Heaven (1978), The Thin Red Line (1998) and The New World (2005). Back in 1991, when it seemed Malick would never return to cinema, Michael Cimino, director of The Deer Hunter, reflected on the film-maker as though rehearsing an obituary. "Terry was the one, he was something special," Cimino said. "He had a genuine gift, it wasn't just an eye, it wasn't just a facility with a camera, it wasn't just an ability to write, it was something inside him that was special. He was the greatest director of his time." His comments still stand - but in the present tense.