It was while dodging explosions of hype and litigation that Bruce Springsteen made the decision to pare his 1978 album Darkness on the Edge of Town back to 10 songs focusing on a mood of sombre reflection. The ambition was to make his audience view him in a different light: as a man of conscience rather than a mere purveyor of exuberant revivalist rock'n' roll, the guise in which he had made his appearance earlier in the decade. "I wanted to deal in reality," he said recently, thinking back to a time when he was grappling with the troublesome implications of his new celebrity, "and I wanted to start a conversation about that."

The decision involved risk, and in the fond view of his longtime friend and collaborator Miami Steve van Zandt, the consequent gains came with a price tag. "It's a bit tragic, in a way," Van Zandt says during a new documentary film examining the making of Darkness, "because he would have been one of the great pop-song writers of all time."

Van Zandt was being only half-serious. Springsteen's decision to embrace a sense of the struggle against life's compromises and limitations in his songs ensured that his appeal would create a deeper and longer lasting bond of loyalty and affection among his audience. And that gift for compressing joy into a hook-laden three minutes was never entirely extinguished.

Directed by Thom Zimny, the 85-minute documentary forms part of the deluxe edition of the Darkness reissue, along with a film of a marvellous concert from Houston, Texas in 1978, a stirring performance of the entire album by Springsteen and the E Street Band shot last year in a small Asbury Park theatre, and two CDs compiling 21 of the best of the 60 songs Springsteen rejected while spending three years on the project that would occupy a special place in his canon. That process is further illuminated by the documentary's extensive use of 8mm monochrome footage shot in the studio while the album was being prepared.

"Lean", "angry" and "relentless" are the words Springsteen now uses to describe the feelings and sounds he was trying to capture in the follow-up to Born to Run, the hit album that landed him on the cover of Time and Newsweek and led some to write him off as just another victim of record-company excess. Delayed by a bitter legal battle over his management contract, he finally presented the public with a version of Darkness from which all traces of joy and release had been purged, replaced by a sense of desperation. The claustrophobia of ordinary lives trapped by work and family ties provided his metaphor.

The result, at its best, was a song such as Racing in the Street, in which the riff from the Crystals' Then He Kissed Me, a motif immediately indentified with the rush of teenage euphoria, was slowed down to provide the dirge-like underpinning for a first-person meditation – inspired by Monte Hellman's 1971 existential road movie Two-Lane Blacktop – on the predicament of a hot-rodder whose adolescent dreams are fading into obsolescence. This formed the album's epic centrepiece, and still arouses complex feelings when Springsteen performs it in concert. Like all his songs, it was the subject of extensive development and careful revision: the two CDs of previously unreleased material contain an earlier version that lacks the borrowed riff of the finished article, and therefore its special poignancy.

Springsteen's distinctive compositional method involves the permutation and re-examination of familiar phrases, shuffling sequences of notes or words from one song to another until they find their place. The inclusion of Racing in the Street on Darkness on the Edge of Town necessitated the omission of The Promise, a song quite its equal, which shares its tune and mise-en-scène while also incorporating a reference to Thunder Road, the great Born to Run song. Much bootlegged and issued in a later, solo version a dozen years ago on the box set Tracks, The Promise is now unveiled in its original full-scale majesty.

Its elegiac tone would have fitted Darkness perfectly, but most of the other 20 previously unreleased tracks demonstrate that Springsteen never actually stopped writing the hook-laden, audience-rousing crackers with which he made his name. They include the sultry Fire and the towering Because the Night, made famous by other singers, alongside the delirious Rendezvous, the comic Ain't Good Enough for You and the headily romantic Someday (We'll Be Together), which could still give Frankie Valli his biggest hit in decades.

Tagged on to the end of the second disc as a "hidden" track is The Way, a gem of a love song whose exclusion from the original artifact exemplifies the demands Springsteen was making of himself and his art, while making themost fateful decision of a remarkable career.

Hobbled by legal wrangles, a frustrated Bruce Springsteen turned Born to Run's optimism on its head – and Darkness on the Edge of Town was born. With a hugely expanded reissue finally compiled, he talks to Keith Cameron