They call him the Boss, and apparently for good reason. Bruce Springsteen tonight told an audience in London that he almost drove his group, the E Street Band, "crazy" with his exacting demands during the two-year recording process for their 1978 album Darkness on the Edge of Town.

Despite coming the success of the enormous 1975 hit Born to Run, Springsteen described being racked with uncertainty during the sessions, which marked the first time the band had recorded together for more than a year following a legal dispute between the singer and his former manager, Mike Appel.

"I was so full of doubt that the only way I knew to get somewhere was to struggle away for hours and hours," he said. "Each one of us has a story inside, and that story doesn't let you go. I kept having to answer the call of that story but I couldn't explain that at the time to the other guys. So they thought I was just trying to drive them crazy."

The marathon recording sessions were shot by an amateur film-maker friend but remained hidden from public view for 30 years. Director Thom Zimny has now assembled them into a new documentary, The Promise: the Making of Darkness on the Edge of Town, which was screened at the BFI Southbank prior to Springsteen making his comments. In it, the singer and his band are seen agonising for hours over the sound of a snare drum, and discarding scores of songs because they did not fit the rawer, less commercial aesthetic which Springsteen was determined to pursue after the success of his previous album and having taken note of the emerging punk sound.

In total, 70 songs were recorded during the sessions, with only 10 making it onto the final cut. Some re-emerged on later Springsteen albums, while others – such as the Patti Smith hit Because the Night – were recorded by other artists. Those which were never released have now been packaged along with the new film, and Springsteen insisted they were far more than out-takes.

"The music that was left out was really a separate record," he said. "The songs which made it on were the ones which were right for the album, but there are a lot of beautiful songs left that we had a chance to finish over the summer.

"We were interested in making a record that was not just the best songs," he explained. "Some of the music we recorded had too much modulation going on, too much melody. Too much richness of arrangement. Everything had to go."