'I thought it would encounter difficulties," says John Fry, with delicious understatement, down the phone from Ardent Studios in Memphis, which he founded in 1959. "I thought people would find it so unconventional and so unfriendly that we would have difficulties." He's remembering 1975, when Ardent's promotions man, John King, and Jim Dickinson were visiting the major labels trying to sell a new album that had been recorded at Ardent and produced by Dickinson. "Jim used all his contacts – and he had some high-level ones, as did John. One of his friends at a large label said: 'Jim, I find this music very disturbing.' Another guy said to him: 'Jim, I hope I don't have to listen to this again.'" No one wanted the third album by the Memphis group Big Star, until it crept out in two markedly different versions on tiny labels in the UK and the US in 1978.

That's the album that a group of musicians will attempt to recreate in its entirety in London later this month. It's an album that doesn't have an official title or running order. There are even arguments about who it's really by. What's more, it's littered with mistakes – not deliberate mistakes, but errors that were left in place because the album's creator liked the disturbance they signalled.

"It's hard to nail the chaos," says Mitch Easter, one of the musicians – along with members of REM, Hot Chip, Teenage Fanclub and the Raconteurs, among others – who will be trying to perform Third (or is it called Sister Lovers? Or even Beale Street Green?) by Big Star (or is it really an Alex Chilton solo album? Or even by a group also called Sister Lovers?) at the Barbican on 28 May. "The guitar on some parts has to sound like it's going to crash and burn at any minute. Sometimes it truly crashes and burns and I'm like: 'Oops! Sorry!'"

Third is an album of soft moonlight and deep black holes. It's the sound of confusion and dislocation. It's an album that sounds as if it was being demolished even as it was being recorded, where a heartstoppingly beautiful melody might at any moment be washed away by a scree of white noise. It's an album where Chilton, the record's creator, might one minute be singing about a "wasted face … a sad-eyed lie … a holocaust" and the next be thanking his friends – "wouldn't be here if it wasn't for you". The guiding principle behind its recording, all its surviving personnel agree, was chaos, and the guiding theme behind its songs was disintegration – of Chilton's personal life, of his career, of his music.

By the time Chilton came to record Third in late 1974, he was already a music industry veteran, and he had been scarred and embittered by his experiences. He had been a star since the age of 16, as singer with the white soul group the Box Tops, who had a worldwide smash in 1967 with The Letter. He chafed against the unyielding grip the band's producers and writers exercised and quit in February 1970, eventually inveigling his way into a band formed by a Memphis musician called Chris Bell. The band, which took the name Big Star after a southern supermarket chain, recorded #1 Record, an album of delicious, muscular Beatlesesque pop, which was praised to the skies by the critics but went unheard by the public because of disastrous distribution failures. Bell left, and Chilton led the group into another album, Radio City, which suffered exactly the same fate.

Scattered around the US, though, were those handful of people who did get to hear Big Star, became obsessed with them, and who became champions and sometimes colleagues of Chilton. Among them, in Winston-Salem, North Carolina, were Easter and his schoolfriend Chris Stamey, who's the musical director for the Third show. "I heard Big Star in the car," remembers Easter. "They played it on the top 40 radio station here in Winston-Salem." Easter, Stamey and their friends all bought #1 Record – and set about learning its songs for themselves, something that was to prove useful later in the decade.

By the time Chilton came to write the songs for Third, Big Star were barely a band. They were down from their original four members to two: Chilton and Jody Stephens, the drummer, hence the confusion about who it's really by. "I don't think anybody was quite certain about whether it was a Big Star record or what it was," says Fry, who engineered all three of Big Star's records. "Well, I gotta tell you," says Stephens, "I always had a sense of it being a Big Star record, except for a conversation with [Alex] saying: 'Why don't we call ourselves Sister Lovers?' It was a suggestion. Outside of that, I'll leave it up to people to assess whether it is to all intents and purposes an Alex solo album." The one person who might settle the question never showed any great interest in doing so, and nor, now, is he able to – Chilton died of a heart attack in March 2010. The album's other great progenitor, Dickinson, died a year earlier.

Whoever it is by, Third marked a staggering change in tack. Gone were the Beatles, the presiding influence this time being the Velvet Underground at their very darkest, not that it sounds much like the Velvet Underground, despite a cover of Femme Fatale. What it sounds like is a man letting everything inside him, the good and the bad – and Chilton had plenty of both – spill out.

"We called him, lovingly, 'the tortured artist'," recalls Chilton's long-time friend Pat Rainer. Nowaday's she's a travel agent in Beverly Hills, but in 1974 she was working alongside Dickinson on the recording of Third. "Alex had a vision. He was not someone who was ready at that point to let anyone tell him what to do, because he had been just the opposite for so long in his professional career. He was finally free." That freedom was employed to create a record that served to exemplify what the Memphis music historian and writer Robert Gordon describes as a key strain in the city's musical heritage – "one of individuality, one of letting the masses be damned. In a way it's the heart of the blues, which goes back to the beginning of the 1900s, the idea of 'I'm gonna tell my tale and I'm gonna do it for myself because it makes me feel better.' The first two Big Star albums are reaching for big public acceptance and the third is definitely not."

Four things shaped the sound of Third, all of which are inextricably linked: the presence of drugs and alcohol, the atmosphere of Memphis's Midtown district in the mid-70s, the interaction between Chilton and Dickinson, and Chilton's relationship with his girlfriend and muse, Lesa Aldridge, nowadays a high-school English teacher known as Elizabeth Hoehn, but then, by her own admission, a bit of a wild child.

Hoehn was 18 when she first met Chilton in early 1973 in a Midtown bar. Across the room, she saw "the most beautiful man. My heart just skipped a beat". By the autumn they were going steady, but theirs was a volatile relationship. "He was very, very jealous, but he could be so charming, so considerate." The jealousy might show up in violence, the charm in the times the couple spent alone together. Third's alternate title, Sister Lovers, comes from the fact that while she was dating Chilton, her sister Holliday Aldridge was dating Stephens, and some of its most tender moments were written with the times of peace at their heart.

"He finished Blue Moon while I slept," she says, of one of the album's loveliest songs. "He was inspired by I'll Be Your Mirror. To me it's always been the same tone of song as Blue Moon. But as with I'll Be Your Mirror, Alex did not know the beauty he was. So he wrote the song and he finished it while I was sleeping. When I woke up, there was the cassette recorder and a note saying: 'It's all cued up for you, baby.' And I played it and I loved it. It was a validation of his love for me – he loved me every bit as much as I loved him. You can't write that kind of a song for someone if you don't."

She looks back on her relationship with Chilton – which lasted on and off until 1980 – with a mixture of sadness and nostalgia, remembering the walks in the sun, the evenings spent listening to the Velvet Underground or watching movies. "I loved him so, so much," she says. "We used to love to watch the old black and white movies on TV, particularly The Thin Man. He always made me feel that I was as pretty as Myrna Loy. He would make dinner for me and we would sit and watch this station that played old black and white movies at night, and it was just so dear and sweet. Our personal relationship was in its best light when it was just the two of us, alone. Sober and alone."

"It was, 'I love you … I hate you … I love you … I hate you … off and on, off and on, constantly," Rainer remembers of the couple's relationship. "They loved each other, they fought. Lesa threw all his clothes out of the window. We were all still really young at this point, but it was an extreme thing – extreme love or extreme hate. Extreme mayhem."

If many of the songs were written during moments of extreme love, many of the final decisions about their recording were made when extreme hate was the rule. Though Hoehn recorded vocal parts for many of the songs, all bar one were wiped from the final versions, the one that remained being a version of the Velvet Underground's Femme Fatale. "I always thought Femme Fatale was directed at me," Hoehn says. "Unfairly. But there was nothing I could do about it."

Talk of final versions of the songs, though, isn't always helpful. As Fry explains, it was hard to know when a song was completed. "You really didn't," he says. "It was one of those things where we kept recording and recording and recording and you really didn't know when you were done, or when you were going to go back to a track and add something to it, or modify it in some way. I thought the recording process was wearing on everybody and certainly it was wearing on me."

Dickinson, however, embraced mayhem. He was a decade or so older than Chilton, and as a musican, he'd already played with the Rolling Stones and Aretha Franklin, among many others, and produced records for Ry Cooder, as well as recording in his own right. He had a range of strange experiences in his past, too – he claimed to have had LSD tested on him by the US government while at university in Texas in the early 60s. His introduction to the recordings, as he told Big Star's biographer Rob Jovanovic, was crazy enough: "Alex set the tone of the recording on the night of the first session, when he turned to me and shot Demerol down his throat with a syringe."

Jovanovic also quotes Chilton on the modus operandi of recording: "I had a big drink and drugs lifestyle. I'd get up at four in the afternoon. Smoke a few joints, drink, see some bands. Party all the time. I'd go into Ardent studios late at night after I'd been to bars and just throw gin over the mixing board. Seemed a pretty revolutionary idea at the time … [There was] a lot of Mandrax … downs, generally. Whatever there was there. And unbelievable amounts of alcohol. So if you take enough bad drugs and drink, you're gonna be writing some pretty strange music."

Dickinson had to win a rapport with Chilton, which he did by embracing the singer's desire to confront. One of the album's strangest and most beautiful moments is Kanga Roo – one of two songs, along with the spectral piano ballad Holocaust, later recorded by This Mortal Coil – which owes its oddness to Chilton testing the producer. He recorded both vocal and acoustic guitar to the same track, making it near impossible to mix. He handed the tape to Dickinson, saying: "You want to produce? Produce this." Dickinson added sheets of feedback, layered on some of his own rudimentary drumming, and gave a ghostly sheen of mellotron. The bond between the two was forged.

If Chilton and Dickinson rejoiced in the chaos, others did not. "It was a hard record to do," Fry says. So hard, in fact, that he stepped back from engineering after recording, concentrating on running Ardent and setting up a Christian label. "It was a number of years before I could go back to the record and enjoy it without it bringing back unpleasant memories." He recalls one incident where he entered the studio to find what appeared to be a homeless man being asked to record overdubs for a version of Jerry Lee Lewis's Whole Lotta Shakin' Goin' On. "It just kind of struck me the wrong way. I felt like it was making fun of him or taking advantage of him."

Others, though, have happier memories. Stephens recalls the sessions being "a wonderment of wow. For me, I could get up and leave if I felt things were getting a bit tense or thick. I don't know that it happened too often, but it could be that John had a lot more presence in the making of it than I did."

In many ways the album reflected the lifestyle of the Midtown crowd of which Big Star were part. "Midtown really speaks volumes about the scene," Rainer says. She talks of characters like John McIntire, an art lecturer, whose open-house home was known as Beatnik Manor, and Randall Lyon, "better known as the Guru Biloxi. One night Alex and Jim were playing at Procapé [a Midtown venue] and Randall came in wearing a choirboy cape and danced with a human skeleton while we all burned money." Then there was Hoehn's cousin, the photographer William Eggleston, who plays piano on Third's version of the Nat King Cole standard, Nature Boy.

Rainer calls Midtown, centred around Overton Square, "the creative epicentre. Even though Dickinson wouldn't live down there – he always wanted to keep his family at arm's length from the mayhem." And perhaps away from Chilton, who needed a minder to stop him getting into trouble in bars such as Trader Dick's: Chilton was not afraid to provoke confrontation, as the stories Gordon tells in his book It Came From Memphis make clear.

But when you listen to Third, amid the chaos you can hear order. The demos for the album, which appeared on the 2009 box set Keep an Eye on the Sky – usually just Chilton singing to his own acoustic 12-string guitar – show the songs were properly and conventionally structured, even if the album versions weren't. Downs, for example – with a lyric by Hoehn, and a title that reflects one of their pharmaceutical interests – is sung to a conventional rock'n'roll guitar line on the demo, while on the album that's replaced by a clatter of marimba and percussion.

"I have the original multitrack of that," Chris Stamey says, "and it was based around that same guitar, but it's like an art technique – you base something around that one element and then you remove the element. We're talking about a record that was reviewed as being nuts when it came out. Now, if we were to listen to it with fresh ears, which is probably impossible, it sounds like a very careful, organised record compared to what has happened in the world of indie record making in the last three decades."

The level of care, of thought, was demonstrated in the string arrangements that adorn the record, providing a sense of structure and human warmth, where one might fear only despair. "Wow, you hear those strings and it's really amazing," Stephens says. "At the end of the day, the strings are almost the stars." The string parts were composed by a 24-year-old called Carl Marsh, who has gone on to write string arrangements for scores of huge-selling albums.

"In the mid 70s I was working at Ardent quite a lot with Steve Cropper [of Booker T and the MGs] as a guitarist and arranger on Steve's R&B records," Marsh says. "We were in Studio B daily for 10 to 12 hours turning out album after album. So it was easy to run into the members of the Big Star production team during the course of the day, usually in the hall around the atrium or the lounge." He'd known Jody Stephens for some time, and when Stephens asked him to write some string parts for the first song the drummer had written for Big Star, For You, Marsh obliged. When Chilton heard there was a string arranger to hand, he asked for four more orchestrations, and then involved himself in the composition for the fifth, Thank You Friends.

"It was painstakingly dictated to me line by line for eight hours by Alex," Marsh says. "Interestingly, when the session came around, he listened to the strings play his lines and decided to throw them all out and start again. In the middle of the string session. I sang him examples, which he would OK, then I conveyed to the string players how to write them. This took a while. After the session I overheard a violist say that her chart had so many erasures and additions that it resembled the bombing of Dresden."

Third was never really finished. Fry, finally, had enough and told Dickinson the sessions had to end. And that might have seemed to be that: an album no label wanted, with no real band to promote it. Big Star had always been a studio band rather than a touring band, but by the time Third was finished, they were non-existent. The one Big Star live show from this era is neither a show, nor by Big Star – it's a brief set on the Memphis student station WLYX, in which a clearly refreshed Chilton (legend holds he had dropped acid earlier in the day, then taken Mandrax before going on air to bring himself back down) runs through a brief selection of Third songs, plus I Will Always Love You and The Letter.

Even though it hadn't been released, the legend of Third began to spread around those in the know. In 1976 Chris Stamey saw an advert in the back of a music magazine for a bootleg of the album that Ardent had been hawking round the record companies. He ordered a copy, "and it sounded wild, like nothing we'd ever heard, really. I have to say that at the time I attributed that to it being a dub of a dub on cassette. Then when I heard the actual record, it sounded the same way. It was hard to take in the first time I heard it, but it was a big leap."

The following year Stamey had moved to New York and became Chilton's sideman, playing bass in his band the Cossacks, with Chilton putting the lessons of Third into practice. "He had learned a lot from Jim Dickinson when making that record and he was very interested in applying what he had learned from Jim when we went into the studio that year." Live, the Cossacks were playing songs from Third, but altered once again – Kanga Roo began to feature improvisatory passages, Holocaust was reinvented as a cracked and raw full-band number. Live tapes of the Cossacks suggest a band daring themselves to stand on a musical cliff-edge, playing Chilton originals and 60s covers in a manner that veers between ragged and unhinged.

Mitch Easter, who also worked with Chilton in New York in that period, remembers a man who could display two sides of his character equally suddenly and equally unexpectedly. "When I went up to New York to see him for the first time – I can't remember how this worked out – my mom was with me. We all went out to eat later, and we ended up at this diner with him and his band, and he had this great conversation with my mom about astrology, and he was totally for real. But if he didn't like someone, he could just slash them to ribbons."

Chilton's maverick spirit, and presence in New York, made him an attractive elder statesman to the punk scene, and began the rehabilitation of Big Star. He returned to Memphis, bringing the Cramps with him to produce an EP and LP for them. He recorded a solo album, Like Flies on Sherbert, whose derangement makes Third seem sedate – "That's a descent into madness, for sure," says Pat Rainer, who engineered it – and helped form the bizarro rockabilly-punk outfit Tav Falco's Panther Burns. Chilton's name cropping up on the sleeves of these hip records made him a contemporary figure again, and spurred interest in his past. "It's the auteur theory, right?" Robert Gordon says. "We get interested in the artist's lives. Alex is an example of, and he defies, the auteur theory. You certainly wouldn't put The Letter, #1 Record, Third and Like Files on Sherbert together and go: 'This is one person's career.'"

And so Third began its long, slow rehabilitation, aided first by its eventual release, then by Big Star becoming an article of faith among alternative musicians of the 80s, notably REM in the US, and This Mortal Coil in the UK. By the time it got its biggest release, in a version compiled by Dickinson in 1992, it was regarded as one of rock's classic albums. Big Star's reputation was further burnished, and their music spread more widely, when Chilton unexpectedly agreed to re-form the band in 1993 – that version of Big Star, featuring Stephens and two members of the Posies, continued to tour until Chilton's death, and even managed to record a new album in 2005.

As Gordon says, Third's status is inextricably linked to its story. "I am no longer able to separate the music from the narrative," he says. "And that applies to more than just Big Star – you get into reading about the artist and it affects how you hear the work." And, crucially, the story of Third proves the truth of its music: "You can't create that many songs, that many performances, and that many sounds by just imagining yourself being miserable. To that end it's like a documentary. We, the audience, are looking to find ourselves in other peoples' greatness and I think in Big Star that misery becomes our own misery, if we want it to."

The misery, at least, would pass. By the time of his death, Chilton was content with his life after years of restlessness. "I'm very happy that he met his widow Laura when he did because I really saw a great change in him and I saw him become very happy," Pat Rainer says. "I'm so glad that happened for him."

And now, without its creator, Third is to be performed in its entirety for the first time. How would Chilton have felt, given his famed awkwardness about the subject of Big Star? "I don't know that Alex would have joined in," Stephens says. "He might have said: 'I don't want to be a part of this, but whatever …' But Alex is definitely present in all these things. The air is thick with Alex."

Big Star Third is at the Barbican, London, on 28 May. Details: barbican.org.uk. Chris Stamey plays the Slaughtered Lamb, London, on 29 May, and his own group, the dBs, release Falling Off the Sky, their first album in 25 years, in August.