Michael Jackson's life and career were so extraordinary that it's tempting to comb each chapter for clues and pointers to the next. You can find flashes of his solo superstardom in his thrilling contributions to the Jackson 5's pop. You can detect hints of paranoia and self-hatred in his years as a commercial colossus. And it's all too easy to hear his later, darker solo work as presaging the end of something.

All of which risks making Jackson's transformation into the king of pop seem inevitable, whereas it was no such thing. In 1978, he was an ex-solo star, modestly successful in his day but now re-absorbed into the Jacksons machine, where he played his part in creating marvellous platform-heeled stompers such as Can You Feel It?. This wasn't the only string to his bow, mind you: that year he could also be seen in Motown's grand filmic folly, The Wiz, playing the Scarecrow. The critics were kind to his role in this reimagining of The Wizard of Oz, which is to say they didn't think he was as godawful as Diana Ross's Dorothy. And speaking of clues and pointers, Jackson first appeared in the film lashed to a metal cross, struggling for freedom as wiseass crows make him sing inanities for their entertainment.

The man who helped him down in real life was veteran musician and producer Quincy Jones, working as an arranger on the film. Jones got on well enough with Jackson to put his name forward when the younger man asked who ought to produce his next solo record. Demand for the record was far from certain: Jackson's previous solo album Forever, Michael had come out four years earlier and had hit a dizzying No 101 on the US charts. The collaboration with Jones managed rather better, of course. Off the Wall sold 20m copies, impressive enough even before you remember that was only one-fifth of the business its follow-up, Thriller, did. In an industry shifting away from individual sales, we can say with more certainty that Thriller will keep its position as the highest-selling LP ever – but it's Off the Wall that critics routinely hail as Jackson's masterpiece.

Why? What's impressive more than 30 years later is how tight and self-disciplined the album is. It was a disco record released at disco's overripe peak, and it's by a man whose own decadent and kitschy impulses would become ever more apparent. But Off the Wall is joyously compact – shot after shot of lean grooves that never get a chance to wear out their welcome.

This sense of restraint is a vital part of all Jones's collaborations with Jackson. The singer's vocal tics – the gasps and shudders that punctuate almost every song – play a big role in this, creating the impression of a singer desperate to cut loose and express himself in movement. But there's never any excess in the music to undermine Jackson's hunger. On Thriller and Bad, the restraint starts to curdle into tension on songs such as Billie Jean and Smooth Criminal, and Jackson sounds compellingly trapped. After Jones suggested Jackson find newer, more modern collaborators, this sense of musical restraint began to vanish: epics such as Stranger in Moscow or Earth Song were more grandiose than anything Jackson had tried before, but also somehow more private, too.

Off the Wall, though, finds Jackson at his most relatable. It announces itself – and Jackson's arrival as a solo songwriter – with Don't Stop 'Til You Get Enough, whose spoken intro finds him struggling like a Marvel comics character with the power inside him, then giving it glorious release. At this stage, he's not the Peter Pan of pop so much as its Peter Parker, a troubled young man learning how to use his staggering abilities, and every emotion on Off the Wall seems like it's being felt and expressed for the first time. By the time Jackson's voice cracks on centrepiece ballad She's Out of My Life, the record is already a triumph.

Oddly, since Jackson's death in 2009, Off the Wall has been one of his less influential albums. Perhaps it's too innocent. The Jackson tracks that resonate with today's stars are later ones, where Jackson treated his fame as vocation and millstone simultaneously – an attitude familiar to the self-consciously tormented likes of Kanye West. The last record with an obvious debt to Off the Wall was Justin Timberlake's fine Justified, almost a decade ago. Like its inspiration, that record is a boy announcing himself – in pop terms – as a man, in an act of homage that gets to the emotional core of Off the Wall's appeal. Whatever happened afterwards, with this record Jackson and his mentor Jones made pop's great coming-of-age album.