The New Yorker critic Whitney Balliett once described Thelonious Monk's idiosyncratic timing and unexpected resolutions as feeling like "missing the bottom step in the dark". His pauses could be so prolonged, you'd wonder if he'd left the studio. John Coltrane observed that playing Monk's music could be so rhythmically and harmonically taxing that if you missed a chord change it felt like falling into an empty elevator shaft. But these were among the qualities that turned Monk into one of the great jazz composers, whose rigorously beautiful, remorselessly pared-down themes are nowadays reinterpreted by both jazz and classical artists.

This great moment is the 1956 recording of his Brilliant Corners album with Sonny Rollins and others – acclaimed ever since as a jazz landmark, and a remarkable insight into the working of Monk's wayward musical mind. It was also a sensational comeback for him, since by the mid-1950s his sales were dwindling, and the loss of his crucial cabaret card on a drugs charge meant he couldn't play gigs in New York.

Monk emerged in the 1940s as one of the high priests of bebop, and was the house pianist at Minton's Playhouse, bop's after-hours laboratory. He learned piano while accompanying his mother's gospel singing, and his first influences were blues, hymns, swing, boogie and the "stride" piano style derived from ragtime. Monk fell in with young beboppers like Charlie Parker, Charlie Christian and Dizzy Gillespie in the after-hours Harlem jamming-joints of the early 40s, and his bohemian tastes and eccentric choices of headwear identified him as a hipster iconoclast. But he didn't really play like a bop pianist. Unlike his contemporary Bud Powell, he didn't sound as if he was delivering a piano version of Parker's skimming sax lines. He played percussively with fingers flattened and splayed, he left unexpected spaces others would have filled with showers of notes, and his chords clanged with dissonance.

The 50s had not been a good time for Monk, but though neglected he continued to explore. The results became apparent when he was rediscovered by two young jazz obsessives, Orrin Keepnews and Bill Grauer, who ran a jazz magazine called the Record Changer and wanted to move into production. Keepnews bought Monk out of an unsatisfactory record deal for $130, and thought at first he'd give Monk a fresh start on the new Riverside label by relaunching him as an interpreter of classic jazz themes. Monk made an album of Duke Ellington tunes that wasn't well received (Keepnews and Grauer were accused of manipulating him for the marketplace) but in December 1956 Riverside recorded Brilliant Corners, with Sonny Rollins and Ernie Henry on sax, Clark Terry on trumpet, Oscar Pettiford on bass and Max Roach on drums.

In an interview for BBC Radio 3's Jazz File in 2004, Keepnews told me that the session had been the toughest time he could ever recall in a studio. It took 24 takes to make the title track, as the opening dirgey theme with its bleakly reverberating phrase-turns shifted into swing and back, and even some of the most accomplished players in New York couldn't time it right. The final version had to be spliced from different takes, but the result conveys the essence of Monk's melodic and rhythmic audacity. When I asked Keepnews about the album's challenges, he said: The most obvious thing was a tempo change every eight bars. The nature of almost any Monk composition is that it's difficult to play – but Thelonious's problem was that he didn't want to admit there was anything unusual or difficult. If he understood it, then the others should. Monk was an impatient taskmaster."

The result of that singlemindedness, however, was some of the most innovative and unsentimentally beautiful music in jazz.