The saxophonist, clarinetist and microtonal-music explorer Joe Maneri, who has died aged 82, was 65 before the wider world acknowledged his talent. That year he played alongside his violinist son, Mat, and one-time Ornette Coleman pianist Paul Bley, at the Montreal jazz festival, and thrilled an astonished audience. Adventurous jazz and new-music labels, led by Leo Records in the UK, queued up to document a mysterious, sometimes discomfiting, yet captivating sound that had hardly been aired in public before, even if its message had filtered out through many of Boston's New England Conservatory graduates during Maneri's long teaching career there. Maneri went on to record extensively (usually with his son) through the 1990s and into the 21st century, and with those sessions a missing piece in the jigsaw of jazz and improvised music finally found its rightful place.

Maneri thought that western diatonic scales and key signatures were outmoded, reinvented the octave as a 72-note unit full of pitch-intervals mostly unconsidered in European traditions, and improvised in that language with fearless verve. He founded the Boston Microtonal Society in 1988, and taught players such as the pianists John Medeski, Jack Reilly and Matthew Shipp and the saxophonist Matana Roberts.

Born and raised in Brooklyn's Williamsburg district, Maneri was the only child of a Sicilian immigrant father and mother who respectively played clarinet and sang. Fascinated by music, Maneri was taught to play clarinet by two neighbourhood players who also made furniture and repaired shoes, but learning difficulties, then undiagnosed, led to him leaving school at 14. Before he was out of his teens, however, Maneri was playing dance-music for money – and he developed his craft busking for weddings, barmitzvahs and belly-dancers in New York in the 1940s, blending the sounds of American jazz, Greek, Turkish and klezmer music into a pungent stew. Attracted by the liberating possibilities of free improvisation, he was drawn to jazz, from the warped-pitch eloquence of the swing clarinet virtuoso Pee Wee Russell to the fiery modern-sax sermons of Charlie Parker.

In 1947, playing tenor saxophone, Maneri began frequenting Ernie's, a Greenwich Village haunt, with the pianist-arranger Ted Harris and a coterie of mavericks who were splicing jazz phrasing and improvisation into Arnold Schoenberg's modern atonal structures. Through Harris, Maneri met the Austrian composer Josef Schmid, a former student of Schoenberg's disciple Alban Berg. Maneri studied with Schmid for the next decade.

Maneri began composing, and his Divertimento for Piano, Drums and Double Bass was performed at Carnegie Recital Hall in 1961. He was also commissioned to write for the Boston Symphony Orchestra, while teaching harmony, 16th-century counterpoint and composition at the Brooklyn-Queens Conservatory of Music, where he met his wife-to-be, Sonja Holzwarth. But it was Maneri's tenure as a teacher of theory, composition and performance technique at the New England Conservatory (NEC), in Boston, that was to become the focus of his musical life. When NEC president Gunther Schuller, who had played on the Miles Davis Birth of the Cool sessions, created a new department to investigate the "third stream" classical/jazz crossovers explored in the early 1960s, Maneri was the ideal recruit.

Maneri worked on microtonal projects, including the 588-note keyboard, and a book (co-authored with Scott Van Duyne), Preliminary Studies in the Virtual Pitch Continuum (1986). In 1977 he co-founded the Enchanted Circle contemporary-music concert series at the NEC, and today's New York "downtown" scene is widely held to have some of its deepest roots in the inspiration of those events.

As his violin and viola-playing son Mat's stature on the free-jazz scene grew at the end of the 80s, Maneri Sr accepted his offspring's invitation to return to the stage. After their Montreal appearance, Leo Records released Get Ready to Receive Yourself (1993), and Let the Horse Go (1995), which fully displayed Joe Maneri's power and vision, from music celebrating Anton Webern to a highly original version of Body and Soul.

In 1995, Maneri recorded a soundtrack for a film by his painter friend, George Dworzan, using improvised reeds and piano parts and varying the playback speeds in the mix. A Maneri test recording that Schuller had fixed with Atlantic Records in the 60s was released three decades later as Paniots Nine (on John Zorn's label), and Bley drew ECM Records into the Maneri orbit for Three Men Walking (1995), partnering both Maneris with the guitarist Joe Morris. Switzerland's Hat Art label then released Maneri's Dahabenzapple, Comin' Down the Mountain and Tenderly, and ECM brought out the highly divergent sequence In Full Cry, Blessed, Tales of Rohnlief and Angles of Repose.

The pianist Ran Blake described Joe Maneri's playing thus: "When one hears Maneri on his clarinet, one hears the cries of the Middle East and Brooklyn, where Joe lived for many years. His own microtones come not from his theory but from life's experiences. Some of his notes scream with ghetto resistance." Maneri was awarded an honorary doctorate by the NEC in May this year.