Born in Cincinnati, Ohio, Mundy began developing his arranging skills in the 1920s while playing with local bands led by Erskine Tate, Tommy Miles, and Carroll Dickerson. In 1932 he wrote and sold a few arrangements to Claude Hopkins, and at about the same time joined Earl Hines' famous "Grand Terrace" ballroom band in Chicago. Hines hired him originally as a saxophonist, for whom he worked over the course of the next four years. During this period Mundy developed a reputation as a prolific arranger in the emerging "swing" style and began writing and selling arrangements to other bandleaders in order to supplement his income.

Late in 1935, Goodman and his band worked their way east to Chicago where they began their historic six-month booking in the Joseph Urban room of the Congress Hotel. After selling one of his arrangements to Goodman, Goodman hired Mundy on a full-time basis. Until 1938, Mundy became one of Goodman's principal staff arrangers, joining Spud Murphy and Fletcher Henderson. From the moment he was hired, it was Mundy upon whom Goodman relied to create up-tempo "flag-waving" musical numbers. Mundy's list of "killer-dillers" include the 1936 (revised) version of "Bugle Call Rag", "Jam Session" (an original composition by Mundy), and the band's 1937 adaptation of "Ridin' High". Mundy was adept at arranging standard popular tunes: "You Turned The Tables On Me" (1936) and "And the Angels Sing" (1939).

When Gene Krupa left the band in 1938, Mundy left shortly after as well to write for Krupa's new outfit, although he continued to contribute scores to Goodman on a free-lance basis. He briefly led his own band in 1939 and throughout the 1940s Mundy supplied a significant number of original compositions and arrangements to Count Basie (ca. 1940 to ca. 1947), Artie Shaw (1944–45), Dizzy Gillespie (1949), Harry James, Charlie Spivak, Paul Whiteman and many others. He wrote the score to the 1955 Broadway musicalThe Vamp which starred Carol Channing. The 1957 musical Livin' The Life and the 2010 dance revue Come Fly Away also used some of his music. In 1959, he moved to Paris, where he was musical director for Barclay Records. He returned to the U.S. in the 1960s and continued with his active career as a writer into the 1970s.