In 1964 Taussig took a job as a physicist for Ford-Aeronutronics Corporation in Orange County and began studying art and photography with John Upton at Orange Coast College. In 1965 he recorded his first solo album, Fate is Only Once, under the name Harry Taussig. Played in a single 45-minute take, the record is partially improvised and, according to Taussig, full of mistakes. Issued on a private label, few copies of the original record existed, making it a collectors item for fans of American primitive guitar and fingerstyle guitar.[3]

The following year Taussig recorded two songs for the compilation Contemporary Guitar: Spring '67 for John Fahey's Takoma label. The album featured similar American primitive guitarists, including Fahey, Max Ochs, Robbie Basho and Bukka White.[4] Taussig's only live performances during this period were informal sessions for friends. He taught guitar for some years and wrote several books on guitar and autoharp instruction for Oak Publications before retiring from music altogether.[3]

While finishing his scientific studies, Taussig was also beginning to exhibit his photography internationally, as well as work in other visual mediums, including collage and film studies, which he taught at Orange Coast College under the name H. Arthur Taussig. He gradually became well known as a film analyst, curating film for the Orange County Museum and publishing the 1997 book Film Values/Family Values: A Parents' Guide.[7] As a visual artist, Taussig has produced a series of illuminated books of classic works including Dante’s The Divine Comedy, James Joyce’s Ulysses, Shakespeare’s Hamlet, among others. Taussig also creates collages based on Jungian archetypes and has published a series of his collages as a deck of tarot cards.[2]

In 2006 the label Tompkins Square Records reissued Fate is Only Once—the only album Taussig had recorded to date—41 years after its original release. In 2012 Taussig put together a second album—Fate is Only Twice—at home on a laptop, affording him the luxury of recording the multiple takes he was denied by the 1965 recording process. In 2013 he made his public performance debut at the South by Southwest Festival in Austin, Texas where Wired Magazine' listed him as one of the top 50 performances.[3]

In 2014 Tompkins Square released Taussig's third album, The Diamond of Lost Alphabets, and 2016 his fourth album, Too Late to Die Young, in which the guitarist explored polyrhythms and unconventional time signatures.[9] Tompkins Square also commissioned Brooklyn-based experimental musicianKid Millions to remix some of Taussig's original material, and in June 2016 released Beyond The Confession: Kid Millions Reworks Harry Taussig. With engineer Matthew Cullen, Millions overdubbed guitars, drums and organs onto selected tracks from Taussig's first three albums, melding Taussig's compositions into a new soundscape.[10]

In 2017 to commemorate the 50th anniversary of Takoma's Contemporary Guitar: Spring '67, Tompkins Square reuntited Taussig with fellow fingerstylist Max Ochs for the release of the album Remembrance of Things Past featuring five songs by Taussig and three by Ochs.[11] The two musicians also made live appearances to support the release.[12]