Kramer's first experience in the New York music scene came when he played in the band New York Gong, led by Daevid Allen, in 1979 and 1980.[3] Kramer played organ on one song on their 1979 album About Time.

In 1980, Kramer joined the band The Chadbournes, led by Eugene Chadbourne, which also included David Licht, Tom Cora and John Zorn. They played together until Chadbourne, Kramer and Licht formed the band Shockabilly, which toured from 1982 until 1985.[3] The pressures of living on the road led to personal differences between Kramer and Chadbourne. The band dissolved while on a US tour early in 1985. That tour included a brief tour of Texas with the then-unknown Butthole Surfers. Forging a close friendship with co-founding Butthole Surfers members Gibby Haynes and Paul Leary, Kramer was available when the band needed to replace their bassist. Kramer bought a Höfner Beatle bass and joined the Butthole Surfers' debut European tour in 1985.

After touring, Kramer took over a New York recording studio named "Noise New York", using a loan of $5,000 from an uncle.[3] The studio was to serve as a mainstay for artists and bands both local and international, and Kramer became one of the busiest indie music producers in NYC. The first recording at Noise New York was the Buttholes' rendition of "American Woman".[3]

In the early 1980s Kramer met Ann Magnuson, New York City performance artist, when he ran the sound for a band she was in, Pulsallama, during their frequent performances at Club 57. After Pulsallama disbanded in 1984, the two began to collaborate, and in 1986 they formed Bongwater. Together they released five LPs, including Double Bummer, and culminating with their 1991 swansong, The Big Sell-Out.[2] In 1991 they began a romantic relationship while Kramer was separated from his estranged wife, who was at that point three months pregnant with their daughter.[3] After several months, Kramer decided to end the romantic relationship and reconcile with his wife, and the end of the relationship also spelled the end of the band.[3] Several months later, Magnuson sued Kramer for $4.5 million for breach of contract, among other charges, and Kramer responded with a counter-suit.[3] The subsequent legal battle resulted in the financial crippling of the Shimmy-Disc label, which never recovered. The lawsuits were eventually settled out of court in 1997 for undisclosed terms.[3]

A pivotal moment in Kramer's early career came when Jad Fair of Half Japanese introduced him to Penn & Teller. Kramer soon found himself working eight shows per week as Sound Consultant on Penn & Teller's 1987 Broadway show, and composing the music for their Cruel Tricks for Dear Friends special. In 1992, Kramer formed the band the Captain Howdy with Penn Jillette, and together with guest artists Debbie Harry (of Blondie) and Billy West, they made two highly eclectic CDs together, both released on Shimmy-Disc. Following Penn's permanent relocation to Las Vegas in 1997, the group disbanded.

In 1988 Kramer and Fair released the record Roll Out the Barrel together on Kramer's Shimmy-Disc label. They reunited in 1998 and published the record The Sound of Music. An Unfinished Symphony in 12 Parts that year.

In 1990 Kramer first met the two-man band Ween when they played a show at a small New York club, the Pyramid Club. Kramer struck up a friendship with the two, who had already been fans of the Shimmy-Disc label, and he convinced them to release their home-recorded demos on Shimmy-Disc, on what became the 1991 album The Pod. In 1992 the three went on a brief tour of England, where Kramer played bass. The tour did not go well, due to personality and creative differences between Kramer and the two members of Ween, although Ween band member Mickey Melchiondo said in a 2007 interview that "we're still friends".[3]

In 1992 Kramer sold his Noise New York recording studio and moved just across the Hudson River, where he'd found a house going into foreclosure with a state-of-the-art 24-track recording studio built in. He dubbed the studio Noise New Jersey, and continued to produce recordings there,[3] including, most famously, Urge Overkill's cover of "Girl, You'll Be a Woman Soon". However, family illness and personal challenges weighed on him during these years, and the pressures of balancing his profile as an artist with his work as a producer and label head proved too heavy. Though it was during this time that he produced some of his greatest recordings, the consistency of his output had begun to suffer. It was at this point in time that Kramer began to look for a way to move the day-to-day management of Shimmy-Disc into what he had hoped in vain would be more able hands.

Shortly following the sale of Shimmy Disc and his recording facility to the Knitting Factory in 1998 (in which he was contracted to play a continuing role in the label as producer and Director of A&R), Kramer sued the Knitting Factory for breach of contract[3] and soon found himself without a creative base for the first time in his professional career. This experience left him emotionally devastated and looking to exit the music business without delay. He did so immediately following his last European tour in November 1999, dubbed "The Last Tour of the Century", which, according to Magnet magazine, was "a creative flop and a financial bust."[3]

Envisioning a complete about-face, Kramer turned to his lifelong passion in film and theater, and in late 2000 he began studying directing under film and stage director Arthur Penn, whom he had met in 1989 when Penn directed Penn & Teller Get Killed. Kramer spent the better part of four years at New York's Actors Studio, where, in addition to learning directing, he did sound design and music for various productions at the Actors Studio Free Theater on 42nd street. This phase of Kramer's career culminated in 2002 when he composed the music for Fortune's Fool, the Tony Award-winning Broadway play directed by Arthur Penn and starring Alan Bates and Frank Langella, both of whom swept the Broadway acting awards for that year. Kramer had just been appointed assistant director on Arthur Penn's next Broadway play (Sly Fox) when his mother Rosalyn was stricken with a debilitating stroke, which drew him to Florida in 2003. She died 16 months later of complications thereof.

After moving to Ft. Lauderdale area, Kramer worked for the James Randi Educational Foundation from 2004 until February, 2006. His main job was to manage the foundation's One Million Dollar Paranormal Challenge with a million dollar prize to anyone who could demonstrate paranormal ability (the money remains unawarded.) Kramer also maintained the foundation's video library and oversaw the digital transfer of over 700 archival VHS tapes to DVD, comprising the most complete document of the life and career of James Randi.[citation needed]

Kramer is associated with the formation of the so-called "slowcore" movement, thanks mainly to his production work for two seminal bands of the era: Low, and Galaxie 500. He continues to produce, mix and master a great variety of artists worldwide.

In 2006, Kramer worked on the LP Exaltation of Larks, a solo release from Dot Allison. The LP was released in September 2007 on Cooking Vinyl in the UK and P-Vine in Japan.

Kramer premiered his composition "Things to Come" in Tokyo in 2007. Hoping to perform the piece annually, he has subsequently performed it in Tel Aviv in 2008, Melbourne in 2009 and Paris in 2010. In January 2008 Kramer toured with Fair; 14 cities in 14 days in Japan with Mike Watt and Samm Bennett in a "dueling bass" trio called Brother's Sister's Daughter.

In 2012 Kramer released The Brill Building on Zorn's Tzadik label, an album of cover songs written in the Brill Building in the early 1960s.