Born in Philadelphia, Klugman began acting after he was discharged from the Army in 1945. He started on the stage in a Broadway revival of Golden Boy in 1952 and eventually scored his first film role in 1957 as Juror No. 5 in the seminal Sidney Lumet film 12 Angry Men. From there, Klugman went on to make hundreds of appearances on live television anthology series in the 1950s, including Playhouse 90 and Studio One. He also made four appearances on The Twilight Zone.

Klugman's most famous role was as Oscar Madison, the sloppy roommate of Tony Randall's uptight Felix Unger on the ABC sitcom The Odd Couple, which ran from 1970 to 1975. Klugman, who had previously replaced Walther Matthau on the Broadway production of The Odd Couple, won two Emmys and a Golden Globe for his performance.

A year after The Odd Couple ended, Klugman starred on the NBC forensic drama Quincy, M.E. as the title character, a tough medical examiner who solved crimes. The show aired until 1983.

Diagnosed with throat cancer in 1974, Klugman lost his voice in 1989, but learned to talk again. He continued acting until 2010, making guest appearances on numerous TV shows, including The Love Boat and Third Watch.

Klugman is survived by his wife Peggy Crosby, whom he married in 2008, and his two sons, Adam and David, from his first marriage to Brett Somers.