The fourth and final special prosecutor to investigate the Watergate break-in and cover-up, Ruff remained in the public eye. He left his post as a federal prosecutor in 1982 to join the Washington law firm of Covington & Burling. Among his most prominent clients were former Ohio senator John Glenn, whom he represented during the Keating Five hearings. He also represented former Virginia senator Charles S. Robb during a 1993 grand jury investigation of the senator's alleged involvement in the illegal taping of a rival's cellular phone call.

In 1995, then-Washington Mayor Marion Barry tapped Ruff to serve as corporation counsel, the city's top lawyer, and in January 1997 former president Clinton appointed him White House counsel. The new job immediately threw Ruff into the middle of several ongoing controversies, including Whitewater, Democratic Party fundraising and the 1993 firing of employees in the White House travel office. As White House counsel, Ruff defended Clinton during his impeachment trial.