In 1976, Rust left the Alouettes to become an assistant with Vermeil's Philadelphia Eagles. He served as Linebackers coach for two seasons before leaving to take the Defensive Coordinator position with Levy and the Kansas City Chiefs. After Levy's firing in 1982, Rust became defensive coordinator of the New England Patriots. Head coach Ron Meyer fired Rust midway through the 1984 season, but he was later reinstated (with Meyer himself fired). Rust and the Patriots went to Super Bowl XX (under head coach Raymond Berry) in 1986, but he left the team after the 1987 season. He returned to the Chiefs for the 1988 season, and moved again to the Pittsburgh Steelers in 1989.

The Patriots hired Rust as head coach in 1990, but fired him after a 1–15 season, the worst showing in team history.

Rust was named the head coach of the Montreal Alouettes of the Canadian Football League in 2000, and was fired during the 2001 season after a six-game losing streak. Rust spent 2002 as the defensive quality control coach of the New York Giants. He returned to the CFL in 2005, taking the coordinator position with the Winnipeg Blue Bombers; he left abruptly halfway through the season. He became defensive coordinator of the Ottawa Renegades in February 2006; the team suspended operations before the season began.