Avery began his career in the 1980s with appearances in television series such as Showtime sitcom Brothers as Bubba Dean, Amen, FM and L.A. Law. In the 1990s, he achieved prominence for his role as Phillip Banks (Uncle Phil) in The Fresh Prince of Bel-Air, a character that was ranked #34 in TV Guide's "50 Greatest TV Dads of All Time." He played the lead role of Alonzo Sparks in the comedy series Sparks that lasted for two seasons. Other notable roles in television included Dr. Crippen in The Closer, Charles Haysbert in The Division and Michael Kelso's commanding officer at the police academy late in the series run of That '70s Show.

On December 31, 2013, Avery died aged 68, from complications following open heart surgery at Glendale Memorial Medical Center.[3][8][9][10]

Janet Hubert, who portrayed his original on-screen wife Vivian on The Fresh Prince of Bel-Air, said after his death, "RIP James, all the world is a stage, and we are all merely players in this production called LIFE."[11]Will Smith commented on Avery's death by saying, "Some of my greatest lessons in Acting, Living and being a respectable human being came through James Avery. Every young man needs an Uncle Phil. Rest in Peace."[12]Joseph Marcell called Avery a "gentle giant".[9]

Avery's remains were later cremated and scattered near the Pacific Ocean.