While at Jackson State, Gray played football for the Tigers from 1970-1973 as an offensive tackle. At the end of his senior season, he received several honors, including a First-team All-SWAC selection and team MVP. He was also named to the All-American teams of both Playboy and Ebony Magazines, as well as the Pittsburgh Courier. Gray was nicknamed "Big Dog" in college due to his size (295 lbs.)[3]

Gray was selected in the third round of the 1973 NFL Draft by the Miami Dolphins. However, due to their depth at offensive line, he never played for the team. He was cut before the start of the 1973 NFL season and claimed on waivers by the New England Patriots. Gray was recommended to Chuck Fairbanks, who had just taken over as head coach of the Patriots, by an Eastern Airlines pilot named Bruce Kostamo who had played for Fairbanks in high school.[5] Gray played in the final nine games of the '73 season, starting eight at left tackle. He continued to develop during the 1974 and 1975 seasons and by 1976 he was viewed as one of the best left tackles in the game. Gray was not only strong at run-blocking, but in pass-protection, as well. He was a major factor in quarterback Steve Grogan being sacked only 14 times in 1977.[2] Along with guard John Hannah, the two formed what was generally considered the best guard/tackle tandem in the NFL.[2] Gray and Hannah also combined with tight end Russ Francis to form one of the strongest left-side trios in the league. At the end of the 1976 NFL season, Gray was selected to play in the Pro Bowl for the first time. Due in no small part to the efforts of Gray and Hannah, the Patriots enjoyed one of the finest seasons in franchise history in '76, finishing with an 11-3 record. This earned them the AFC Wild card spot and a rematch with the Oakland Raiders. The two teams had met at Schaefer Stadium in week four of the regular season and the Patriots had handed the Raiders their only loss, a 48-17 drubbing. The rematch in Oakland would not go as well and the team's hopes of winning it all were ended by a close and controversial loss to the eventual Super Bowl champions in the 1976 AFC Divisional playoff round.

“Gray had everything-the best footwork, the best balance, the greatest strength.”[6]

Despite the tough loss, the Patriots were seen by observers as one of the main Super Bowl contenders going into the 1977 NFL season.[7] Unfortunately for the team, trouble was looming. At the convincing of agent Howard Slusher,[8] Gray and Hannah walked out on the Patriots before the team's final preseason game in an effort to get their contracts renegotiated. The two were unsuccessful in getting new deals and returned before the week 4 regular season game against the Seattle Seahawks. In the meantime, the Patriots had gotten off to a 1-2 start that included losses to a Cleveland Browns team that finished 6-8 and a New York Jets team that finished 3-11. The team won eight of the eleven games that Gray and Hannah played and started in, but their 9-5 overall record was not enough to make it into the 1977 AFC playoff field. The team rebounded from the tragic injury suffered by wide receiver Darryl Stingley during a pre-season game against the Raiders to once again emerge as contenders during the 1978 NFL season. The emergence of second-year pro Stanley Morgan and the acquisition of All-ProHarold Jackson to go along with Russ Francis made their passing attack more potent. Running the ball was still first priority for the team, however, and the Patriots finished that season with a team-total of 3,165 yards, an NFL record that still stands. Gray was rewarded for his role in this by being named to the All-Pro First-teams' of the AP, Pro Football Weekly and the PFWA and was also selected to play in the Pro Bowl for the second time. Despite all this, controversy would once again derail the Patriots championship hopes. Late in the regular season, with the team 11-4 and the AFC East divisional winner, Chuck Fairbanks announced that he had decided to leave the Patriots after the conclusion of the 1978 season to become head coach at the University of Colorado. The news angered some players and particularly enraged owner Billy Sullivan.[9] Sullivan suspended Fairbanks for breach of contract and the team lost their regular season finale against the Dolphins, 23-3. Fairbanks was reinstated for the AFC Divisional playoff game against the Houston Oilers, but the damage was done and the Patriots were beaten by the Oilers 31-14 in the first postseason game played at Schaefer Stadium.

Shortly before the start of the 1979 NFL season, Gray was traded to the Oilers for first and sixth-round draft picks. Patriots management saw the deal as a money-saving move. Upon hearing the news, John Hannah said, "We just traded away our Super Bowl."[10]Ron Erhardt, who had replaced Chuck Fairbanks as the team's head coach, also admitted that he was against the trade. Gray's one-time opponent and now Oiler teammate, Elvin Bethea, would later say that the trade for Gray was one of the happiest days of his life.[11] He also stated that Gray was at least in the top three of offensive linemen that he had to face during his career. The Oilers had a strong 1979 campaign, finishing the regular season 11-5 and winning their first two postseason games. However, as with the previous season, they were denied a trip to the Super Bowl by their chief rival (and defending champion), the Pittsburgh Steelers, in the AFC Championship game. The team again made the playoffs in 1980, but lost to the eventual Super Bowl champion Raiders in the Wild card round. Gray continued to excel during his time with the team, being named 1st Team All-Pro in both 1979 and 1980. He was instrumental in helping Oilers' star running back Earl Campbell rush for NFL-leading totals of 1,679 yards in 1979 and 1,934 yards in 1980. After being named to the AFC Pro Bowl team for the fourth time at the end of the 1981 NFL season, Gray was traded from the Oilers to the New Orleans Saints in exchange for quarterback Archie Manning prior to the start of the 1982 NFL season. While with the Saints, Gray blocked for two-time Pro Bowl running back George Rogers. Gray played two seasons with the Saints before retiring from the NFL after the 1983 season. He is still considered arguably the best offensive tackle to play for the New England Patriots.[12] Gray was named to the Patriots All-1970s Team, but was notably absent from their 35th (1994) and 50th Anniversary (2009) Teams.