Playing collegiately at Boston College for three years beginning in 1940, Holovak was a starter at running back at under head coaches Frank Leahy and Denny Myers, averaging more than five yards per carry. During his sophomore year, the undefeated Eagles competed in the Sugar Bowl.

Two years later, Holovak served as co-captain of the Eagles' squad that entered the final weeks of the 1942 campaign as the top-ranked college team in the country. However, on November 28 they were upset by arch-rival the College of the Holy Cross, a defeat that canceled a victory party that night at Boston's Cocoanut Grove hotel. That decision proved to be chillingly wise when a horrendous fire swept through the building that night, killing 492 people and injuring hundreds more.

The team regrouped to compete in the 1943 Orange Bowl game, a contest in which he rushed for 158 yards on just 10 carries and scored a record-setting three touchdowns. He earned All-America honors that season and finished fourth in voting for the Heisman Trophy.

On April 8, 1943, the Cleveland Rams, who had earlier announced they were suspending operations for the duration of World War II, selected Holovak in the first round of the NFL Draft. Holovak then enlisted in the United States Navy and attended Officer Candidate School. He went on to command a PT boat that sank nine Japanese ships, and later battling malaria. However, despite his heroics, he was dealt a heavy personal blow when one of his brothers lost his life during the four-year conflict.

Holovak officially signed with the Rams on December 29, 1944, but didn't take the field until 1946, by which time the franchise had relocated to Los Angeles. During his one season with the team, he gained 211 yards and scored three touchdowns.

Traded to the Chicago Bears on January 27, 1947, Holovak would lead the pass-oriented Windy City squad in rushing during each of the next two years, also three touchdowns. Prior to the start of his first year with the Bears, Holovak got his first taste of coaching by assisting with the spring practice of University of San Francisco head coach Edward McKeever.

After his two years in Chicago, Holovak retired on March 1, 1949 to become the freshman football coach at his alma mater. After two seasons in which he compiled an undefeated record, he was promoted to head coach at Boston College on December 30, 1950, following the resignation of Myers.

In his first five years, Holovak compiled a mark of 24–16–5, winning Coach of the Year honors in 1954 from New England football writers. Those efforts were good enough to earn him a new four-year contract on November 22, 1955, but even after four more winning seasons, he was fired on December 3, 1959, after a year in which Eagle fans had subjected him to constant verbal abuse.

Two weeks after his dismissal, Holovak signed as a scout with the fledgling Patriots, then was named an assistant under Lou Saban on March 29, 1960. After a 5–9 record that year and a 2-3 start in 1961, Saban was fired on October 10 and replaced by Holovak. His first game as Patriots coach was a 31-31 tie (the first in Patriots history) with the Houston Oilers. Finishing with a 9–4–1 record, Holovak matched that record the following year, earning him a new three-year contract, with an option for another year.

In 1963, the Patriots reached the postseason for the first time, despite only compiling a 7–6–1 record. They then defeated the Buffalo Bills in an Eastern Conference playoff, but were then pounded in the AFL Championship game, 51–10, by the San Diego Chargers. The next year, Holovak added the duties of general manager to his job description, and after the team improved to 10–3–1, he was named AFL Coach of the Year on December 15, 1964 and received a new five-year contract.

However, the veteran unit slipped to 4–8–2 in 1965, but rebounded into contention the next season with the addition of rookie running back Jim Nance. Needing only a victory in their final game against the New York Jets on December 17 to reach the AFL title contest, the Patriots instead saw their 1966 season end with an 38-28 upset loss. The defeat was softened somewhat when Holovak again won the league's Coach of the Year accolade.

Age finally caught up with the team over the next two years with the Patriots managing only a 7–20–1 record. The end result was that Holovak was fired from both positions on January 7, 1969, ending his Patriots tenure with a mark of 53–47–9. As in his previous stint at Boston College, Holovak saw growing fan discontent, including chants of "Goodbye Mike," play a role in the decision. He would remain the winningest coach in Patriots history until Bill Belichick passed him in 2005.

He would later coach in one game for the New York Jets, replacing Lou Holtz who had resigned before the last game of the 1976 season. His overall career record as a NFL head coach was 53–48–9 for .522 winning percentage.

After first considering an assistant coaching position with the Oakland Raiders, Holovak instead remained a Patriots scout, but shifted gears somewhat by taking a position with the Massachusetts State Racing Commission. However, just two months later, he left that position to become an assistant with the San Francisco 49ers. Personal tragedy, though, soon intervened when his 18-year-old daughter Ann, one of two girls, was killed in an auto accident on the night of her senior prom on June 13.

After two years with the 49ers, Holovak remained in the Bay Area by accepting the post of receivers coach with the Raiders on March 6, 1971. When the team failed to reach the postseason for the first time in five years, Holovak was let go, but signed as a scout with the New York Jets on July 10, 1972. Returning to sidelines the next year as running backs coach, he then moved into the Jets' front office following the 1974 NFL season as Director of Player Personnel.

He remained in that role for two seasons, but when Jets head coach Lou Holtz resigned before the final game of the season, Holovak handled the duties in a 42–3 loss to the Cincinnati Bengals on December 12. He then announced his resignation on May 20, 1977, citing personal reasons.

By 1988, Holovak was being phased out by Oilers' general manager Ladd Herzeg, with his duties limited to scouting. However, after Herzeg left in a contract dispute after the season, Holovak was named general manager on February 2, 1989. He remained in that role until the end of 1993 NFL season, but remained a scout for the franchise, which later shifted to Tennessee, until retiring for good in 1999. In an episode of A Football Life that chronicled the 1993 Oilers team, a team riddled with many off-the-field issues, several Oiler personnel criticized the actions taken by the front office (chiefly Holovak and owner Bud Adams) for not taking direct action on the bitter issues that plagued the team, such as the season long feud between defensive and offensive coordinators Buddy Ryan and Kevin Gilbride that eventually turned physical during a nationally televised game. Despite this, the '93 team finished 12-4 after a 1-4 start before receiving an early exit from the Kansas City Chiefs in the divisional playoffs.