Packer inherited his media interests on his father's death in 1934. In 1936, he joined with Ted Theodore's Sydney Newspapers and Associated Newspapers to form Australian Consolidated Press.[3] He was chairman of ACP from 1936 until 1974.

When television was introduced to Australia in 1956, Packer, along with the other major newspaper publishers (Fairfax, HWT and David Syme), became a significant television network shareholder under the federal government's "dual formula", which allowed each capital city to have two commercial networks and one ABC.[3] He launched the first Australian station to broadcast a regular schedule, TCN in Sydney, which became the nucleus of the Nine Network.

Packer was a keen yachtsman, boxer, golfer and polo player. He was on the Australian Jockey Club's committee for 12 years and won the Caulfield Cup with his horse, Columnist. He was also chairman of a syndicate that built the yachts Gretel and Gretel II to challenge for the America's Cup in 1962 and 1970.

In 1992, journalist Max Walsh told the House of Representatives Select Committee on the Print Media that Frank Packer had exerted undue newsroom influence. "Sir Frank was knee-deep in [the] editorial policy of the Telegraph", Walsh said.[4]

Frank Packer was married to Gretel Joyce Bullmore (1907–1960) on 24 July 1934 at All Saints Anglican Church, Woollahra. He had two sons, Clyde and Kerry, with his first wife, Gretel. Gretel Packer died in 1960.

Packer married for the second time in June 1964 to Florence Adeline Vincent (née Porges) in London. She died in 2012.[5]

On 1 May 1974, Sir Frank Packer died of heart failure, leaving an estate valued at $100 million. On his death he passed his empire to Kerry, as he had fallen out with his elder son Clyde Packer in 1972. He was interred at the Packer family mausoleum at South Head Cemetery.

In the 1984 television miniseries Bodyline, Frank Packer, as employer of Donald Bradman, released him from a writing contract so he could play in the 1932-1933 Ashes; he was portrayed by Brian McDermott.