He was the son of farmer Thomas Carling, who immigrated from Etton in Yorkshire, England to Canada in 1818. In 1839, the family moved to London, where Thomas founded the Carling Brewery in 1843, using a recipe from his native Yorkshire. In 1849, the brewery was turned over to John and his brother William.

After losing the 1891 election to Charles Hyman, he was appointed to the Senate by Prime MinisterJohn A. Macdonald. However, the election was disputed and declared void, and Carling resigned from the Senate in order to run in a by-election in 1892, which he won. He served in the House of Commons until just before the 1896 election, when he resigned and was re-appointed to the Senate.

Meanwhile, Carling remained active in London affairs, using his positions in the federal government to influence politics and business. In 1875, John and his brother William built a new Carling Brewery, and an even larger one was built when this burned down in 1879. The brewery was one of the largest in Canada and rivalled the production of fellow London brewery Labatt.

He also ensured that the Great Western Railway, the London and Port Stanley Railway, and the London, Huron and Bruce Railway passed through the city. Due to his influence, the Grand Trunk Railway began to manufacture their cars in London. In 1878, he established a water commission to provide a water supply to the city. He also established the Ontario Hospital for the Insane in London, and in 1885 he provided the land on which Wolseley Barracks was established, now the Home Station of The Royal Canadian Regiment and the garrison of the Regiment's 4th Battalion. Carling also facilitated the establishment of Victoria Park.

He was knighted in 1893, and served in the Senate until his death in 1911.

In 1927, Carling Brewery sponsored a trans-atlantic flight from London (Canada) to London (UK). The plane was named the Sir John Carling, but both it and its pilots, Terence Tully and James Medcalf, disappeared over the Atlantic Ocean.

Today there is an arena in London named after him. The town of Port Carling, Ontario is also named in his honour as well as the agricultural buildings in Ottawa, Ontario. Carling Avenue in Ottawa is named after Sir John Carling as well, as are the neighbourhoods of Carlington and Carlingwood.