Between 1874 and 1880 Adam was a Liberal whip. When the Liberals returned to power under Gladstone in May 1880, he was once again made First Commissioner of Works.[6] In December 1880 he was appointed Governor of Madras, which he remained until his death in May of the following year.

Adam married Emily, daughter of General Sir William Wyllie, in 1856. They had several children. Adam died in May 1881, aged 57,[1] in Ooty, British India, where he was also buried.[8][9] His wife Emily was awarded the Order of the Crown of India in October 1881.[10] In 1882 their eldest son Charles Adam was created a baronet, of Blair Adam in the County of Kinross, in honour of his father (see Adam baronets) and the same year Emily Adam was granted the precedence of a baronet's wife. She died in November 1906.[1]