Ellis died childless in February 1802, aged 88, and was succeeded in the barony according to the special remainder by his great-nephew, Henry Welbore Agar, 2nd Viscount Clifden, who assumed the surname of Ellis two years later.