Suspected of Jacobite activity, he was imprisoned twice, but took the oath in 1696 and was admitted to the House of Lords. Yarmouth died heavily in debt on 25 December 1732 at Epsom, Surrey, aged seventy-eight. As his sons, his brothers and their male heirs had predeceased him, his titles became extinct.