Georgiana Cavendish became a socialite who gathered around her a large circle of literary and political friends. She was painted by Thomas Gainsborough and Joshua Reynolds; the Gainsborough painting was disposed of by the 5th Duke and was recovered only much later, after many vicissitudes.

The 5th Duke also had a daughter—Charlotte, given the surname William—by his mistress, Charlotte Spencer, the daughter of an indigent clergyman. His first child was born shortly after his marriage to Lady Georgiana Spencer (no relation to his mistress). Charlotte was later married off suitably.