In 1623 the manor was granted by the Crown to the Goodwin family, who enlarged the manor house into a mansion. It then passed into the Wharton family, one of whom was made the Duke of Wharton in 1718 for his services to the Crown. He later had all his possessions seized for being a supporter of the Young Pretender (Bonnie Prince Charlie), after which time the house fell into disrepair and has since been demolished.

Philip Wharton, 4th Baron Wharton (1613–1696), soldier, Parliamentarian and art collector, gained the manor of Upper Winchendon through his second wife, Jane Goodwin, only daughter of Arthur Goodwin. He and other family members were painted by Anthony van Dyck.[4]

Samuel Clarke (1626–1701), Nonconformist minister and Biblical scholar, spent 26 years in Upper Winchendon under the auspices of Philip Wharton, after his ejection from the rectory of Grendon Underwood in 1662, and set up on an Independent congregation there.[5]