In 1967 he married Lucy Caroe, historical geographer. She was the daughter of the architect Alban Caroe and her grandfathers were W. D. Caroe and William Bragg. They had no children, and on his death in 1995, the title Baron Adrian became extinct.