Work on the estate began in 1903, when Phipps promised his British fiancee Margarita he would build her a home in the United States that resembled her family estate in Battle Abbey. The estate was ready for Phipps, his wife and their children in 1906.