Hanbury-Tracy was a gentleman-architect who was influenced by the work of John Carter of the Society of Antiquaries. As one of the earliest Gothic Revival houses, the building shaped the course of British architectural history in an indirect way: when the Houses of Parliament were to be rebuilt after the fire in 1834, Hanbury-Tracy headed the jury to the competition, and the architect of the winning design, Charles Barry, obviously adapted his entry to the taste exemplified in Toddington.[1]

The last private owner, a Mrs Andrews, died in 1935 and it stood empty until September 1939, when it was purchased by the National Union of Teachers, who had moved out of London to avoid air raids. The NUT staff both lived and worked in the building. Following Dunkirk a tented encampment was erected in the grounds and temporarily occupied by men evacuated from the beaches. They were later followed by units of the British Army. In 1942 the Pioneer Corps built a more permanent hutted encampment, which was occupied by units of the United States Army from October 1942. In August 1943 the NUT moved back to London and the US Army took over the house as well. After the war the Congregation of Christian Brothers rented the property and in 1948 the NUT sold it to them.

In the late 1970s, it was converted into an international co-educational boarding school known as Toddington Manor College. Most of the students were from Asia, pursuing 'O' and 'A' level certification.

In 2004, planning permission to convert it into a hotel was denied after the scheme had attracted considerable local opposition.

In 2005 it was purchased by the artist Damien Hirst who plans to restore it and use it as a family home and a gallery, for both his own works and his collection of works by other artists.[2] Since 2006, Toddington Manor has been encased in what Hirst claims is the world's biggest span of scaffolding.[3][4]