It was built originally in the 13th century as a tower house, and enlarged and fortified in the 14th century.[2][3] At this time the castle was owned by Gerald Widdrington and, although it was still owned by the Widdringtons in the early 14th century, the Swinburns were living in it.[4]

By the 16th century the castle fell into disrepair and ruin and it was attacked by Border reivers.[4] A survey of 1541 reported the roof and floors to be "decayed and gone".[2]

The property was acquired by the Smith family in about 1640,[3] but in 1715 a further survey stated the building to be ruinous.

Significant alterations were carried out for the Smiths between 1816 and 1845, latterly by architect John Dobson to convert the ruin into a substantial mansion.[2] The Crawshaw family came into possession in 1862,[4] and a west wing was added for them by Anthony Salvin in 1876.[2]

Part of the castle served as a hospital during the Second World War.

It is now an oblong tower house and one of the best-preserved hall houses in the north of England. It is a Grade Ilisted building. In 1888 it was acquired by the Cruddas family, and it is currently owned by the Braithwaite family.