Dally Castle was a rectangular, early 13th century defensible stone hall house, probably founded in his manor of Chirdon by David Linsey. Standing on a ridge within a curve of the Chirden Burn, the two storey 'house with remarkably thick walls in the form of a tower', is defended to the north-west by a substantial ditch. In the centre of each gable wall is a fishtail arrow slit and along the other walls are three more regularly spaced slits. The manor was mentioned in 1237, confiscated by the Crown in 1289 and granted to John de Swinburne in 1296. In the late 13th and early 14th century the hall with its columned basement, was probably altered into a three storey tower house, flanked by square towers on three angles. When held by the Crown in 1326 the manor had been burnt by the Scots but by 1604 it was occupied by the Dodds family. A roofless ruin in the 18th century, most of its masonry had been taken by the early 19th century. Adjacent to the east are the foundations of a probable chapel and further east are the slight foundations of a small square building. A mile north-east is Tarset Castle and a mile east is Snabdaugh.

Dally Castle is located south-west of Birks, by the Chirden Burn. 4 miles west of Bellingham, off the B6320.