Traditionally, the Gordons of Clanyard had a castle situated in Glen of the Hole. It was located where there is a very deep hole, from which the glen derives its name. There are no visible remains of the castle or tower-house which is said to have stood almost 120m SSE of High Clanyard farmhouse and to have been built for the Gordons of Clanyard.