In 1298, during the Battle of Falkirk, Alexander de Welles, Master of Torphichen Preceptory, was killed. Based on the heraldic evidence there is very little doubt that Alexander de Welles was a member of the Lincolnshire Welle(s) family. Also at Falkirk was Adam de Welle(s) of Lincolnshire and of the Castle of Yester in Lothian, to whom the English King Edward I,during his occupation of Scotland, gave various properties.[2]

In 1357, there being no male line left of the Giffards, Joanna, a daughter of the last Sir Hugo de Giffard, married Sir William Hay of Locherwart (now known as Borthwick, Midlothian), the Sheriff of Peebles. He was invested with the barony and lands of Yester through his wife. The barony has stayed with the Hay family ever since.

David Hay of Yester was in 1487 created a Lord of Parliament, as the first Lord Hay of Yester. In 1513 during the disastrous Battle of Flodden, John, second Lord Hay, was killed along with a great proportion of the country's fighting men. Later that century, in 1547 during the period known as the Rough Wooing, Yester was attacked by an English force, and was defended stoutly by John, 4th Lord Hay. Lord Hay was to be captured later that year at the retreat from the Battle of Pinkie and was held in the Tower of London for three years.

1557 saw the death of the 4th Lord Hay. His son, also John, abandoned the castle and moved into a new tower house on the site of the present day mansion of Yester House. In 1646 the 8th Lord was created Marquess of Tweeddale. The castle gradually fell into disrepair, and by the late 17th century was in a very parlous state, the stones having been much quarried for building material. Although the castle almost disappeared completely, Sir Hugo's original Goblin Ha' was tenanted by the Marquess' falconer until 1737. Yester House with its Adam interiors was sold in 1972 to Italian/American operatic composer Gian-Carlo Menotti then left to his son, Chip, who sold it in 2013.

Sir Hugo de Giffard was known as the 'Wizard of Yester', and was considered to be a powerful warlock and necromancer. It is in the undercroft of the castle that he was thought to practise his sorcery. 14th century chronicler John of Fordun mentions the large cavern in Yester Castle, thought locally to have been formed by magical artifice. Legend supposed that Hugo was able, via a pact with the Devil, to raise a magical army to his aid, and use them to carry out his will. It is this army of hobgoblins that was considered the builders of Yester Castle.

When his daughter Margaret was to marry, Sir Hugo gave her and her husband-to-be, Broun of Colstoun, a hand picked pear with the proviso that should anything happen to this fruit it would spell disaster for the Broun family. The pear was encased in a silver box and kept safe; the Brouns prospered.

A few hundred years later however, in 1692, on her wedding night, the fiancée of Sir George Broun, a Baronet of Nova Scotia and inheritor of Colstoun estate, decided to remove the pear from its casket. The fruit looked as good as when it was picked, and she could not resist taking a bite. Misfortune quickly followed. Sir George Broun amassed enormous gambling debts and was forced to sell the estate to his brother Robert. Robert with his two sons was soon after killed, en route to Edinburgh; they were swept away by a flash flood caused by the River Tyne bursting its banks. In destitution, Sir George died in 1718 without a male heir. It was said that after the pear was tasted it turned as hard as rock, and with its bitemark in evidence, it is still at Colstoun House to this day.

"A clerk could tell what years have flown since Alexander filled our throne third monarch of that warlike name, and eke the time when here he came to seek Sir Hugo, then our lord: A braver never drew a sword, a wiser never, at the hour of midnight, spoke the word of power; the same, that ancient records call the founder of Goblin hall"----"Lord Gifford deep beneath the ground heard Alexander's bugle sound, and tarried not his garb to change, but, in his wizard habit strange, came forth, —, a quaint and fearful sight: His mantle lined with fox-skins white; His high and wrinkled forehead bore a pointed cap, such as of yore Pharaoh's Magi wore; His shoes were marked with cross and spell, upon his breast a pentacle"----"and in his hand a naked sword without a guard".Marmion, Canto III, Edinburgh 1808.