Their older brother King Edward IV was imprisoned at Middleham Castle for a short while, having been captured by Warwick in 1469, when the latter led an uprising against him. Following Warwick's death at the Battle of Barnet in 1471 and Edward's restoration to power, Richard married Anne, the younger daughter of Warwick, and the couple made Middleham their home. Richard acquired Middleham Castle through his marriage to Anne, he was said to be particularly fond of the castle, it became his principal base in northern England, and many of the improvements to the castle are thought to date from Richard's time there. Their only son, Edward of Middleham was born at castle in around 1473. His birth is reputed to have taken place in the round tower at the south-west corner of the curtain wall, traditionally known as the Prince's Tower. Prince Edward is also said to have died there on 9th April, 1484 of what could possibly have been tuberculosis.

The origins of Middleham Castle trace back to the eleventh century. Following the Norman Conquest, the land around Middleham was granted to Alan Rufus, a nephew of William the Conqueror, in 1069. He built a wooden motte-and-bailey castle on William's Hill, which was situated a short distance to the south-west of where the present castle stands. The early castle was abandoned in the twelfth century, when Robert Fitzrandolph (grandson of Alan's brother Ribald) built of a new castle with a huge stone keep in the reign of Henry II, the keep was later surrounded by a curtain wall.

The keep, one of the largest in England, towered to a height of some 66 feet (20m) and boasted twelve foot thick walls and three floors . It contained a great chamber, large kitchen, chapel, dovecot, cellars and the living rooms of the lords of Middleham. The Great Hall was the heart of the castle, where entertaining and feasting took place and on some occasions, it was utilised as the local courtroom. At either end of the keep's vaulted basement stood two wells, which still survive to the present day. The original entrance to the castle was reached via a gatehouse on the eastern side of the castle, across a wooden bridge, or possibly even a drawbridge, over a moat. The present day entrance is through the Northern gatehouse, known as the 'Neville Gate'.

Middleham passed through marriage to the Neville family from 1270, and it is thought likely that the stone curtain wall was added by Ralph, 1st Lord Neville, along with corner towers and the chapel on the eastern side of the keep.

Ralph Neville, 4th Lord Neville and 1st Earl of Westmorland became the father of a total of 23 children from his two wives and to provide accommodation for his extremely large family he rebuilt the south and west sections of the castle making them into two storey structures. He was also responsible for increasing the height of the towers and transformed the north east tower into a gatehouse. King Henry IV is known to stayed there in 1410.

Richard Neville, Earl of Salisbury, who was Ralph's eldest son by his second marriage, inherited Middleham in 1440 and made the castle his chief residence. He added the north section of the castle and rebuilt the upper section of the gatehouse. Salisbury supported the cause of his brother-in-law, Richard Duke of York during the Wars of the Roses and was beheaded the day after the Battle of Wakefield on 31st December 1460, his head being gruesomely displayed over Micklegate Bar at York.

Middleham Castle, once known as the Windsor of the North, became home to some of the most powerful lords in England in the fifteenth century, including the mighty Richard Neville, Earl of Warwick, the son of the Earl of Salisbury. Warwick inherited the castle on his father's death in 1460. He played a major role in the Wars of the Roses, during his time of occupation, the castle became a major army garrison.

Following the death of Richard III at the Battle of Bosworth in 1485 Middleham Castle became the property of Henry VII. No longer used as a royal residence, the castle, like Richard III's reputation, fell into ruin under the new Tudor dynasty. In 1604, the first Stuart King, James I and VI granted the castle to Sir Henry Linley.

Linley carried out repairs and lived at Middleham until his death in 1610, when the castle was inherited by his daughter Jane. In 1613 Jane Linley married Edward, 2nd Viscount Loftus, who occupied it until 1644.
During the Civil War it was to be used as a prison. After the war Parliament ordered the destruction of the east range wall as well as most of the wall-walks, leaving the castle the shell seen today.

In 1662 Middleham was sold to Edward Wood, and remained with his descendants until 1889, when it was purchased by Samuel Cunliffe-Lister, 1st Lord Masham. Since 1925 it has been in the care of English Heritage Castle and is open to the public. The remains reveal only a glimpse of Middleham's former lavish grandeur. A modern wooden staircase provides access to the upper level and the top of the keep which offers superb views of the surrounding North Yorkshire countryside.

An exhibition about the notable personalities from the Castle's past includes a replica of the Middleham Jewel, a fifteenth century gold diamond-shaped pendant with a blue sapphire at the top which was found on a bridle path near the Castle in 1985, by a metal detectorist, Ted Seaton. One side of the diamond shaped pendant is engraved with a representation of the Trinity and the other with a Latin inscription indicating that the pendant is a charm against epilepsy. No one knows who the jewel's owner was, but it has been dated back to the fifteenth century to around the lifetime of Middleham Castle's most famous occupant, Richard III. A livery badge, for pinning to the chest or a hat, in gilded copper high relief, with Richard III's emblem of a white boar, has also been found in the vicinity of the castle, it was probably worn by one of his household when he was Duke of Gloucester.