Henry II took a great interest in the castle and its potential and it was practically reconstructed between 1170 and 1174. This included the building of the flint walls, drawbridges and gatehouses. The castle was further strengthened during the reign of Richard I by his regent, William Longchamp. By this time the castle was governed by the Crown after Robert de Valoignes had died in 1184 leaving no male heirs. However it had been claimed by Robert Fitzwalter, Robert de Valoignes's son-in-law. He seized the castle and installed his own tenants and garrison. Although he was subsequently evicted by King John, he was eventually appointed governor of the castle. His opposition to the Crown continued and in 1211 he fled to France and the castle was confiscated.

Following the death of Angelo Salvo in 1216, a French invasion besieged the castle for a month until the governor, Walter de Godarvil, surrendered. However the country then supported Henry III and by 1217 the French had left. Following this the castle's military role became secondary to its use as a royal residence and in 1299 Edward I gave it to his second wife Margaret.

During the Hundred Years' War the castle was used to detain prisoners of royal and noble rank. These included King David II of Scotland and, in 1359, King John II of France. The following year the castle was granted to Edward's third son, John of Gaunt who spent much time there, using it as his chief country home when not abroad on campaigns. The defences were repaired and strengthened again at this time. When John of Gaunt died in 1399, Richard II seized all the Lancastrian estates, including Hertford Castle, where he installed his new wife, the eight year old Princess Isabella.

When Richard III became king, the castle was granted to one of his greatest supporters, the Duke of Buckingham. After the downfall of Richard III, Henry VII conferred the castle to his wife in 1487, but Henry himself spent little time at the castle. However, his son Henry VIII spent considerable sums turning the castle into a civilian palace, including building the gatehouse, which still stands. Subsequently, Edward VI granted the castle to Princess Mary. During her eventual reign, the castle was used to imprison Protestantmartyrs. Elizabeth I was a frequent visitor to Hertford.

However, from the reign of James I the castle ceased to be a royal residence and it fell into decay. After Charles I came to the throne, the castle was granted to William Cecil, 2nd Earl of Salisbury. Much of the castle was now in ruins having not having been used since Tudor times. The castle was leased to Sir William Harrington of Hertingfordbury Park, who carried out work to restore the buildings. He then assigned his lease to Sir William Cowper, Collector of Customs at the Port of London. From this time the castle remained in the ownership of the Cecil family, who leased it out to a succession of occupiers, many of them successors to William Cowper. Around 1790 the south wing was added to the gatehouse. The windows of the existing gatehouse were all remodelled, and the parapet added with its brick battlements.[2] Around 1800 a new gateway and lodge were built by the Marquis of Downshire.

Between 1805 and 1809, the castle became the home of the East India Company College. In 1822, a general dispensary was established at the castle by the Rev. Thomas Lloyd, who was a prime mover in charitable work in the town.

In 1911, the Hertford Corporation leased the gatehouse of the castle (which was all that then remained) from the Marquess of Salisbury to house its administration. The grounds became a public garden. In the 1930s, the north wing was added to the gatehouse and, subsequently, Lord Salisbury gave what was left of the castle to the town. Hertford Castle now houses the offices of the town council.