According to legend, the castle was raised by the first Storm King, Durran Godsgrief, who married the goddess Elenei. Because this marriage doomed her to die a mortal's death, her parents, the god of the sea and goddess of the wind, raised a great storm that shattered the castle and drowned the wedding party, save for Durran, who was protected by Elenei.

When the storm had ended, Durran rebuilt his castle, but again the gods destroyed it. Refusing to give up and move inland, he built five more castles in succession; four were battered down by the storms. The seventh was finally built strong enough to withstand the storms blowing in from Shipbreaker Bay; the curtain wall that faces the sea is eighty feet thick. There are several theories about the building of the final castle; one myth is that a child who would become Brandon the Builder told Durran how to build it, another that the children of the forest helped raise it with magic.

During the War of the Usurper, the castle was under seige by Mace Tyrell. Stannis, holding the castle in the name of his brother Robert, refused to surrender to him, preferring instead to starve his men. Davos Seaworth smuggled food into the castle, helping Stannis to fight the Tyrell army off and keep his hold on Storm's End.