It's been a while since I played it, but did you fight a boss guy in the castle? If i remembered right you fight a boss, then it unlocked a door. Took me like 10min to find my way out too. I'll boot up my game and see if I'm anywhere near the castle, but I think that I'm on a different island now.

Aight, you get on the roof of the castle and run around to the top left, go down that stairwell. Run through that room to the door on the bottom right which puts you on the outside of the castle on level down from the top. Run to the bottom middle where above you is the large entrance and run straight south. You can't see the door from that view, but you run through the large door that goes under the top level of the castle. Probably a quicker way, like running straight south from the main room of the castle and skipping the stairwell.