I have recently bought the game and loved it, but the world map now worries me.

Of all the spots in the map, are they all part of the storyline in some way, does it go through of all them? Can I go to places not directed by the storyline with no fear of breaking it? Or even, will I miss anything if I don't go to place X before the storyline directs me to it?

This is the sort of issue in kind-of-open RPGs that walls my progress, in that I start fearing if I'm missing something by going on, or if by deviating from the storyline I'll break it.