Personally I can see Balder being hid away because he might have more knowledge on how to use something like a Tesseract, and to be safe from others trying to force him into doing things, they hid him away.