Well, castles ARE maybe too strong, since they sost nothing after you build them. Only they cant attack.
The biggest advantege is that they are the only one left alive if your soldiers "die from hunger".

The biggest disadvantege is that they become too weak when knights come.

They're not all that useful against the AI since the
AI is so dumb. Against humans they're essential for
survival in the midgame, until the provinces are big
and the knights come out. Even after that it is
still an effective trench warfare tactic to build
a castle forest. (I do think that castles should
cost $1/turn, though.)