In my very limited experince, I see more canted FSBs than any other problem. As you stated it can be anything from the barrel trunion crooked with receiver shell, rear sight base askew, FSB askew, and on some cheapo kits I have seen a bent barrel. I can be all of the above also I suppose.

Take off the upper hand guard, and look it over with the old MOD I eyeball, you should be able to see the misaligned parts.

FSBs can be fixed by driving out the pins, straightening the sight base, and reassembling with over sized pins and redrilling, or peening our welding up the former slots for the pins.

If the rifle shoots to POA, I personally would not worry about it one bit. Its not like you are going to dope in windage with the AK sights.