These are some high-level thoughts about game balance. I don't expect to be able to implement these ideas for a while, but I want to share them so we have a common frame of reference. Feedback and ideas welcome.

ECONOMY
There should be multiple, viable ways to create a strong economy, and thus a strong military. I can imagine the following paths to power:

1. Lots of worlds, large total population, but low population per world.
2. Fewer worlds, but high population per world.
3. Fewer worlds, but high productivity per world.
4. Fewer worlds, but high trade efficiency in empire.
5. Fewer worlds, but high tech military.

Right now, only the first path is really viable. In order to make the others viable, we need to add appropriate trade-offs so that you can't optimize for everything. We never want an empire that has lots of worlds, high productivity and high tech military.

Essentially, each of the 5 paths above is a way of specializing. We need to make sure that you can only specialize on one of those 5 paths at a time. My solution to that is doctrines. Each of the path should come with a penalty that can only be overcome by the proper doctrine. And since an empire can only have one doctrine, one is forced to choose a path.

Here are my ideas for trade-offs:

PATH 1: Lots of worlds -> Law & Order doctrine
Having lots of worlds increases the chance of secession by ambitious sector capitals. The Law & Order doctrine counters this.

PATH 2: High population -> Faith & Destiny doctrine
Clone banks allow you to increase your population beyond natural limits, but most people resist this path. The Faith & Destiny doctrine allows your empire to accept a clone population.

PATH 3: High productivity -> Industry & Growth doctrine
Automation and efficiency improvements allow you to produce more given the same population. But this can lead to environmental degradation and industrial accidents. The Industry & Growth doctrine counters many of these downsides.

PATH 4: Trade efficiency -> Trade & Enterprise doctrine
High density trade networks allow you to effortlessly move resources to where they are needed. But this invites piracy and fraud, which can sap all your gains. The Trade & Enterprise doctrine counters these risks.

Note that there isn't a hard limit in any of the cases above. For example, with the Law & Order doctrine you can still have some high tech shipyards. But as the number of high tech worlds increases, the chances of disaster increases too (and the disasters will cause your worlds to regress).

So doctrines won't affect what units you build on sector capitals any more, but rather prevent nasty things from happening to your empire. Thus an empire is limited in some ways but can benefit in others. Sounds like a good way to rebalance doctrines.

I was also wondering if players would get the carrot as well as the stick. Instead of merely preventing disastrous outcomes, could doctrines also come with unique advantages not available to other doctrines?

Thus could tie in nicely with that other old thread in Transcendence about sovereign alignments (Republic, Federation, and their corrupted counterparts of Collective, Empire etc etc) I'll link to it later as I'm on mobile...

So doctrines won't affect what units you build on sector capitals any more, but rather prevent nasty things from happening to your empire. Thus an empire is limited in some ways but can benefit in others. Sounds like a good way to rebalance doctrines.

I was also wondering if players would get the carrot as well as the stick. Instead of merely preventing disastrous outcomes, could doctrines also come with unique advantages not available to other doctrines?

Thus could tie in nicely with that other old thread in Transcendence about sovereign alignments (Republic, Federation, and their corrupted counterparts of Collective, Empire etc etc) I'll link to it later as I'm on mobile...

No, I didn't mean to imply that you wouldn't get new unit types or "carrots" from a doctrine. I think you should get both also. I think the imperial guards for Law & Order are a good idea, and I can imagine having special units for Fire & Movement and Strength & Honor too.

So yes, doctrines should also have advantages not available to other doctrines. It's all about specializing.

EDIT: I didn't mention Fire & Movement and Strength & Honor in the original post because I haven't quite figure out their balancing mechanic. Maybe they just get special units. Or maybe there is some downside to specializing in jumpships/starships. Not sure yet.

It seems like you are envisioning that a lot of game systems will be limited in effectiveness by increasing penalties as empires' complexity increases in various ways (larger size, more integrated trade, higher population, etc.)

Are you envisioning achieving this through:

quantified penalties that increase (e.g. if you have 3 trade routes to a non-hubworld they all operate normally but if you add a 4th 10% more materials must be exported on each route due to losses, if you add a 5th 25% more must be exported, etc.)

Random disasters that increase in frequency as complexity increases (e.g. capital secessions right now, which are extremely disruptive)

Hard caps that can only be exceeded under certain doctrines (e.g. you can only get 11 million on a TL8 planet with maximum arcologies, but F&D has higher intrinsic population caps for any given arcology level

Bonuses that are specific to certain doctrines

Something else?

Of these approaches, I think random disasters need to be treated with caution since in the current form that the game takes people do not play all the time and may believe that they have been unfairly punished if they return to their empire in chaos that resulted from a semi-random in-game event rather than from other players' actions. Maybe random disasters as a game mechanic could be restricted to high-risk high-reward doctrines, like the Science & Technology doctrine you've described.

Another thing to consider is logical transitions between doctrines. Players switch doctrines, maybe several times during a game. You don't want situations where a doctrine change results in catastrophic decline immediately, which can happen right now when leaving the T&E doctrine. If a T&E player is using capitals as hubs, all the exporting trade routes instantly break when you switch to a different doctrine (the workaround is to tediously double all the trade routes beforehand one by one, and then to cancel the routes to the capital once the new routes are up and running to prevent sudden shortages). I think that when players switch between most doctrines they should be able to expect that their empire will continue to operate, albeit at reduced capacity, until the player can tweaks it to be more effective. Practically, this means no instantaneous resource shortages as a result of doctrine changes, no sudden civil wars as a result of populations that are too high, no completely broken trade routes, etc. For T&E, this would mean that capitals should probably not be trade hubs at all, and the advantages should come from something else (for example, trade efficiency penalties that are lower for T&E empires).

There could be doctrines that are risky or onerous to get in or out of, but this should be clearly spelled out in their descriptions or it should not even be possible to switch to them if certain conditions are not met.

Another possibility is having certain doctrines be "ideologically opposed" and designed in such a way that going from one to the other is extra disruptive. This would make sense if strategies arise that can exploit rapid, regular transitions between different doctrines (which may arise if future doctrines are designed in a way that provides special bonuses). For example, in Transcendence a Federation sovereign has no aspects in common with an Seeker sovereign because they have fundamentally different beliefs. If one of them is "corrupted" (Empire or Perversion) there is also an altruism/selfishness division in place. In Anacreon, maybe a Federation might benefit from cultural exchange via stationing infantry on every world that gives a production bonus, while a Seeker would get a stability bonus from not having troops stationed on planets (no coercive limitation on individual creativity). Switching between these two doctrines would be harder than normal since you would both lose a bonus and gain a penalty. These are just examples and I don't think either would be particularly interesting as actual doctrines, but it's an demonstration of how such a system might work.

I think all of your suggestions for penalizing complexity are reasonable. I think we'll end up using all of them in different combinations, depending on the doctrine.

You're absolutely right about random disasters being demotivating. We'll have to use them cautiously. At minimum, it should take a long time (days to weeks real-time) for even the worst disasters to destroy an empire. Revolution is a random disaster today and it's been toned down since the beginning to give players a chance to react. We'll probably need to continue to balance it. And as you said, we should only use these mechanics for the risky doctrines.

We'll end up testing this over time and see what works (and what doesn't).

As for changing doctrines, that's really a good point. I like your suggestion about a grace period in transition (or something like that). That's probably the hardest to implement, but it's probably the most natural for the player. Still, changing doctrines should be a major event, and there should be real costs to doing it. [It's the equivalent of Brexit or even worse.] We may even prevent changes in doctrine under certain conditions (e.g., if you've recently changed).

I hadn't thought of different affinities between doctrines, but I think that makes sense. This might also factor into bonuses/penalties for attacking empires of different/same doctrines.