However, this contradicts my original idea to make a modular system in which if one part fails, the whole system does not have to be replaced.

There's no contradiction: You can still have a separate summing module AND a control module. But just use a digital summing module instead of an analog one. Only a digital summing module can provide the helpful plausibility checks to detect all kind of sensor failures.