Engineering always involves compromises. You have to choose what is really important in your scheme.
If you can afford some loss of capacity, there are also ldos which are smaller/ simpler than the switch.