I'm finding Server 2016 Standard easiest to strip down and still have it remain functional (I think I have 35 processes on startup, with everything I want working), as well as being quite snappy because of this, but the build (1607) is too old to have support for the retpoline patches, so it will only match I/O performance of newer builds if the spectre mitigations are disabled.

Server 2019 looks decent too I wonder if it can be stripped down that far. I have the ISO just never installed it.