The difference between skin color and the other "soft" factors adcoms take into account is that discrimination on the basis of skin color is specifically prohibited by the 14th Amendment. Such discrimination is also presumably immoral, at least to those who oppose racial discrimination generally.

If you want to take educational disadvantages into account on the basis simple fairness, feel free. There's no real reason that needs to be tied to race, though.