X20 vs X100s @23mm

The sensor difference will probably be the biggest issue, comparing to the X20. I'm not hugely familiar with the X20's lens, but I'd imagine the X100s' lens is also likely to be better since it's a prime compared to a zoom. Between the X-E1 and the X100s, the 23mm f/1.4 will obviously be a stop faster, but the combo will also be larger and heavier. Depending on the light levels in question, having an OVF may also be handy, since EVFs can run into some difficulties in low light–lag, noise, etc.