Hm... That would require a proper USB driver + hostmode implemented in a u-boot module, I suppose.
I know its possible to boot off a USB->ATA adaptor connected to an internal CD drive, at least with an x86-compatible PC.

What is the smallest distribution that boots straight to terminal and still has wide HW support out of box?

I´m thinking that just downloading more than 100 Mb just for flashing purpose is just bit too much.

Gentoo Live CDs aren't much bigger than 100MiB and boot to a shell. You'll have to run dhcpcd or configure the network manually yourself after booting. I've used them on a plethora of desktop and server machines and never had a problem with hardware drivers, although I've never tried to use WiFi from the Live CD; this may present problems.