Depends entirely on the speed of the USB port (and bus), in addition to the speed and quality of the USB key itself, but it certainly can be done.

The typical enterprise way of doing what you're attempting is to buy a server with a micro-SD slot, load up the hypervisor onto a high-speed micro-SD card, and run the VMs from locally-attached storage or a SAN, but there's no reason that you can't use a USB key in lieu of a micro-SD slot.

Just make sure you don't plug the USB key into a USB 1.0 port, or your performance will suffer. (Likewise, don't buy the low-speed bargain bin USB keys you can pick up for ~$1/GB, or you won't get the transfer speeds you need, even when plugged into a USB 2 or 3 port.)