You always need a signal ground for reference. Your board will not draw poweroff the USB port as long as you don't connect the USB power pin to your ckt. Yourboard will be much more capable if not drawing 5V and 3V from the USB port.