Particularly early War, all there was on either side was contract wagoners. The wagons of a regiment (battalion) were organized as a company under a "Wagonmaster" (six in a Union Infantry Regiment four in a Confederate). Neither side issued uniforms to the wagoners in the beginning. I try to have some kind of patterned (striped or plaid) trousers with a civilian roundabout or sack coat so no one takes me for a soldier. Russell, Waddell and Majors provided thousands of wagons to the Western Armies throughout the War, and all their drivers were civilian. Look at William Henry Jackson's artwork for an idea of what these looked like.