Agree completely. Yellow jackets and hornets are the only bee/wasps
that I will attempt to exterminate, and those only if nesting near
people or where I mow. Solitary bees are great pollinators. Bumblebees
nest in the ground, too, but they are communal, frequently utilizing
abandoned mouse burrows. Solitary wasps prey on the "bad guys" in your
garden. Please try to coexist with them!

If they're black, or definitely not yellow jackets, leave them since
they're beneficial. As you say, they're not aggressive. Most "ground
bees" are solitary nesters and destroy Gypsy Moth, Broccoli worms
and other caterpillars.

If they're Yellow Jackets, there will be one central hole. It can be
treated with a long-reach spray in the evening, or dusted with Sevin
or another insecticide (also at night so you don't tick them off and
get stung) .