I'm cheap. I scrimp. I buy in bulk.
Farm stands around me sell veggies at about 20% higher than supermarkets, but I never balk at spending the extra for their produce.
As winter has finally come to New England, I start thinking about the first spinach and aspagagus that will soon be coming to the stands. My first handful of local spinach is like the signal that it is spring, it's the opening ceremony. Fresh greens. The spinach I buy from local field is nothing like the pre-washed bags at the Stop and Shop. It's sweet, tender, soft... a true treat. Later, the green beans from the farm stand are truely "snappy". The radishes, carrots are crisp. In fall, the potatos are buttery, the corn is like candy. I love going to the local stands, to pick out what is in season. They can be prepared in the most simple manner and be a treat.
If you wonder whether it's worth it to make that extra stop, to pay a bit more, I say Yes!

i really hate to be a wet-blanket, but chances are quite strong that the much of the produce at a farm stand is not "local" at all. Just more carefully selected for quality & freshness. That stand-in-a-field might be more indicative of how much rents are in empty fields are, than anything else. Try going there real early in the morning and watch where the stuff is coming in from.

There are many farmers who'd LOVE to sell their produce directly, without a middleman.... ask them how much traffic passes by in places where truck farming actually happens. And ask them how much time they have for playing retailer.

Tell me how much produce you think is grown in Massachusetts in 2007? Don't forget to EXCLUDE all crops which were pre-sold for future delivery under contract. Yep, farmers like to have regular income!

The stand I usually go to does carry veggies in the winter which are obviously not local.
But in the growing season, they mark their own produce with a tag that says "Our Own" and I believe them. The stuff is in season, and you can see people picking in the fields. They grow a lot of heritage veggies that you would never see in a supermarket, and the fields are right next to the "stand" which is actually a regular building (in Sudbury, MA which is by no means a low-rent district).
And the taste/texture is truely superior by miles.
I think what you said is true of some places, not of others, and partially true of many.

Here on Long Island, NY many local town codes often state that between 60-80% of what is sold at far stands must be locally grown, enforcing this is quite difficult though as the farmers are in charge of policing themselves. I try to patronize local stands and markets, but you really have to know who you are dealing with.