The U.S. Department of Agriculture census seems to back up Vilsack's complaint and his stated "need to expand the rural economy from the middle out." Large farms with over $1 million in sales account for only 4 percent of all farms, but 66 percent of all sales. That's up considerably from 1 percent of all farms and 50 percent of all sales a decade ago.

However, three quarters of all U.S. farms gross only $50,000 a year and currently account for only 4 percent of product sales. But one analyst doesn't see that as a problem.

"It's not terribly surprising that a relatively small number of farms produce the lion's share of products," said Darren Hudson, a professor of agriculture and applied economics at Texas Tech University.

"Most of the smaller farms are 'hobby' farms with significant sources of off-farm income," he said. "They are large in number but have very small output."

Hudson argued it's because of the production from bigger farms that consumers enjoy lower food prices than they would otherwise—since those farms produce large amounts of crops cheaply through modern technology.

If the recent findings—which cover the years from 2007 to 2012—do one thing, said Justin Gardner, a professor of agribusiness at Middle Tennessee State University, they dispel a common misconception.

"Most farms in the U.S. are not the big corporations people think," he said. "If you look closer I suspect that you will find that the majority of the 'corporate' farms are large family farms that found it beneficial to form a corporation."

More Latino farmers

The census, released every five years and conducted since 1840, also found that 30 percent of all farm operators are women and that the number of Latino farm operators has increased 21 percent since 2007. Sales of organically farmed food have increased 82 percent since 2007, to $3.1 billion in 2012.

Also increasing was the direct sale of products from farms to consumers, up 8.1 percent since five years ago, to total more than $1.3 billion in 2012.

"Most of it is driven by the local, fresh food movement," said Texas Tech's Hudson. "This is good in that it offers farmers some diversification in market channels, so they're not completely dependent on large agribusiness firms to move products."

The census also said animal agriculture accounts for 45 percent of all industry sales. Grains, oilseeds, beans and peas make up a third, while fruits and vegetables account for 11 percent of sales.

As for which states lead the way in agriculture sales, California is first with nine of the 10 top counties by value of sales. Fresno County was number one in the U.S., with nearly $5 billion in sales in 2012—which is bigger by itself than the sales of 23 individual states.

Second operators, typically a spouse, relative or business partner, were slightly younger and most likely to be female. However, young, beginning principal operators who reported their primary occupation as farming increased 11.3 percent to 40,499 from 36,396 between 2012 and 2007.

That's a good sign to show a changing of the guard, said Texas Tech's Hudson.

"Getting into farming without an existing land base to inherit is expensive," he said. "So few want to do it. But seeing this increase gives me hope that the corner has been turned and we can get some new blood in farming."

In order to help smaller farms and others that need assistance, USDA Secretary Vilsack said on the department's website, the USDA "is going to create new market opportunities for farmers and ranchers, and provide access to capital, credit and disaster assistance for producers of all sizes."

But there's one census statistic that stands out above all that's a true positive sign for the industry, said Middle Tennessee State's Gardner—farms with Internet access rose to 69.6 percent in 2012 from 56.5 percent in 2007.

"To me this is the most exciting trend," he said. "The Internet is a great learning tool. With an Internet connection, a farmer can become a better producer and a better marketer."