State Rankings More Art Than Science

Over the years, the Policy Dialogue on Entrepreneurship has featured an assortment of rankings that examine just about everything you can imagine to do with the economic well-being of all 50 U.S. states. Of course, most informed readers understand that different organizations lead to different points of emphasis, different methodologies and you guessed it—different results.

Still, there is just something so comforting about seeing those rankings and wanting to use them to validate opinions and even action plans. A new paper from the Kauffman Foundation—which has funded similar efforts—is asking policymakers to pump the brakes a bit.

State economic rankings, though popular with think tanks, policymakers, media and even some academics, cannot be taken at face value, according to this article published online today by the .

The article, titled "How Can I Create My Favorite State Ranking?" appears in the Journal of Applied Research in Economic Development. The authors point out that such rankings typically are subjective and provide little meaningful information, despite their pervasiveness.

"Often, these rankings are plagued by lack of clarity about what is being measured or whether the ranking accurately assesses what it claims to measure," said Kauffman senior scholar Yasuyuki Motoyama, Ph.D., who wrote the article with Jared Konczal, a senior analyst. "Further, the connection between these indexes and actual economic growth and performance at the state level is ambiguous, at best."

Forthcoming Kauffman research, the article says, shows that many rankings fail even to correlate with company owners' business climate perceptions. Further, the proliferation of state rankings can lead policymakers and economic development consultants to misuse them, either celebrating a conveniently positive ranking or initiating efforts to address a poor ranking, when neither action is based on valid economic indicators.

Everyone is a winner

The Kauffman researchers created a simulation analysis with randomly generated weights, which revealed that, among 1,000 different scenarios, five states were eligible to be No. 1, 16 were eligible for the Top Five and 22 could be ranked in the Top 10.

The subjectivity and limitations of state economic rankings led to four observations, according to the article:

Policymakers should not rely on a single indicator to gauge economic conditions.

Aggregating indicators does not provide solutions because indicators are highly variable.

Policymakers should not focus on improving their states' rankings because the rankings lack meaning.

Rather, they should employ a scorecard approach, which does not create a normative, quantified measure, but descriptively assesses various conditions of each state.