The personal blog of Reyn Bowman, a Durham NC resident, 40-year veteran of community-destination marketing and still an explorer in community sense-of-place. Opinions expressed here are those of the author.

Wednesday, July 14, 2010

Statistics Don't Lie, People Fudge

Bumper Sticker slogans are cute and memorable but that doesn’t mean they are accurate or used in the appropriate context. Take for example, a quote an accountant friend of mine loves to use as a way to dismiss research he doesn’t like, “there are lies, damn lies and statistics.” Always gets a chuckle although especially a bit troubling when an accountant uses it.

But it isn’t the statistics that lie, it's the people who misuse and abuse them including those who would rather dismiss them than inform their opinions.

I'm not talking about the blatant misuse of polling such leading questions recently used by one done by the billboard industry. I'm talking about good data used inappropriately or out of context. It is more often that people lie than statistics.

Typically they use only a subset of visitors which makes the percentage seem more impressive but out of synch too. This can be done by limiting visitors to just those person-stays that are “leisure” or an even smaller subset, “vacation.”

Rarely do they distinguish participation from motivation and when they do they are typically not weighted (same reason some political polls are so off) which means the percentage only applies to that sample and can’t be generalized.

They typically haven’t been “quantity weighted” for multiple responses, e.g. responses on activities add up to greater than 100%.

And/or they typically haven’t been weighted for propensity to make sure the sample is truly representative of the general population and not skewed to one segment or another.

And/or they typically haven’t been weighted for nonresponse bias when those responding to an outcome variable differ significantly from those who don’t respond. It is a mistake not to adjust for those who state they did not participate in any activity.

Sometimes the information hasn’t been weighed because those commissioning the study don’t understand the importance or want it to be or they don’t want to go to that expense or never included the steps in their request to make the information applicable in the way it is later being used. I’ve personally given agencies a heads up only to see them go ahead and misuse the information to serve an agenda.

People in any activity area who deliberately do this are either idiots or playing agenda politics by hoping never to be discovered or believing if they are, they can run for cover and the misinformation will stick in people’s minds. I’ve also checked with people who do these studies only to learn they are horrified the information is being misused.

It isn’t statistics that lie. It's people. Unfortunately, the manipulation creates credibility issues and when it backfires, blood gets spattered everywhere not just on the perpetrators.

I’m no genius at this stuff but I’ve learned enough from people who are to be suspicious. I’ve also been certain when involvede with studies myself to circulate findings with which I might not agree. If the science is right, its right.

Information shouldn’t be about agendas. I was on a board once where the organization rejected and suppressed some new calculations because they didn’t jive with information they had used for years from much smaller samples that hadn’t been appropriated weighted. Some egos around the table just couldn’t face up to how to explain the differences and opted for suppression instead.

It is the responsibility of DMO’s at local and state levels to vet information and have the courage to challenge it, especially when something “tourism” is being commissioned by non-tourism entities. It may initially result in a tad of friction but if folks who unwittingly misuse or purposely misuse the information aren’t confronted, it will only get worse. Read or re-read the book Crucial Confrontationsif you need to know how or where to start.

It is also up to elected officials to have agencies with expertise in a particular area, e.g. tourism, vet research by other agencies on that topic and then reconcile issues, even if behind closed doors to prevent misuse and misinformation to legislators, news media and the public.

It is also important for scientists conducting the studies to be insistent on how the information is to be used and intervene when it isn't. It isn't statistics at risk, it is the credibility of the people who commission, conduct or interpret the information.