Statistics

Tis the season for predictions. If one has an audience one seem compelled to make predictions. You are better off reading the book Superforecasting than this article. The book explains in depth the simplest elements you need in making predictions and forecast.

It starts with – Base Rate – which is how frequent does the said event happen in general relative to all other events. For example

What percentage of tweets are retweeted?

What percentage of people are named Bill?

What percentage of startups achieve $1B valuation?

What are the chances of you winning Survivor when you start the season with 19 others?

The next step is an iterative process that refined this prior knowledge by seeking new information and refining your estimate. That is the posterior probability.

Most likely you won’t read the book, so I present here these two concepts set to the tune of Megan Trainor’s song.

Like this:

How many times have you heard the phrase thrown at you by anyone and their guru? No I am not asking you to keep a measure (count) of that. Whether it was a consulting firm or someone doing time and motion study that popularized it, I don’t know its origins. This statement has become a standard quip by someone, usually someone in power and position with nice title, trying to sound data driven. What that has led to is a world where we collect anything and everything that moves – from measuring sleep pattern with wearable fitness device to up to the minute update of sales pipeline.
We now have a new product positioning for this craze – Fitbit for xyz, named after the wearable pedometer that tracks the number of steps you take and the number of minutes you stay restless in your sleep. With a gauge for everything, collecting data all the time, the craziness has led to utter madness – Big Data.

So I urge you, before you put on a Fitbit on your wrist or Fitbit for customer experience, to stop and ask a few basic questions:

Can I change what I seek to measure?

Even if I can change, can I change it at the same cadence I seek to measure it?

If I can change, fast enough, will the change have meaningful impact on the true measures of business performance? Sure you can change the number of retweets, blog mentions, video views, “how likely to recommend on 0 to 10”, etc., but does it matter to revenue, marketing ROI, pricing effectiveness and profit?

Can I measure it cheaper than the effect of change?

Even if the answer is yes to the questions above you need to ask – what other metric I could be collecting with my limited resources?

What do you measure? I hope the answer is, “I measure what I can change to make meaningful positive impact on my business objectives”.
What you can’t change, don’t measure.

Here is an undeniable fact – considering ~50% of the population is women and 50% of the board of directors are not women, we can safely say women are underrepresented in corporate boards. Now let us return to the reported study in question that makes a faulty case for adding women board members. My arguments are only with errors in the research methods and its application of faulty logic. Nothing more.

I will add to this my case against those who keep quoting such studies as evidence for their side.

The title of this article is inspired by Adrienne LaFrance. She wrote a blog post titled , “You’re stuck on an elevator with someone who loved that Sarah Lacy article. Now what?”

In that article Ms. LaFrance makes point by point argument to what seem to be silly questions from a clueless elevator companion who fell in love with Ms.Lacy’s post. For one such question Ms.LaFrance quotes as evidence the research by Thomson-Reuters,

You should check out this study from earlier this year that showed how diverse corporate boards outperform those with no women. You’d think that a company like Twitter would put its business interests first.

She isn’t alone in quoting this study, almost everyone taking Mr. Vivek Wadhwa‘s side use this study. I am not sure how many read this report or looked at its methods and caveats. Let me do that in this article.

Does the board matter?: The study starts with unverified assumption that a company’s board matters to its performance and then goes on to see differences in performance between boards. If your hidden hypothesis you took for granted is false it does not matter what your stated hypothesis is.
What the study does is, If A=TRUE, then A(With Women) > A(Without Women)
You can see that if A =FALSE, the rest does not matter.You might want to stop here as nothing else matters after this error.

Control Variable: When you want to study the influence of a single variable you want to make sure all other variables are held constant. But when you read this report it is clear that they have no way to do that. They started with composition of a company board in 2007 then compared the performance of a group of companies over a period. There are two many uncontrolled variables during this period (tech trends, market trends, industry verticals, etc.) and these affected different companies differently.

Error in Comparing Averages?: The comparisons are done on averages. There is a group of companies with mixed board and then there is another without women in board. The two groups are compared against another group, the benchmark which consist of companies of both kinds.

The report says companies with women on board did marginally better or same as the benchmark while those with no women on board did 10-15% lower than the benchmark.

First you notice that the difference in performance is not as significant s those who quote the study. Next you want to ask a simple clarifying question here. If the benchmark has both types of companies, if one subset is underperforming by 10-15%, shouldn’t the other subset outperform by 10-15% to bring it back to benchmark average?

The only explanation I can think of is average hides details here. There must be a few companies in each side that are significantly different from the arithmetic mean for that group and they account for the difference. If you leave out these samples and compare again, the difference will likely vanish.

Now to the question of what to do when stuck in an elevator with someone who merely heard about the Thomson Reuter’s study?

Here are three articles this week that use or quote statistics wisely.

Connection between women wearing red and fertility: Slate asks the right questions on representative bias and how it was measured. It points out the major flaw that the study looked for best hypothesis that fits data ex post. That is looking looking for more white sand that confirms Madagascar is indeed San Diego.

Phil Mickelson’s regression analysis: Mickelson thinks putting distance alone, chances of making it. Not the variances in the field, slope, weather etc,., just the distance. That is his prediction model. However the truth is unknown unless someone has looked at his data in aggregation.