Of Clotheslines, Black Swans And Bad Measurements

from the we-measure-the-wrong-things-and-we-do-so-badly dept

Back in April, in writing about Union Square Ventures' Hacking Society event, I discussed the importance of measuring the unmeasurable, in noting that we all too often seem to be evaluating information-era economics using industrial-era metrics. That's a problem. Nick Grossman, who organized that Hacking Society session, has a great post discussing this same concept and highlighting Tim O'Reilly's discussion about this topic, in which he describes the clothesline paradox, which actually seems to come from a discussion in the early 1970s, and highlights how metrics can mislead. You can think of the clothesline paradox like this:

If you take down your clothes line and buy an electric clothes dryer the electric consumption of the nation rises slightly. If you go in the other direction and remove the electric clothes dryer and install a clothesline the consumption of electricity drops slightly, but there is no credit given anywhere on the charts and graphs to solar energy which is now drying the clothes.

In my mind, there are two "problems" associated with this, and while I think there is interest in attacking the first one, the second problem is often ignored. The first problem is that we notice that important information is measured with the wrong metrics. We see this all the time in the internet era. People talk about "the collapse" of the music industry, but miss the fact that more music has been produced, recorded and released in the last decade than in any previous decade. In fact, some of the evidence suggests more music was produced and recorded in the last decade than all other decades combined. Of course, that's an example of a metric that can be determined, but not all such metrics are that easy to pin down. For example, we talked about how Craigslist almost certainly helped contribute to the challenge that many newspapers are facing, because it undercut the cash cow that supported many of them: the classified advertising business. And if you used traditional metrics, you'd bizarrely and incorrectly suggest that Craigslist somehow "destroyed" value. But that's because no one takes into account all the value that Craigslist created, not for itself, but for its users. But how do you measure the fact that I can now find someone to take my old couch away for free? There's value in that transaction, but no one "measures" it. What about the fact that I can more efficiently rent out an apartment - without having to pay the local newspaper? Again, there's value, but it's not properly measured.

The second problem is a little trickier to understand. It's that when we have things that we can measure, we instinctively gravitate towards using those metrics, even if they're the wrong metrics! I was thinking about this as I read Paul Graham's excellent thoughts on "black swan farming," which is all about the counter-intuitive process involved in funding startups. There's a ton of tremendously thought-provoking lines in that piece, but I'm going to concentrate on one, which was really more of an aside, unrelated to the larger article (which you should go read), because it helped clarify my thinking on this point. Graham talks about not bothering to measure how many of the YCombinator companies he funds and trains later go on to raise more money after their initial fundraising efforts, noting:

I deliberately avoid calculating that number, because if you start measuring something you start optimizing it, and I know it's the wrong thing to optimize.

And here's where the problem of using the wrong metrics becomes compounded. Even if you know something is the wrong metric, just having the number almost forces you to optimize for it. So rather than looking at, say, what's best for the overall culture of music, we look at "revenue for the record labels" and decide we need to "fix" that. Or, we look at the patent system as a proxy number for "innovation" and then the focus becomes solely on increasing the number of patents we issue, rather than on actually maximizing innovation.

When you have the wrong metrics, not only do you have bad or incomplete information, but even when you know that it's almost impossible not to optimize for those metrics, because you don't have anything else to work towards.

There is a lot of new interest in quantifying all sorts of new data -- and one benefit of the information age is that it also helps to create new data that can be quantified. But not all quantified data is actually that useful, and unfortunately, we often get so focused on the fact that we have a number, we ignore the possibility that the number is not telling us anything useful.

I was recently reminded of Shelby Bonnie's opinion piece from three years ago about why we need to kill the CPM as a metric for advertising (for those who don't know, CPM -- or "cost per thousand" impressions -- is how most banner ads are sold). He noted, quite accurately, that even those with the best of intentions to get away from "CPM-based" advertising seem to end up there in the end anyway. Because we have that number. And it becomes what people optimize around, just because it's there.

All campaigns start with the best of intentions: “let’s do something creative, engaging, and unique!” But unless someone really senior from the agency or client side intervenes, the road for a campaign always leads to the media buyer and the dreaded spreadsheet, where the two most important columns are impressions and cost. Ironically, there’s usually some good stuff in campaigns, but they are thrown in for free as “value adds.” At some point, publishers decide that if all clients care about is impressions, then OK, we’ll give them impressions. The output is an industry that overproduces shallow, superficial, commoditized impressions. Why do we have so many bad sites that republish the same junky content–content that’s often made by machines or $1-per-post contractors? Why do sites intentionally try to get us to turn lots of pages with tons of top 10 lists, photo galleries, or single-paragraph summaries of someone else’s story?

The more I spend time thinking about these issues, the more I think these combined problems -- both not having the right data and then optimizing for the wrong data -- are the keys to many of the issues that we're regularly discussing around here. Figuring out ways to get beyond that, and to find the right data, and break our habits of relying on bad data are going to be increasingly important.

Re: Sometimes Reality gives too Much.

Finding and combining them in the right way is one of the highest levels of scoring in terms of demonstrating your knowledge in a field on universities!

It is the hardest lesson to learn and since reality often is too complex for optimizing or your measurements are too bad, you almost always have to settle for some sub-optimal measures.

Modellers dilemma: Too many variables makes the demand of accuracy on your measurements too high. Too few variables and your model will wilther away in the universe of: Just another mediocre correlation.

For the world that isn't run by lawyers, and we get to talk about that here a lot, it's a world run by accountants. And if they find something they can count easily they will grab it, stuff it into a spreadsheet and you'll pay hell trying to convince them that it's meaningless. Meaning is for Engineers and Designers and such. Just like But, But, Piracy! It's But, But, Numbers!

Re: The problem with economics as a whole

Thanks Suzanne. A simple statement like that seems to sum up why Mike always seems to be making odd claims, because economics tends to measure the wrong stuff.

Measuring marginal costs when the majority of the expensive is up front: WRONG.

Using a lack of sales as a justification for piracy not hurting anyone: WRONG.

Thinking that short term paint color marginal improvements on a product is true innovation: WRONG.

Thanks for stating in simple terms why Mike seems to be wrong very often. I also thank Mike for this big long post, as it clearly explains why he is wrong - because he looks in the wrong places for his data, and ignores the bigger picture.

Re: Re: The problem with economics as a whole

Measuring marginal costs when the majority of the expensive is up front: WRONG.

Ugh. We don't measure marginal cost while ignoring total cost. That's a strawman you made up. We merely point out -- accurately -- that marginal cost is what matters in pricing. Total cost does matter. It matters in making the go/no go call for the investment, something we've stated over and over again.

You're simply making stuff up when you claim that we ignore total costs for marginal costs. And you display an ignorance of economics when you claim that "measuring marginal cost" is somehow "wrong."

That's not a "metrics" problem as discussed here. If you think that, you're even more ignorant of economics than I thought.

Re: Re: Re: The problem with economics as a whole

"that marginal cost is what matters in pricing. Total cost does matter. It matters in making the go/no go call for the investment, something we've stated over and over again."

yet, plenty of economists don't agree with you, most of them pretty much suggest that you shouldn't use the marginal cost model in a high up front cost business, as it's misleading. The marginal costs don't make it possible to set an acceptable price in the market place.

But hey, if you want to intentionally ignore what you learned in school, that's up to you!

Re: Re: Re: Re: The problem with economics as a whole

yet, plenty of economists don't agree with you, most of them pretty much suggest that you shouldn't use the marginal cost model in a high up front cost business, as it's misleading. The marginal costs don't make it possible to set an acceptable price in the market place.

I have never seen an economist suggest this. Perhaps you should go back to school. Once again, you seem confused about what is meant by marginal cost. But, you have been confused about that topic going on five or six years now, so at some point I have to conclude the problem is just you.

Re: Re: Re: Re: Re: The problem with economics as a whole

Mike, I understand marginal cost just fine. I also understand that the marginal cost models don't work when it comes to assigning a true market price for something when the marginal cost is NOT the main factor in the overall cost of producing the good. It is incredibly misleading to use that as a price setting mechanism.

Quite simply, if the cost to produce the first unit is so far beyond what the market will bear, then the pure marginal costs of the next unit don't even begin to address the realistic pricing of the product.

Let's take a simple example: Someone creates a new movie. It cost them $1 million to make it, distribute it, everything. Now, in your "infinite" universe, the marginal cost is effectively zero. The problem? The price doesn't address the million spent up front. That fixed cost, while not part of your magical marginal cost arguments, means that pricing based only on marginal cost will fail, because it will drive producers out of the market place.

Remember: When you reach zero marginal cost, you also reach zero marginal benefit. Without a benefit, nobody comes to the table.

Further, you can look around the internet and discover a number of papers discussing how infinite distribution is in fact not infinite, as you quickly reach the point where you either run out of population (they are limited) or run out of people who are interested in your product. You will never have true infinite distribution, because you will never produce an infinite number of copies.

Finally, in discussions of marginal costs and marginal cost pricing, almost all economists point to total average cost as the true indication of the price needed to sell the product in order for companies to remain in business (it's the break even number). Since we have already shown that your distribution is not infinite (there are a finite number of consumers for any given digital product), therefore total costs can in fact be calculated on a per item basis.

There is an interesting twist here of course: Since the market for digital goods is finite (for each product), piracy does have the effect of shrinking the potential market. So when it comes to total cost calculation for the break even point for selling the product, with the smaller market more of the total cost must be assigned to each remaining potential sale. in simple terms, if you shrink the market enough (so there is only 1 single movie ticket buyer left) you would have to sell the ticket for 1 million, even if you have infinite distribution and no marginal costs - because the marginal costs are NOT RELEVANT ANYMORE.

You really should spend more time investigating this Mike. The economy of abundance basically parks the marginal costs arguments out of the discussion, because they are no longer relevant to making a business work. If anything, it looks like pricing using things like price discrimination may be the long term way of dealing with the digital goods market.

Perhaps one day we can meet up and we can look at each other's papers on the subject. School was fun!

Imagine

Imagine trying to build a building, but instead of feet and inches (or meters and centimeters) you try to use gallons and tablespoons (or liters and milliliters). Could you do it? Maybe! The question is how many wrong turns will it take to get it right?

We measure unemployment by counting how many folks applied for unemployment in the last six(?) weeks. To me, unemployment is how many folks can work vs how many folks are working.

Now those two examples do not equate to what Mike is talking about directly. But they are examples of how political forces (not just government) work to obfuscate real issues.

Re: Imagine

"Imagine trying to build a building, but instead of feet and inches (or meters and centimeters) you try to use gallons and tablespoons (or liters and milliliters). Could you do it? Maybe! The question is how many wrong turns will it take to get it right?"

Imagine someone telling you that the foundation and structure wasn't important, but how you delivered the keys to the clients was absolutely critical. That would be Mike focusing in on marginal costs while dealing with large up front cost business structures. You could make a business case out of it, perhaps, but you would be truly missing most of the deal.

Re: Re: Imagine

Remember how capitalism is supposed to work?

For the most part, we're not arguing against that: we're arguing against unsustainable business models. Selling digital content and only digital content? Not sustainable. Selling digital content as part of a larger bundle? Sure. Selling t-shirts, watches and general merchandise surrounding the content? Sure.

Just because a total cost exists does NOT mean that we must buy at that price or else. What it means is that, for your product to be viable, people must buy at that price. That doesn't apply to technically infinite goods (such as .mp3s and .avis). To try and make the world conform to you is akin to The Folly of Cnut.

Re: Re: Re: Re: Imagine

Re: Re: Re: Re: Re: Imagine

Yes, we agree. If you give your music away for free, as some sort of bait for other things, you are pretty much doing the "give it away and pray" model. You are hoping like hell that someone buys the over priced "thing" to pay for the music.

Re: Re: Re: Re: Re: Re: Imagine

One example of this issue is the 95% piracy rate where people take the number of unauthorised uses of a game or app and match it up to the number of people who brought it. Yet that utterly ignores the fact that a pirate is going to be counted towards vastly more uses of apps and games than most people can buy. Which weights the numbers and can make a minority of the over all market the majority of a use for any given product.

It's a case where the issue of piracy is a market issue not a product issue and by using metrics produced at the product level you are distorting your reality of the over all market. In the case of iOS we can show this in how piracy happens on jailbroken phones and that is a metric that is measured (10-20% when I last looked) yet app devs talk about a 90-95% piracy rate. How you deal with a market is vastly different when you are looking at most 20% of people who might be pirates then 95% of people.

That. I've learned so far that sometimes the direct measurements aren't what you are supposed to be looking at or trying to optimize but rather a composite indirect measurement that is formed by other direct ones. And sometimes you have to add a factor to your indirect that simply can't be expressed in a numeric value. That's pretty challenging.

Your music industry example is right on spot. You simply MUST add shows, merchandise and other forms of making money that actually have nothing to do with selling CDs or mp3. Heck, I'd say you need to include crowdsourcing efforts, small groups presenting on bars and many other sources of revenue before stating anything about the music business. The RIAA is but a small part of the pie and the fact that this slice is shrinking steadily doesn't mean the pie isn't growing fast.

Software development

I'm a software engineer. All I could think about while reading this article is management's obsession with SLOC (Source Lines of Code). They seem to base everything on this number, even though it is practically meaningless. But that's the number that is easiest to measure! So they do!
It kills me a little every time someone starts talking about SLOC...