Site

Tag Archives for measurement

The capability and appropriateness of measurement systems and related metrics are not just things that scientists and engineers must care about. This seems to be obvious to just about anyone, unless you were on Wall Street before the financial crisis.

The Deloitte Center for the Edge has published a report on the decline in the return on assets of American businesses over the past 40 years. Jon Taplin, a professor at USC, posted a very insightful summary of the report, likening this decline and how it was hidden to a shell game. What was interesting to me in his post was how the blind obedience to a particular metric has been in large part to blame for our current financial insanity.

What the Deloitte report points out is that companies have been able to “juice” their return-on-equity (ROE) numbers by consistently taking on more and more debt. Meanwhile, their return-on-assets (ROA) have fallen steadily. If you are even a casual investor or small businessperson, you’ve probably heard of ROE and why it is important. You may not have heard of ROA. Let me briefly explain the difference

Return on equity is a company’s annual net income divided by total shareholder equity. Shareholder equity is essentially how much money investors have put into your company, so ROE measures how effective you are at generating a return on invested funds. Return on assets, on the other hand, is your annual net income divided by total assets. ROA, therefore, measures your effectiveness at generating a return on everything the company owns and is in the bank.

You may already be seeing the disconnect, just based on my choice of words when I defined ROE and ROA above. Let me give you an example: Let’s say you have two companies, A & B. Each of these companies generates 1 M$ per year in net income. Each of these companies has 5 M$ in equity on the books, meaning that the investors have 5 M$ in them. In each case, the ROE of the company is 20%. Not too shabby. But there is an important difference between them. Company B also has 5 M$ of debt outstanding. Company B will thus have 5 M$ more assets on the books than Company A, and thus their ROA will be lower.

If you’re a CEO and you’re managed by your board on the basis of your ROE, you thus have a substantial incentive to leverage your company with loads of debt in order to have more resources with which to expand your business, since that debt doesn’t show up directly on your measurements. People will still invest in your company on the basis of your keen ROE (so long as they don’t look at your debt-to-equity ratio, or your actual return on assets.)

The bottom line here is that a lot of people had a warning right in front of them about what was happening with GM and other companies, but couldn’t see it because one of their chief metrics hid it from them. As with so many other things, relying on a few simple metrics is dangerous and sloppy. Simple metrics are useful, but they must be cross-checked and reviewed with a constant eye on exactly what they tell you and what they do not.

Its sort of a joke, especially among industrial scientists, that physicists are ok at lots of things but not excellent at anything and that this explains why there are so few physicists in industry doing physics. While there is some accuracy to the joke, the truth is that the one thing that physicists excel at is measurement. As my graduate advisor used to point out, all of physics is counting. The trick is just to figure out the right things to count and the right way to count them. That’s the essence of measurement and its not always as easy as it seems.

Everybody needs to measure stuff. And whether you’re in a traditional business or a own a Web 2.0 startup or are just an average gal or guy, the need to measure things quickly and precisely has gotten a lot more intense in the past decade. You want to understand where your business is in the Long Tail or how “sticky” your website is or how much your coffee habit is costing you annually. And when I say precisely, I mean precisely in a, well, precise sense. To speak precisely, precise and accurate are not the same things. And this is the first thing to understand about measurement – a measurement is only valid when it is both sufficiently accurate and sufficiently precise.

Accuracy vs. precision

When you measure something accurately, your measurement gives you a number that is very close to the truth. You may not get the same number each time you make your measurement, but you know that its close to the actual value. When you measure something precisely, you’ll get close to the same result each time, but you may not be close to the actual answer. Ideally, we want our measurements to be both accurate and precise. In reality, most folks have a higher tolerance for lack of accuracy than they do for lack of precision. As long as the measurement is reasonably accurate, most people will settle for something that is off from the truth by a good bit so long as they get consistent answers from it. If you check your measuring cups in your kitchen drawers, you will find that they are pretty precise. Fill your 1/4 cup measure up 4 times and dump it in your 1 cup measure and it will fill it up exactly. (Or at least it did on each of the three sets of measures I had in my kitchen.) Yet, I have no idea – nor do I care – if the cups are calibrated properly. Do they deliver exactly 1 cup? If you care about that kind of accuracy, you’ll probably be using a graduated cylinder, not a plastic measuring cup. For most of us, the fact that the cups are precise is more important.

Does our tolerance for inaccuracy seem surprising? If you use Google Analytics to track your website stats, it shouldn’t be. Google can’t know, accurately, how many unique visitors actually visited your site. How can they? Even though they set a cookie to track your visitors, a lot of folks using Firefox will only accept cookies for that session, thus preventing Google from counting them over multiple visits. I do essentially the same thing with Omniweb. A lot of folks using IE will occasionally flush all of their cookies as a privacy measure. Each time the Google Analytics cookie for your site gets deleted, that user looks like a new user to Google. This means your unique visitor count is artificially high, as is your percentage of new users visiting your site. But, really, it doesn’t matter. You don’t care how accurate that number is, because whether you have 570 or 450 unique visitors per day isn’t as important as the trend. Is that number going up or down? Is it higher on Saturday mornings or weekday nights? As long as the measurement is precise, then those trends can be analyzed meaningfully.

Now we know what makes a measurement valid, and we understand that a large fraction of the time, we don’t need as much accuracy as we need precision. While I didn’t explicitly talk about it, it’s important to note that validity is predicated only upon sufficient accuracy and precision. Your car’s fuel gauge is neither terribly accurate nor terribly precise, but it represents a valid measurement because it gives you the data with sufficient accuracy and precision to keep you from running out of gas.

There are four other things to keep in mind about a measurement, which I’ll call the Four ‘R’s: Relevance, Range, Resolution, and Reproducibility.