Sunday, September 28, 2008

How to Measure Anything

My background and practice have been about quantitative analysis of many different forms. Before you do any kind of analysis you have to have data. To have any kind of data you have to measure something. This book makes the point that most anything can be measured.

We had a course called 'economic analysis', which tried to cover some if the same things, but this book does a better job. The author provides a number of rules-of-thumb than can be useful to anyone. I particularly like the section that deals with calibration in measurement when using multiple human experts. Very nicely explained and useful.

Not many equations, but the individual sections need to be carefully read, so this is still not the book for just handing out to managers. His general modeling philosophy reminds me of the work of Sam Savage.

He makes a good case for really understanding the economics of the information that you are gathering. What does it cost to gather the information and what value will be provided by various levels of reducing model uncertainty? What is the difference between expected value of information and the expected value of perfect information? I have rarely see this concept covered.

Another section deals with the ever important problem of how many samples or measurements you need to come up with for a useful model. Often people would come to me with a half dozen measurements and I would recite the statistics litany that you needed more measurements to get statistically valid results. Hubbard makes a good case that such results are rarely measured outside of a context and you can get valuable knowledge from just a few measurements.

More 'modern' Bayesian and Prediction Market methods are well outlined. He also describes the Internet as an instrument, and how sources like the Wikipedia can be used, with care, as a way to construct data models. He also cautions about methods that will add error to initial estimates. Some good Excel examples of applications, though he should have provided more and moved them to an appendix.

A section on using human experts to make judgement is useful since often measurements of intangibles are in people's heads, and you have to consider how non-rational and non-cognitive elements of their opinions will influence the results.

The book is based on years of practical applications, good practical examples throughout. I highly recommend this book for the practitioner.

1 comment:

"Hubbard has made a career of finding ways to measure things that other folks thought were immeasurable. Quality? The value of telecommuting? The risk of IT project failure? the benefits of greater IT security? Public image? He says it can be done -- and without breaking the bank. Many IT steering committees won't approve projects that "can't be measured," so it behooves CIOs to figure this out! ...... If you'd like to fare better in the project-approval wars, take a look at this book.-------------------geovani