Analysis of Competing Hypothesis

Kristan over at Sources and Methods has a post up about the efficacy of ACH. I’m a huge fan of the system and actually used the freeware version to help Mrs. TwShiloh and me in deciding on a house to buy.

We were trying to juggle multiple factors on multiple properties. As you know from reading Richards Heuer, you’ll know that the human mind can really only track 7-9 things at one time and attempting to do more just results in all sorts of cognitive train wrecks. We pllugged in the properties with a whole list of criteria (from the important to the trivial), weighted them and ranked them. When it was all set and done one house that we thought we loved dropped to the bottom of the pack and we realized that we had gotten swept away by one or two really attractive features and blotted out the many, many problems such a house would give us.

So, I was pleased to read that ACH actually does have some utility.

The results were in favor of ACH in terms of both forecasting accuracy and bias. In Drew’s words, “The findings of the experiment suggest ACH can improve estimative accuracy, is highly effective at mitigating some cognitive phenomena such as confirmation bias, and is almost certain to encourage analysts to use more information and apply it more appropriately.”

For three years, Matthew Burton has been trying to get a simple, useful software tool into the hands of analysts at the Central Intelligence Agency. For three years, haggling over the code’s intellectual property rights has kept the software from going anywhere near Langley. So now, Burton’s releasing it — free to the public, and under an open source license.

This is really exciting and I’m anxious to see it when it’s finally released. Burton’s improvement is that his ACH version will be collaborative, allowing multiple analysts to add evidence and share views.