ROI and Me

January 4, 2005

This morning I re-read Jack Phillips’s Return on Investment in Training and Performance Improvement Programs. Jack gave me the book at TDF last year after we’d had a lengthy and pleasant chat one evening. He’s a genuinely nice guy, traveling the world and working like crazy to proscelitize his message of ROI.

On page 1, Jack writes, “Return on investment is characterized as flawed and inappropriate by some, while others describe it as the only answer to their accountability concerns. The truth probably lies somewhere in between.”

I have always thought the truth lies real close to the “flawed and inappropriate” end of the scale. The first editions of my ebook, Metrics, are positively vitriolic about the utility of ROI, which I consider a vestige of the industrial age, when the only assets that mattered were those that you could see or touch. One reviewer began, “I don’t know what accountant kicked your puppy….”

Trying to figure out how I could see the world so differently from Jack, who is clearly a brilliant guy, minutes ago the answer dawned on me. It’s in the quotation above “…answer to their accountability concerns….”

Accountability, like accounting, deals primarily with the past. It’s the state of being held liable or answerable for what you’ve done. One dictionary says it’s “liability to give account, and to receive reward or punishment for actions.” Another dictionary pegs it as “an obligation or willingness to accept responsibility or to account for one’s actions.” Wikipedia’s definition brings the distinction home: “Accountability differs from transparency in that it only enables negative feedbackafter a decision or action, while transparency also enables negative feedbackbefore or during a decision or action.”

In constrast, my orientation is almost entirely with the future. The only use I have for the past is the lessons it offers to help us create a better future. Rigorous measurement of the results of past engagements is great if it either shows you how to do better next time or builds credibility among skeptics who’ll be making resource allocation decisions in the future. Jack’s conservatism (“pick the lowest ROI figure you come up with”) and thoroughness (“here are ten ways to collect post-program data”) serve these ends well, and I recommend his book if that’s your interest.

* * *

Let me tell you where I’m coming from. I am an insanely curious generalist. I’m also quite conceptual. I’m interested in learning, business management, psychology, software, design, and a host of other things.

A big part of my mission in life is helping people figure things out. If something’s too hard to understand, I try to simplify it. If elements seem unrelated, I draw lines between the dots. If something appears out of nowhere, I try to provide context. When I have an insight others might find worthwhile, I post it on my websites. When I see something others may miss, I snap a photo to share. When someone doesn’t see what’s in it for them, I try to reinterpret the benefits in language they can understand. I empathize with the clueless and confused. That’s just me.

Business managers and training professionals don’t speak the same native language. In more than two decades in the training business, I’ve frequently been asked to translate the benefits of training into business terms. I believe that any training activity can be translated into profit or loss with the right set of assumptions and chain of logic. Neither business models nor the impact of training are rocket science, but one must be multilingual to meld the two together.

* * *

Jack’s analysis of the past and my vision of the future diverge when it comes to sorting out hard data from soft.

Later (p 165) Jack talks of the credibility problem, “Could these results be presented to senior management with confidence? If the process does not meet this credibility test, the data should not be converted to monetary values and instead listed as an intangible benefit. Other data, particularly hard data items, could be used in the ROI calculations, leaving the very subjective data as intangible improvements.”

Looking backward to assess the accountability of a project, maybe it makes sense not to attribute a number to soft data such as job satisfaction, turnover, customer churn, complaints, customer satisfaction, and employee loyalty. Senior management might well argue with specific claims.

Looking forward, choosing among future projects, it is unsound to treat soft benefits you expect to receive as second class citizens who count for nothing.

Our differences become more clear-cut when we get to intangibles.

Jack tells us “Intangible measures are the benefits or detriments directly linked to the training program, which cannot or should not be converted to monetary values. For most intangible data, no specific analysis is planned. Previous attempts to convert intangible data to monetary units results in aborting the process, thus no further data analysis in conducted.”

This is circular logic. Define intangibles as tough to measure; then remove them from the analysis because they are tough to measure. Nonetheless, I can live with this so long as it refers to the past; Jack wants to build a rock-solid case. Unfortunately, the past inevitably becomes a guide to the future, no matter what the prospectus warns you.

“Since the value of this data is not placed in the ROI calculation, intangible measures are not normally used to justify additional training or continuation of existing training. Consequently, a detailed analysis is not justified. Intangible benefits are viewed as supporting evidence of the programs [SIC] success and are presented as qualitative data.”

See what this implies? Intangibles go into the ROI calculation at zero. Intangibles do not justify additional training or continuation of existing training. This will come as a shock to people in management development, leadership training, and workshops on innovation.

Jack allows as how “Although [intangibles] may not be perceived as valuable as specific monetary measures, they nevertheless are an important part of an overall evaluation.” They just get measured with different yardsticks and treated as off-balance-sheet items.

* * *

Jack looks at the past with the most solid and unassailable methodology he can muster. He evaluates what has taken place. He values credibilty highly. He’s accountable.

Jay looks to the future. It’s a world of uncertainties but standing still is not an option. Rationally assessing opportunities leads to better decisions than denying that opportunities exist.

The historian is obligated to report the past honestly; the decision-maker, to use best judgment in making future choices.