You ever hear someone say something, and all of the sudden you realize that you’ve been trying to say exactly that, in exactly that manner, but hadn’t been so succinct or elegant at it? That someone much smarter than you had already thought about the subject a whole lot and there’s actually formal studies and definitions and so forth, and you feel dumb because there’s no way you could have actually googled for the subject in that way, but there it is on wikipedia in black and white? Happens to me all the time.

Was reading the following from the NYT this morning on the Dunning-Kruger effect, and had a little bit of synchronicity when I realized that my entire problem with certifying people about risk and controls has to do with exactly this subject. My issue with ISACA and CRISC is that because I know that there’s so much that I don’t know, and indeed – that we, the infosec industry does not know, and so in my mind if we wanted to rationally, ethically “certify” someone about risk and controls as a domain expert, about the only thing we can do is to test that they are aware of their (and our) limitations. To do otherwise seems rather irrational to me.

TAKE ALEX’S “APPARENTLY OK” RISK PROFESSIONAL EXAM:

A dozen years or so ago, I was PM for a firewall product that had to go through certification. The certification involved several different tests, but at the time the key to certification was simply that your firewall would not pass packets when set in default-deny state. And it cost a lot to certify. This upset me to no end, mainly because I’d have rather spent the $50k certification cost on building a new GUI.

With that spirit in mind, and with all apologies to Marcus, let me present Alex Hutton’s Apparently OK Risk Professional Certification Exam! Because frankly, the problem isn’t with “using risk management” – no I’m still a very big proponent of that. The problem is that risk analysis is steeped in critical thinking, and not identifying uncertainty is, well, less than professional in my opinion.

THE ALEX HUTTON “APPARENTLY OK” RISK PROFESSIONAL EXAM

Dear Prospective Risk Professional,

Congratulations on deciding to enter the exciting field of Information Risk Management! Your journey will be confusing, frustrating, and if you get off on performing sisyphean tasks, rewarding.

To achieve a state of “APPARENTLY OK” we ask that you take the following exam. The exam is one question, and you have three (3) minutes to answer it.

For all the risk assessment methodologies inventoried by ENISA (http://rm-inv.enisa.europa.eu/rm_ra_methods.html) and for RiskIT, please tell us the how the assessment methodology is fundamentally incapable of delivering the results claimed.

7 comments

Typical risk assessment methodologies assume all threats are known, quantifiable, and relatively static. The dynamic nature of attacks and the vectors they use are underappreciated, which essentially keeps these risk assessment methodologies behind the curve.

You can also see the anosognosia phenomenon at work in the I have “nothing to hide” idea, which is well dissected in Daniel Solove’s book of the same name. Put differently if uncomfortably, now that technology is in a positive feedback loop, our individual and collective ignorance increases and can only increase at an increasing rate. The machines win, and thus the question is whether you care, whether your knowledge of your mounting ignorance in comparison to the machines is fruitful. Are the folks who say “whatever” going to be the happiest? Is the desire to know, as opposed to the willingness to accept, trending towards being an anomaly (noting that once it is sufficiently anomalous it will be thought a pathology). Is there a threshhold of irreversibility and is it now in the past?