Archive

This sixth episode of Take5 interviews Andy Jaquith, Yankee Group analyst and champion of all things Metric…I must tell you that Andy’s answers to my interview questions were amazing to read and I really appreciate the thought and effort he put into this.

First a little background on the victim:

Andrew Jaquith is a program manager in Yankee Group’s Enabling
Technologies Enterprise group with expertise in portable digital
identity and web application security. As Yankee Group’s lead security
analyst, Jaquith drives the company’s security research agenda and
researches disruptive technologies that enable tomorrow’s Anywhere
Enterprise™ to secure its information assets.

Jaquith
has 15 years of IT experience. Before joining Yankee Group, he
co-founded and served as program director at @stake, Inc., a security
consulting pioneer, which Symantec Corporation acquired in 2004. Before
@stake, Jaquith held project manager and business analyst positions at
Cambridge Technology Partners and FedEx Corporation.

His application security and metrics research has been featured in publications such as CIO, CSO and the IEEE Security & Privacy.
In addition, Jaquith is the co-developer of a popular open source wiki
software package. He is also the author of the recently released
Pearson Addison-Wesley book, Security Metrics: Replacing Fear, Uncertainty and Doubt. It has been praised by reviewers as both ”sparking and witty“ and ”one of the best written security books ever.”Jaquith holds a B.A. degree in economics and political science from Yale University.

Questions:

1) Metrics. Why is this such a contentious topic? Isn't the basic axiom of "you can't manage what you don't measure" just common sense? A discussion on metrics evokes very passionate discussion amongst both proponents and opponents alike. Why are we still debating the utility of measurement?

The arguments over metrics are overstated, but to the extent they are contentious, it is because "metrics" means different things to different people. For some people, who take a risk-centric view of security, metrics are about estimating risk based on a model. I'd put Pete Lindstrom, Russell Cameron Thomas and Alex Hutton in this camp. For those with an IT operations background, metrics are what you get when you measure ongoing activities. Rich Bejtlich and I are probably closer to this view of the world. And there is a third camp that feels metrics should be all about financial measures, which brings us into the whole "return on security investment" topic. A lot of the ALE crowd thinks this is what metrics ought to be about. Just about every security certification course (SANS, CISSP) talks about ALE, for reasons I cannot fathom.

Once you understand that a person's point of view of "metrics" is going to be different depending on the camp they are in -- risk, operations or financial -- you can see why there might be some controversy between these three camps. There's also a fourth group that takes a look at the fracas and says, "I know why measuring things matter, but I don't believe a word any of you are talking about." That's Mike Rothman's view, I suspect.

Personally, I have always taken the view that metrics should measure things as they are (the second perspective), not as you imagine, model or expect them to be. That's another way of saying that I am an epiricist. If you collect data on things and swirl them around in a blender, interesting things will stratify out.

Putting it another way: I am a measurer rather than a modeler. I don't claim to know what the most important security metrics are. But I do know that people measure certain things, and that those things gives them insights into their firms performance. To that end, I've got about 100 metrics documented in my book; these are largely based what people tell me they measure. Dan Geer likes to say, "it almost doesn't matter what you measure, but get started and measure something." The point of my book, largely, is to give some ideas about what those somethings might be, and to suggest techniques for analyzing the data once you have them.

Metrics aren't really that contentious. Just about everyone in the securitymetrics.org community is pretty friendly and courteous. It's a "big tent." Most of the differences are with respect to inclination. But, outside of the "metrics community" it really comes down to a basic question of belief: you either believe that security can be measured or you don't.

The way you phrased your question, by the way, implies that you probably align a little more closely with my operational/empiricist view of metrics. But I'd expect that, Chris -- you've been a CSO, and in charge of operational stuff before. :)

2) You've got a storied background from FedEx to @Stake to the Yankee Group. I see your experience trending from the operational to the analytical. How much of youroperational experience lends itself to the practical collection and presentation of metrics -- specifically security metrics? Does your broad experience help you in choosing what to measure and how?

That's a keen insight, and one I haven't thought of before. You've caused me to get all introspective all of a sudden. Let me see if I can unravel the winding path that's gotten me to where I am today.

My early career was spent as an IT analyst at Roadway, a serious, operationally-focused trucking firm. You know those large trailers you see on the highways with "ROADWAY" on them? That's the company I was with. They had a reputation as being like the Marines. Now, I wasn't involved in the actual day-to-day operations side of the business, but when you work in IT for a company like that you get to know the business side. As part of my training I had to do "ride alongs," morning deliveries and customer visits. Later, I moved to the contract logistics side of the house, where I helped plan IT systems for transportation brokerage services and contract warehouses the company ran. The logistics division was the part of Roadway that was actually acquired by FedEx.

I think warehouses are just fascinating. They are one hell of a lot more IT intensive than you might think. I don't just mean bar code readers, forklifts and inventory control systems; I mean also the decision support systems that produce metrics used for analysis. For example, warehouses measure an overall metric for efficiency called "inventory turns" that describes how fast your stock moves through the warehouse. If you put something in on January 1 and move it out on December 31 of the same year, that part has a "velocity" of 1 turn per year. Because warehouses are real estate like any other, you can spread out your fixed costs by increasing the number of turns through the warehouse.

For example, one of the reasons why Dell -- a former customer of mine at Roadway-- was successful was that they figured out how to make their suppliers hold their inventory for them and deliver it to final assembly on a "just-in-time" (JIT) basis, instead of keeping lots of inventory on hand themselves. That enabled them to increase the number of turns through their warehouses to something like 40 per year, when the average for manufacturing was like 12. That efficiency gain translated directly to profitability. (Digression: Apple, by the way, has lately been doing about 50 turns a year through their warehouses. Any wonder why they make as much money on PCs as HP, who has 6 times more market share? It's not *just* because they choose their markets so that they don't get suckered into the low margin part of the business; it's also because their supply chain operations are phenomenally efficient.)

Another thing I think is fascinating about warehouses is how you account for inventory. Most operators divide their inventory into "A", "B" and "C" goods based on how fast they turn through the warehouse. The "A" parts might circulate 10-50x faster than the C parts. So, a direct consequence is that when you lay out a warehouse you do it so that you can pick and ship your A parts fastest. The faster you do that, the more efficient your labor force and the less it costs you to run things. Neat, huh?

Now, I mention these things not strictly speaking to show you what a smartypants I am about supp
ly chain operations. The real point is to show how serious operational decisions are made based on deep analytics. Everything I just mentioned can be modeled and measured: where you site the warehouses themselves, how you design the warehouse to maximize your ability to pick and ship the highest- velocity items, and what your key indicators are. There's a virtuous feedback loop in place that helps operators understand where they are spending their time and money, and that in turn drives new innovations that increase efficiencies.

In supply chain, the analytics are the key to absolutely everything. And they are critical in an industry where costs matter. In that regard, manufacturing shares a lot with investment banking: leading firms are willing to invest a phenomenal amount of time and money to shave off a few basis points on a Treasury bill derivative. But this is done with experiential, analytical data used for decision support that is matched with a model. You'd never be able to justify redesigning a warehouse or hiring all of Yale's graduating math majors unless you could quantify and measure the impact of those investments on the processes themselves. Experiential data feeds, and improves, the model.

In contrast, when you look at security you see nothing like this. Fear and uncertainty rule, and almost every spending decision is made on the basis of intuition rather than facts. Acquisition costs matter, but operational costs don't. Can you imagine what would happen if you plucked a seasoned supply-chain operations manager out of a warehouse, plonked him down in a chair, and asked him to watch his security counterpart in action? Bear in mind this is a world where his CSO friend is told that the answer to everything is "buy our software" or "install our appliance." The warehouse guy would look at him like he had two heads. Because in his world, you don't spray dollars all over the place until you have a detailed, grounded, empirical view of what your processes are all about. You simply don't have the budget to do it any other way.

But in security, the operations side of things is so immature, and the gullibility of CIOs and CEOs so high, that they are willing to write checks without knowing whether the things they are buying actually work. And by "work," I don't mean "can be shown to stop a certain number of viruses at the border"; I mean "can be shown to decrease time required to respond to a security incident" or "has increased the company's ability to share information without incurring extra costs," or "has cut the pro-rata share of our IT operations spent on rebuilding desktops."

Putting myself in the warehouse manager's shoes again: for security, I'd like to know why nobody talks about activity-based costing. Or about process metrics -- that is, cycle times for everyday security activities -- in a serious way. Or benchmarking -- does my firm have twice as many security defects in our web applications as yours? Are we in the first, second, third or fourth quartiles?

If the large security companies were serious, we'd have a firmer grip on the activity, impact and cost side of the ledger. For example, why won't AV companies disclose how much malware is actually circulating within their customer bases, despite their promises of "total protection"? When the WMF zero-day exploit came out, how come none of the security companies knew how many of their customers were infected? And how much the cleanup efforts cost? Either nobody knew, or nobody wanted to tell. I think it's the former. If I were in the shoes of my Roadway operational friend, I'd be pissed off about the complete lack of feedback between spending, activities, impact and cost.

If this sounds like a very odd take on security, it is. My mentor and former boss, Dan Geer, likes to say that there are a lot of people who don't have classical security training, but who bring "hybrid vigor" to the field. I identify with that. With my metrics research, I just want to see if we can bring serious analytic rigor to a field that has resisted it for so long. And I mean that in an operational way, not a risk-equation way.

So, that's an exceptionally long-winded way of saying "yes" to your question -- I've trended from operational to analytical. I'm not sure that my past experience has necessarily helped me pick particular security metrics per se, but it has definitely biased me towards those that are operational rather than risk-based.3) You've recently published your book. I think it was a great appetite whetter but I was left -- as were I think many of us who are members of the "lazy guild"-- wanting more. Do you plan to follow-up with a metrics toolkit of sorts? You know, a templatedguide -- Metrics for Dummies?

You know, that's a great point. The fabulous blogger Layer 8, who gave my book an otherwise stunning review that I am very grateful for ("I tucked myself into bed, hoping to sleep—but I could not sleep until I had read Security Metrics cover to cover. It was That Good."), also had that same reservation. Her comment was, "that final chapter just stopped short and dumped me off the end of the book, without so much as a fare-thee-well Final Overall Summary. It just stopped, and without another word, put on its clothes and went home". Comparing my prose to a one night stand is pretty funny, and a compliment.

Ironically, as the deadline for the book drew near, I had this great idea that I'd put in a little cheat-sheet in the back, either as an appendix or as part of the endpapers. But as with many things, I simply ran out of time. I did what Microsoft did to get Vista out the door -- I had to cut features and ship the fargin' bastid.

One of the great things about writing a book is that people write you letters when they like or loathe something they read. Just about all of my feedback has been very positive, and I have received a number of very thoughtful comments that shed light on what readers' companies are doing with metrics. I hope to use the feedback I've gotten to help me put together a "cheat sheet" that will boil the metrics I discuss in the book into something easier to digest.4) You've written about the impending death of traditional Anti-Virus technology and itsevolution to combat the greater threats from adaptive Malware. What role do you think virtualization technology that provides a sandboxed browsing environment will have in this space, specifically on client-side security?

It's pretty obvious that we need to do something to shore up the shortcomings of signature-based anti-
malware software. I regularly check out a few of the anti-virus benchmarking services, like the OITC site that aggregates the Virustotal scans. And I talk to a number of anti-malware companies who tell me things they are seeing. It's pretty clear that current approaches are running out of gas. All you have to do is look at the numbers: unique malware samples are doubling every year, and detection rates for previously-unseen malware range from the single digits to the 80% mark. For an industry that has long said they offered "total protection," anything less than 100% is a black eye.

Virtualization is one of several alternative approaches that vendors are using to help boost detection rates. The idea with virtualization is to run a piece of suspected malware in a virtual machine to see what it does. If, after the fact, you determine that it did something naughty, you can block it from running in the real environment. It sounds like a good approach to me, and is best used in combination with other technologies.

Now, I'm not positive how pervasive this is going to be on the desktop. Existing products are already pretty resource-hungry. Virtualization would add to the burden. You've probably heard people joke: "thank God computers are dual-core these days, because we need one of 'em to run the security software on." But I do think that virtualized environments used for malware detection will become a fixture in gateways and appliances.

Other emergent ideas that complement virtualization are behavior blocking and herd intelligence. Herd intelligence -- a huge malware blacklist-in-the-sky -- is a natural services play, and I believe all successful anti-malware companies will have to embrace something like this to survive.

5) We've see the emergence of some fairly important back-office critical applications make their way to the Web (CRM, ERP, Financials) and now GoogleApps is staking a hold for the SMB. How do you see the SaaS model affecting the management of security -- and ultimately risk -- over time?

Software as a service for security is already here. We've already seen fairly pervasive managed firewall service offerings -- the carriers and companies like IBM Global Services have been offering them for years. Firewalls still matter, but they are nowhere near as important to the overall defense posture as before. That's partly because companies need to put a lot of holes in the firewall. But it's also because some ports, like HTTP/HTTPS, are overloaded with lots of other things: web services, instant messaging, VPN tunnels and the like. It's a bit like the old college prank of filling a paper bag with shaving cream, sliding it under a shut door, then jumping on it and spraying the payload all over the room's occupants. HTTP is today's paper bag.

In the services realm, for more exciting action, look at what MessageLabs and Postini have done with the message hygiene space. At Yankee we've been telling our customers that there's no reason why an enterprise should bother to build bespoke gateway anti-spam and anti- malware infrastructures any more. That's not just because we like MessageLabs or Postini. It's also because the managed services have a wider view of traffic than a single enterprise will ever have, and benefit from economies of scale on the research side, not to mention the actual operations.

Managed services have another hidden benefit; you can also change services pretty easily if you're unhappy. It puts the service provider's incentives in the right place. Qualys, for example, understands this point very well; they know that customers will leave them in an instant if they stop innovating. And, of course, whenever you accumulate large amounts of performance data across your customer base, you can benchmark things. (A subject near and dear to my heart, as you know.)

With regards to the question about risk, I think managed services do change the risk posture a bit. On the one hand, the act of outsourcing an activity to an external party moves a portion of the operational risk to that party. This is the "transfer" option of the classic "ignore, mitigate, transfer" set of choices that risk management presents. Managed services also reduce political risk in a "cover your ass" sense, too, because if something goes wrong you can always point out that, for instance, lots of other people use the same vendor you use, which puts you all in the same risk category. This is, if you will, the "generally accepted practice" defense.

That said, particular managed services with large customer bases could accrue more risk by virtue of the fact that they are bigger targets for mischief. Do I think, for example, that spammers target some of their evasive techniques towards Postini and MessageLabs? I am sure they do. But I would still feel safer outsourcing to them rather than maintaining my own custom infrastructure.

Overall, I feel that managed services will have a "smoothing" or dampening effect on the risk postures of enterprises taken in aggregate, in the sense that they will decrease the volatility in risk relative to the broader set of enterprises (the "alpha", if you will). Ideally, this should also mean a decrease in the *absolute* amount of risk. Putting this another way: if you're a rifle shooter, it's always better to see your bullets clustered closely together, even if they don't hit near the bull's eye, rather than seeing them near the center, but dispersed. Managed services, it seems to me, can help enterprises converge their overall levels of security -- put the bullets a little closer together instead of all over the place. Regulation, in cases where it is prescriptive, tends to do that too.

Bonus Question:6) If you had one magical dashboard that could display 5 critical security metrics to the Board/Executive Management, regardless ofindustry, what would those elements be?

I would use the Balanced Scorecard, a creation of Harvard professors Kaplan and Norton. It divides executive management metrics into four perspectives: financial, internal operations, customer, and learning and growth. The idea is to create a dashboard that incorporates 6-8 metrics into each perspective. The Balanced Scorecard is well known to the corner office, and is something that I think every security person should learn about. With a little work, I believe quite strongly that security metrics can be made to fit into this framework.

Now, you might ask yourself, I've spent all of this work organizing my IT security policies along the lines of ISO 17799/2700x, or COBIT, or ITIL. So why can't I put together a dashboard that organizes the measurements in those terms? What's wrong with the frameworks I've been using? Nothing, really, if you are a security person. But i
f you really want a "magic dashboard" that crosses over to the business units, I think basing scorecards on security frameworks is a bad idea. That's not because the frameworks are bad (in fact most of them are quite good), but because they aren't aligned with the business. I'd rather use a taxonomy the rest of the executive team can understand. Rather than make them understand a security or IT framework, I'd rather try to meet them halfway and frame things in terms of the way they think.

So, for example: for Financial metrics, I'd measure how much my IT security infrastructure is costing, straight up, and on an activity- based perspective. I'd want to know how much it costs to secure each revenue-generating transaction; quick-and-dirty risk scores for revenue-generating and revenue/cost-accounting systems; DDOS downtime costs. For the Customer perspective I'd want to know the percentage and number of customers who have access to internal systems; cycle times for onboarding/offloading customer accounts; "toxicity rates" of customer data I manage; the number of privacy issues we've had; the percentage of customers who have consulted with the security team; number and kind of remediation costs of audit items that are customer-related; number and kind of regulatory audits completed per period, etc. The Internal Process perspective has of the really easy things to measure, and is all about security ops: patching efficiency, coverage and control metrics, and the like. For Learning and Growth, it would be about threat/planning horizon metrics, security team consultations, employee training effectiveness and latency, and other issues that measure whether we're getting employees to exhibit the right behaviors and acquire the right skills.

That's meant to be an illustrative list rather than definitive, and I confess it is rather dense. At the risk of getting all Schneier on you, I'd refer your readers to the book for more details. Readers can pick and choose from the "catalog" and find metrics that work for their organizations.

Overall, I do think that we need to think a whole lot less about things like ISO and a whole lot more about things like the Balanced Scorecard. We need to stop erecting temples to Securityness that executives don't give a damn about and won't be persuaded to enter. And when we focus just on dollars, ALE and "security ROI", we make things too simple. We obscure the richness of the data that we can gather, empirically, from the systems we already own. Ironically, the Balanced Scorecard itself was created to encourage executives to move beyond purely financial measures. Fifteen years later, you'd think we security practitioners would have taken the hint.