Thursday, November 1, 2012

People tend to focus on various areas as being important for computer security such as memory corruption vulnerabilities, malware, anomaly detection, etc. However the lurking and most critical issue in my opinion is staffing. The truth is, there is no pool of candidates out there to draw from at a certain level in computer security. As an example, we do a lot of consulting, especially in the area of incident response, for oil & gas, avionics, finance, etc. When we go on site we find that we have to have the following skills:

1. Soft skills. (often most important) The ability to talk to customers, dress appropriately, give presentations or speak publicly, assess the customer staff, culture and politics, and determine the real goals. I can't stress enough how important this is. It's not the 90s anymore, showing up with a blue mohawk, a spike in the forehead and leather pants, not a team player, cussing and surfing porn on the customers system doesn't cut it no matter how good you are technically. If you are that guy then you get to stay in the lab and I guarantee you will make far less money. Even if you can write ASLR bypass exploits and kernel rootkits.

2. Document. This ties with the above for number 1. If you didn't document it, you didn't do it. I don't care how awesome an 0day you discovered, or what race condition in the kernel you found. If you cant clearly document it, the customer doesn't care and sees no value in what you did. The documentation has to be clean, clear, layed out so that an executive can understand it and so that the other security firm the customer hires to validate your results doesn't make fun of you.

4.) Reverse Engineering. This means disassembling binaries in IDA, running binaries in a debugger such as Ollydbg, WinDBG, IDA, memory forensics, and especially de-obfuscation. Can you unpack a binary? How about if the packer is multi-stage and does memory page check summing? What if the packer carries its own virtual machine? Do you know what breakpoints to set, when to change the Z flag, or how to hot patch a binary in memory?

5.) Understanding programming. To be good at this stuff you need to know C, C++, .NET, VB, HTML, ASP, PHP, x86 assembly and another dozen languages, at least well enough to look up APIs, understand standard libraries, discover which imports are important.

6.) Operating systems. You should know the ins and outs including file systems, memory management, kernel, library system and key command line tools of at least half a dozen OS's, especially as they are used in enterprise environments. Domains, NFS, NIS, kerberos, LDAP. So not only windows, linux and OS X, but also solaris, AIX and some embedded or mobile systems.

7.) Exploit development. Often on engagements you run across an exploit or even an 0day that you must reverse engineer, replicate safely and test on the customers particular environment. You have to be able to take it apart, analyse the shellcode, understand everything its doing and re-write your own version of it.

8.) Versatility with a wide variety of tools, many of which are not easy to access outside of the enterprise. At a minimum enough technical base knowledge to use whatever tool is put in front of you. Examples include wireshark, splunk, fireeye, netwitness, arcsight, tippingpoint, snort / sourcefire, bluecoat, websense, TMI, Encase.

All of the members of your team whether you are a consulting shop or an internal incident response team need to be able to do these things and overlap with each other. Some can be stronger in RE than network forensics but everyone has to be able to do all of it to some extent, especially 1 and 2.

The problem with this? These people don't exist, they are unicorns. Those who can do this are either already employed, well payed and tackling more interesting problems than you can offer, or they are running/partners in their own company that you could (and should) outsource to. </shameless self promotion>. But even small boutiques that can do the above are rare, heavily booked, and are charging close to high powered lawyer hourly rates. (when people question rates I point out that big name IR shops are around $400/hr and even the BestBuy geek squad charges $120/hr to reload your OS).

A lot of big contractors are trying to approach security like they did IT in the 90s and 00's. Bid low, win a huge contract, then put out job ads for anyone who knows how to use a computer. The problem is, while you can come up to speed for a help desk or to admin a windows server relatively quickly, the above list of skills takes a decade + to master. So big contractors are failing, badly, and trying to buy up the small guys. But there is another problem there as well.

People who are able to do the above 1.) Value freedom highly and don't want to work 9 to 5 in a cube farm and 2.) Don't want to live or work long periods of time onsite where you are. They don't want to live in Houston or in Cleaveland or in Indianapolis or probably even in the DC area. They want to live in La Jolla and San Francisco and New York and someone, somewhere is willing to pay them a lot to do it, and probably do it remotely most of the time, so you are going to lose there.

In response, many companies try to follow the old plan of recruiting at colleges. In a lot of cases these students come out knowing some Office and probably some Java and that's about it. You might luck out and get a good RIT, Georgia Tech, New Mexico Tech student who knows more but most likely these have already been recruited to the government or somewhere else. And the learning curve time is long enough that by the time they are really good, they have already moved on. This kind of work is PRIME for remote. Let people come in for a week every other month. If you require internal security people to be on site all the time in some crappy city you will fail.

On the security company side you have the same problem, no one to hire. So many security companies, in order to grow (because the way you make money in services is via higher staffing levels) hire whatever they can find and field them. This continues the trend in mediocre security, companies getting owned, PCI, etc. Boutiques cannot grow to the size necessary to win the bigger contracts because there is no one to hire.

The solution many companies have been trying out is to focus on buying appliances and contracting pro services to set them up and hope that automation can solve the problem. It cannot. Here is a perfect example. A customer has a box that detects malware in email attachments. It flagged a PDF as highly malicious. We decided to check it out and at first glance it looked very bad. It had all the classic signs of an exploit, heap spray, etc. You couldn't tell the difference between it and another verified malicious PDF. However, upon further inspection we discovered that a popular autocad type program generated legitimate PDFs that looked this way. This is something that is not automatible. You must have an experienced and skilled analyst to do this. No amount of rack mount, fancy logo appliances will help you. And the bigger your enterprise the more you need. Every enterprise block of 30 - 50k IPs needs a team of 5 - 10 people.

Which leads me to the next issue. How you perceive your staffing resources. Example: One company I saw told they had a staff of 12 analysts to deal with security detection and response. I thought wow pretty good! Lets break the team down:

A manager, full time in meetings, paperwork, etc.

An assistant to the manager, secretarial work, etc.

3 senior advisers, i.e. guys about to retire, smart guys who give great advice and hold institutional knowledge, but not analysts

5 people involved in tool testing, stand up and maintenance (all those boxes I mentioned before). Great guys, not analysts or really involved in analysis

1 Developer mostly focused on designing queries and interfaces for the tools.

1 Actual analyst.

While management believes they have 12 people and doesn't understand why things take so long they actually have 1 person. This situation is very common in big companies. 1 good analyst for an enterprise is not NEARLY enough. And you can't be reliant on a specific person unless you want to set yourself up for a disaster (while at the same time you must cultivate and care for those star players).

That's my case for why staffing is the most important issue we face in computer security. What is the solution? Some would say training, but lets be honest, were you back home writing rootkits for work after taking Hoglund and Butler's class at Blackhat? Probably not. Have you found piles of valuable 0day after completing Halvar's most excellent course in Vegas? I doubt it. A 2 day - 1 week course isn't doing it. Going through the entire SANS curriculum isn't doing it and CISSP sure as hell isn't doing it.

You have to spend around 6hrs a day, after work, highly focused on coding, reversing, etc. for a minimum of 2 years to be decent. That is how the adversary does it. That's how the big name researchers and best staff does it, and unfortunately you only need a couple of attackers for every 10 defenders out there.

16 comments:

What I have found is that you need the right kind of learner. They don't have to be autodidactic to the point of Aspergers. What hiring managers need to acquire is a mix of individuals who not only learn differently, but also view risk differently.

There are three frameworks I use to judge these learning-capabilities and risk-perspectives. The first, for learning, is the Howard Gardner theory of Multiple Intelligences. The best information security professionals will be "on the map" in terms of intrapersonal learning, just as their hiring managers will excel at interpersonal learning. The other frameworks I use to understand a person's risk-perspective is to model using the OCAI framework as well as the Competing Values Framework.

Personally, I learned about Multiple Intelligences from Mercury Interactive's train-the-trainer programs (having been involved with many TTT programs at Cisco Systems in the late 90s/early 200s), which has extended in the HP product/services (disclaimer that I currently work for HP). I learned about OCAI and the Competing Values Framework from the Krag Brotby book on Information Security Management Metrics. I've been searching for equivalent work in other related fields, but feel somewhat unimpressed/underwhelmed by books such as Pre-Employment Background Investigations for Public Safety Professionals.

The most difficult information security expertise area that I have found (besides the communication, instructional capital, and individual capital problems that you describe so clearly -- which really become issues of human resource organizational behavior and organizational development) has been with the merging of the fields of application development and full-scope penetration-testing. Application development has its own set of mirrored problems for instructional/individual capital. I think most of these application development issues are described well in the book, "Emergent Design: The Evolutionary Nature of Professional Software Development". This book specifically mentions problems in the software profession: 1) Lack of a specialized language, 2) No clear path to entry, 3) Little, if any, peer-review, and 4) No authoritative standards and practices. Sound familiar?

Valsmith, I really value your views on operational penetration-testing. I think one of the most underrated expertise areas in our field is actual IT/Ops "in-the-trenches" experience, which only comes by doing simply just that. You can study and play all you want for 2 years or however long, but until you see the cogs move and how computers stay up (the "A" in CIA) and perform correctly -- you might be completely lost compared to someone with this exposure.

The book, "IT Security Metrics" from Lance Hayden has an interesting chapter which covers staffing. Chapter 9 on "Measuring Security Cost and Value" goes over how to staff to the number of incidents (based on probability using Poisson distribution), as well as how to make sure that the necessary skill-types/man-hour fit into the business operational processes.

I agree that hiring managers and HR departments are often looking for skill sets that are extremely rare. It's not necessary to find individuals that have broad experience with as broad a range of tools and responsibilities as you laid out though. It's more realistic to acquire and develop staff that are specialists in some areas and generalists in others and build a culture and processes that allow them to function well as a team.

I'd argue that traditional soft skills, critical thinking, and prioritization/negotiation are the biggest gap areas for those entering the information security field. Universities, conferences, and people that have an opportunity to emphasize this need and develop those entering the field or are preparing to are doing an inadequate job. It's a complicated problem and we're our own worst enemy because it's not something most of us want to dedicate our time towards addressing proactively via blogs, conferences (not that it would appeal to many since it's not sexy), podcasts, outreach, etc.

Dude! I think you nailed it! Thanks for this great blog post! I will definitly share it as I think it's a great wake up call for the infosec community / security industry. I think it provides some really good advice for many us to build ourselves a roadmap on how to better use our spare time in the next months/years ;-)

I have been invited for an interview at Dell SecureWorks next week. There are x5 vacancies. I worry the whole team just left because of problems you describe. I am not a security expert and actually turned down the interview, but they want me to interview anyway. Your post helped me quickly gain an understanding. I am out of my depth (and admitted so) yet intrigued to interview.

I agree with you but I think there is another perspective that you are missing. I worked as a pentester for small companies and have never worked with/for a company with a 12 person IT staff much less a 12 person security staff. Most of the companies I work for lack the infosec maturity to even have a pentest, but they are required to/want to for a myriad of reasons. For these companies, doing deep packet analysis, writing 0-days, etc doesn't make sense because they don't have anyone on staff that would even understand it.For these companies, points 1, 2, 6, and 8 become vitally important while the others tend to fade a way. So, yes, the problem in infosec is a lack of good people but no, you don't have to spend 6 extra hours a day training to be a unicorn to be relevant or useful in the infosec industry.

Great post. I agree with the majority of everything here. However, I think we are approaching a point in which its not feasible to have as many full spectrum experts as we currently do and still be effective.

I compare this to medicine. Less than a hundred years ago, you simply had "doctors". A doctor was a general practitioner, an internist, a surgeon, an ENT, and even a dentist. The scope of everything you could know about medicine was small enough that individuals could have all of this knowledge and do all of these things.

However, as our understanding of the human body grew, it became impossible for individuals to gain all of the necessary knowledge and experience in every area and still be effective. As a result, specialities came about. Effective patient care now relies on the combined efforts of specialists.

That said, what do physicians have that we don't? Training standards, licensing requirements, residencies, and things like that which are all very effective. Every doctor has to get a baseline of skills required for all specializations, and then they complete further training for their speciality. Not only that, they are trained in how to effectively work with other specialities.

Unfortunately, I'm not sure if our industry will every accept anything like that.

"we are approaching a point in which its not feasible to have as many full spectrum experts as we currently do and still be effective"

@ Chris Sanders : An excellent point, and one that I think about a lot now that I have somewhat of the perspective about how it would even be "possible" to divvy up the work that would normally require a full-spectrum expert.

In my mind, you actually DO need a full-spectrum expert as valsmith describes. However, that person doesn't have to be on HQ (e.g. US/Canada or UK/EU) soil and doesn't have to attend official meetings. They can be offshore and provide the partial automation and partial analysis necessary to pass that info on to a full-spectrum and involved analyst.

For example, there is no way that I could pass on any of my job to an offshore person unless that person did understand network, operational, and app pen-testing, and be able to utilize sufficient HTTP/TLS/TCP/IP/Unix-systems knowledge along with sufficient object-oriented programming knowledge (as well as the specific target managed code frameworks, underlying patterns and libraries involved, etc). This person may not need valsmith's number 1 ("soft skills") but they would definitely need 2-8. In fact, I may even add some to his list!

There is a reason that full-spectrum analytic capability is required. It all comes down to combinatorial explosions. If you can't analyze the issues (e.g. issues between HTTP/TLS and the OO apps that service that type of traffic) and how they relate, then there is no way to take Issue A (from HTTP/TLS space) and relate it to Issue B (from OO space) in order to uncover Issue C, which may or may not lead to Issues D,E,F,etc. Note that this applies to pen-testing, incident handling, and really any technically-focused activities in infosec.

Scaling and dividing individual capital is the real challenge here. This actually is a hiring manager problem, but this does not obscure the "biggest problem", which is acquisition and retention of these nearly non-existent full-spectrum analysts. You can't scale or divide something that you don't have in the first place.

Your post was an excellent litany of the problems we face as an industry – no question. But, like most litanies, it’s a set of supplications – wouldn’t it be nice if… Well, yeah, it would great if things were better. Also, if your aunt had balls she’d be your uncle.

Don’t get me wrong, your points are spot on. Ultimately, they distill down to your unicorn argument: hyper-technical, divergent, but fractally similar skill sets in an individual are a rare a precious commodity; market driven scarcity defined!

So, when the stakes are high, and markets confused, solutions unclear, what is one to do? It simple, breed unicorns.Since you live in Mordor, um, New Mexico, we can look a set of solutions that were tried out in your neck of the woods that history proved workable. When faced with scarcity for technical knowhow, Kelly Johnson and Robert Oppenheimer redefined the resource, protected it ferociously and, in the face of mind-blowing secrecy grew cultures that attracted and retained the best minds in physics and aerospace the world had ever seen. The obviously question is, how did they do it? For Johnson, he defined his 14 management rules. He built and demanded an environment of excellence using positive peer pressure and a hard-core nerd culture that naturally arrived at solutions to hard problems. Solution had to be based on elegance, usability and consensus. Everyone recognized when a hack was the answer, they sensed when a solution to a requirement was optimal. If it wasn’t, or the technology didn’t exist, they invented it, put it on a bird and flew it. If you made the cut to join the skunk works, even as a junior member, that culture drove you to the limits of your ability and beyond.

Oppenheimer was a different story, unlike Johnson who ruled with an iron fist; he had to answer to Leslie Groves (The very definition of a hard ass major general). Faced with an intensely smart team (multiple future and current Nobel winners) an impossibly hard problem, and the fate of the free world on his shoulders he did what needed to be done.

Imagine that you had a project team of 200 people, when you finally wrap your mind around the magnitude of the task; you realize you need to grow that by 30x, in the next 24 months!

Reading Hans Bethe’s book you find that Oppenheimer was a hands on manager. He understood the nature of the work in detail, he had a mind that was able to understand the technical details of the sub-tasks in a project and most importantly, he knew how to integrate that knowledge into the whole.

That talent in a manager is as rare as a fucking magical mite riding on the back of a gold bug stuck in the ass crack of a unicorn!

If that wasn’t enough, he also insulated his team from the hierarchical structure of the US Army during war time. He created a team of spectacular geniuses, with egos to match. Egos, that where well and truly justified. (How many people do you know that have fundamental constants or families of sub-atomic particles named after them?)

Answer 2: Create a culture of technical exceptional-ism: Make your people earn entrance to it. Protect them from false hierarchies; take bullets for them. Settle for nothing less than extraordinary. Understand your business and your customer’s; drive solutions that fit them flawlessly. Find genuinely smart people --even outside your industry, listen to their insights. Finally as a leader, be present in every sense of the word. Present physically where the work is done. Present when listening to customers, employees and advisers.

Great article. Also, great comments. The little i've learned so far about infosec, spending close to 20 years in ict, repeatedly being ahead of the curve, gutfeeling/vision. None of this matters. Ego's are huge where I live, often cause for 'mistakes'. For far too long emotionally challenged personality's defined who got a chance. Often from backroom conversations.