An incident response team is like the Avengers

Today, after leaving a meeting, one of my junior analysts said “I guess we need to lower the threshold for declaring an incident.” Intrigued, I asked what brought him to that conclusion. He said “well, our incident response plan says that we organize the team if more than ten computers are affected, but you just said that an incident is any event that indicates harm or malice.” Realizing that further clarification was required, I resorted to my favorite tools: analogies and metaphors.

I gave the example of Captain America hearing a car alarm going off. This is just an event, and not yet an incident. Why? Surely if a car alarm is going off, something bad is happening, right? No, not necessarily. It could have been set off by the owner by accident, by a teenager zipping by on his skateboard and accidentally slamming into the car, or another car bumping into it while parking. While these are all (noteworthy?) events, none of them represent an incident in the security context because nothing really bad was happening, and there was no evil intent.

This is the equivalent of an analyst detecting an alert on the corporate SIEM. The analyst notes that there are a large number of failed logons occurring on a system. Investigating, he finds that they are all originating from a single workstation, which was unable to log on due to a recent password change. Is this an incident? No.

Taking it to the next phase, let’s say that Cap does investigate, and he sees that the alarm was set off not by some passing skateboarder, but by a common street thug. Does Cap yell “AVENGERS, ASSEMBLE!?” No, not yet. Some two-bit thug trying to jack a car is well within Captain America’s ability to cope with without needing to call upon other members; you don’t call for Thor and Iron Man just because you’ve got them on speed dial, and Cap knows that.

This could be likened to the analyst detecting malware on the affected system. Perhaps this malware was trying a brute force attack against a random system on the network. The analyst tasks the antivirus software to re-scan the system and do a cleanup, which it does successfully. Problem solved, with no need to bring on additional help.

Even if the guy breaking into the car turns out to be Batroc, Cap doesn’t go crying for help, because he can has handled the likes on his own many times in his extensive career. But what if Batroc is not alone, and is accompanied by his brigade of super villains? This is where it gets a bit subjective. Being a seasoned incident responder, Cap is likely to attempt an immediate intervention in their nefarious activities, even though he’s outnumbered. However, if he starts to realize he’s being overwhelmed, Captain America is not too proud to reach out to a nearby resource like Bucky, the Falcon, or Spider-Man if he’s nearby. Since he has an established relationship with many other super-powered resources around the world, help is just a quick call away.

The analyst continues to investigate, and finds that there is malware on the affected system, and also notices alert that other nearby systems are apparently infected with the same malware. Contacting the department IT liaison, he finds that they are aware of the infection, which was passed around by a shared USB drive. Working together, the department contact and the analyst are able to clean up the malware before it spreads beyond the handful of machines.

But, let’s say that Cap notices it’s not just an ordinary thug, and not even just Batroc’s Brigade, but it turns out that, for some reason – don’t ask why – it’s Ultron who has decided to steal this car. As a villain who has tried to destroy the world, and come awfully close to doing so, this is immediately dubbed a very serious incident.

Further investigation reveals that multiple systems have been infected, and are communicating to an external IP address in a remote location. There is now evidence that classified data is being exfiltrated from the network. It’s time to call on extra help. The incident response team, consiting of members of the networking, server administration, and department IT teams, are notified, and a command center is established. The team immediately begins to compare notes to determine what is going on, and how to contain the threat.

This is why Captain America’s got not just Bruce Banner’s cell phone, Nick Fury’s SUV phone, and Tony Stark’s office number, but also Pepper Potts’ pager (because she’ll be able to get a hold of Tony, regardless of what lady has caught his attention this evening). Cap also has a plan, which directs him to notify the regular members of the team.

This is why the incident response plan must be regularly updated to include a current list of contacts, the systems for which they are responsible, and their best contact methods.

If it turns out that things are getting really bad, he may need to call upon a specialist. For example, since Ultron is (these days, anyway) made largely of vibranium, it may make sense to call upon someone with extensive knowledge of the precious metal – good thing Cap kept T’Challa’s Whatsapp handle at the top of his favorites! Now the Black Panther is only a hop, skip, and a jump away from joining the team. If other threats are encountered along the way, say, it’s found that Doctor Doom may be collaborating with Ultron, more help – in the form of the Fantastic Four – may be called upon, as they have extensive experience dealing with Victor von Doom, and can likely help contain the threat in a timely fashion.

The organization maintains relationships with law enforcement, their ISP, and outside security firms who can supplement the incident response team if required. Calling on a professional services firm to assist with containing an advanced persistent threat may be called for, as well as notifying law enforcement or various government agencies, depending on the nature and scope of the breach that has occurred.

Is every day in the life of a security analyst as exciting as that of an Avenger? Probably not. But it can be fun, at times, especially if you like being a hero. If you think you have relevant skills or information that could be useful in a security incident, let your local security team know, and ask how you could be of help during a crisis. Who knows? You may just get a snazzy, new ID card to carry around.