Interview with a Ninja, Part II

Last month, I introduced my good friend Ninja G, an accomplished
professional penetration tester for a large financial institution (who
also happens to be a practitioner of ancient martial arts developed by
ninjas). We talked about his daily activities, his unusual career path, his
perspectives on WLAN and Linux security, and the uncanny preoccupation
among Ninja G and his coworkers with ninjutsu.

This month, I wrap up the interview. I think this installment
is every bit as wide-ranging, thought-provoking and entertaining as
part I. I hope you agree!

MB: I've got to ask a somewhat selfish
question, because so much of
my own career has involved firewalls. In the age of Web applications,
where the browser is the platform and so much of the world's business
is conducted over TCP ports 80 and 443, do traditional layer 3 firewalls
still have a useful role?

NG: I believe so; however, I think most people would agree that a
traditional (state-aware) firewall by itself isn't a complete solution. It
is only the first of many layers of defense. Intrusion detection (and
prevention) systems, system hardening, secure centralized logging and
application-aware firewalls should supplement a state-aware firewall to
help round out areas where it would otherwise be lacking.

MB: What's your opinion of the new generation of (Web) application
firewalls that can be trained to block exceptions to
“expected” Web
application behavior? Are we finally reaching a point in this class of
technology where it's possible to create a complete enough notion of
“expected” application or network behavior to reduce the very high rate
of false positives we've traditionally associated with this approach?

NG: I don't think I ever would recommend a Web application firewall
in lieu of Web application hardening. A lot of the problems that are
being caught by Web application firewalls usually can be solved with
proper tainted variable checks on all user-supplied input. This
includes the Web browser's user-agent string (which may find its way into
log files), all cookies and, of course, GET/POST data, AJAX requests and so
on.

In an ideal world, the Web application should provide real-time logging of
things like cross-site script, SQL injection and brute-force attempts,
but that is more often the exception than the rule. Often organizations
rely on (SSL breaking) Web application firewalls to obtain the same types
of data; however, you have to be very confident in low false positive rates
when deploying in any inline device that actually prevents malicious
traffic from reaching the target host, or else you could be creating
your own denial-of-service condition for what could be business-critical
applications. One should keep in mind that even the top-of-the-line
Web application firewalls occasionally will generate false positives
and plan one's use of them accordingly.

MB: Some people are very skeptical not only about the worth of
pen testing, but also especially of the trustworthiness of anybody who has
amassed experience in system cracking, which they (the skeptics)
seem to think is inherently corrupting. But my own experience has been
that overall, security researchers and penetration testers, precisely because
they understand so well how easy it is to get caught, tend to be very
responsible.

And, I'd unhesitatingly put you in that category. I think of you as a
highly ethical and trustworthy person. So, what's your take on this? Are
security ninjas generally like you, or is criminal recidivism (à la
Alberto Gonzales) rampant?

NG: Of course, I can't speak for anyone
else but myself. People
have their own unique motivations, passions or “demons” that drive them
along this path. I can say it has been my experience that most
people who perform penetration testing for a living generally wouldn't
risk their livelihood by hacking illegally. For a person like myself,
I view my work almost like a flip-side version of the Buddhist concept
of Right Livelihood, which states that people should try to find an
occupation where they won't sacrifice their moral code. Except for me,
I found a job where I get to express myself creatively in a criminal-like
way, yet with the most ethical of outcomes.

At the end of testing, I create detailed write-ups of all of my security
findings, write proof-of-concept code, help vendors understand and
re-create these issues, and assign a risk rating to help others understand
their severity—like a lot of the other “unsung heroes” of this field,
who never release their exploit code and provide full details directly
to vendors and code writers. I would like to think I am helping
improve the overall security of not only the company I work for,
but also indirectly helping all the other companies who use the same
systems, services and products.

MB: It's easy for someone in my position (and that of my readers, I
hope!) to see the value of your methods and your reports. I don't think
“unsung hero” is any exaggeration, having seen firsthand how tangibly
things tend to be improved after an unfavorable penetration test. But how do
vendors and developers react to your work? How do you get them past what
I'm sure tends to be an initially defensive reaction?

NG: Yes, you're correct. The typical
initial response is to view
the request for a penetration test as simply a check-box item on a list somewhere,
which may or may not prove any sort of real due diligence. My assumption
is that this is their usual experience with dealing with other companies
with limited information security personnel and budgets.

Once they actually start hearing about high- or critical-severity security
vulnerabilities, they almost always shift to a defensive position. At
this point, I give them my (somewhat canned) speech about how they are in
essence getting weeks of free security consulting. I assure them that I
will share the details of all security findings completely and recommend
steps for remediation where appropriate.

It is usually around this point when they realize that although their
experience may be painful or humbling, they really are getting something
for nothing, and far better to receive the information in
this way,
rather than having the same issues discovered by others who may not give
the vendor any sort of warning before releasing the information
to the general public.

MB: In my own consulting work years ago, now and then I'd be called on
to do port scanning or security scanning (though not actual penetration testing)
of live, production systems, and I always found that nerve-wracking. Have
you ever inadvertently (or, come to think of it, intentionally) disrupted
an important production system?

NG: I once took down a large group of firewalls simply by port
scanning them. This was completely inadvertent, as I didn't know that
they had been configured to effectively “turn themselves off” when they
detected heavy port scanning. (?!) This caused a large production outage
(and several people were howling for my blood), yet the configuration
was beyond foolish, so by helping to “correct” the issue, I managed to
escape unscathed.

I have never intentionally disrupted an important production system. Even
in my lawless years, the goal was never to “take down” a system. I
viewed that action as providing a very clear sign that
an intrusion had
occurred. The end result would be one less system to explore, which I
viewed as sub-optimal.

MB: There still seems to be tension between two camps in information
security: those of us who self-identify as hackers and those who
don't. Yet attendance at Black Hat Briefings, DEFCON and other hacker
conventions continues to balloon. Do you think the “hackers vs.
suits”
situation is getting better, or are we simply gaining numeric superiority?

NG: I would say that there was
much more tension a decade or two
ago, back when the corporate world wouldn't dare ever send someone off
to a “hacker conference”, let alone pay for it. Now conferences such
as Black Hat and DEFCON are considered valued sources of information for
those defending rather than attacking corporate resources.

Personally, I think there is a rather new element at hacker cons that may
soon upset the delicate “hacker/suit” ecosystem: the US
military—and
I'm not talking about their old guys, I'm talking a new wave of young
military. So don't be surprised in the coming years if that guy sitting
next to you at DEFCON happens to be a SIGINT analyst for the US Navy.

MB: I've already had that type of experience, and agree with your
observation about the military (which, I supposed, tends to be a youthful
demographic, so maybe this isn't too surprising). DEFCON's “Spot the
Fed”
contest stopped being easy ages ago!

I think I'd like to wrap up with some more or less random
questions—feel free to answer as tersely or completely as you like
and, of course,
omit names to protect whomever you like!

MB: What's the stupidest design decision you've ever seen?

NG: Okay, this one goes back way over a decade, so I feel comfortable
discussing it. I once saw a large financial organization solve their
backup issues by installing FIDDI interfaces in all of their UNIX
servers. They dual-homed all of the machines into one large FIDDI ring
and then kicked off all of the backups via rhost. I rooted
one
backup server in the FIDDI ring and ended up owning the entire back
office settlement environment across the globe in minutes, due to rhost
trusting the backup server. Keep in mind these hosts' primary interfaces
were segregated into protected zones by firewalls, and access was limited
into each zone based on your job function.

Epic fail.

MB: What's the coolest security control you've encountered lately?

NG: People are probably sick of hearing about it, but DNSSEC is
actually pretty cool, especially when you factor in the potential impact
associated with traditional DNS. (Yes, I have drunk Dan Kaminsky's
Kool-Aid!)

NG: Hmmm...for this question, I believe a little bit of ninja lore
is necessary. First, forget everything you know about ninjas being the
bad guys dressed head to toe in black uniforms; rather, they were often
highly skilled martial artists acting covertly in enemy territory much
like a modern-day spy. The worst possible fate would be to be caught,
so contrary to all of those bad ninja movies of the 1980s, escape was
always way more desirable than fighting.

So in the study of ninjutsu, you will encounter things like the Santo Tonko
kata (forms of the escaping rat), which includes techniques for escaping
grabs, the use of eye-blinding powders and using small thrown objects
(such as shuriken) to dissuade continued attack or pursuit. Nowadays, a
small tactical flashlight can be used instead of the irritating blinding
power, and as shuriken have been outlawed in most states, a pocket full
of loose change does the same thing when unexpectedly tossed into the face
of a would-be attacker. “Oh! These are a few of my favorite
things...!”

In one particular situation, I was greatly outnumbered, so I decided
I needed to even the odds a bit. Using a Surefire E2E flashlight, I
blinded the advancing “front row” only to have my
“fighting with light”
completely misidentified as a police tactic. Next thing I heard was,
“This guy is a cop!” and everyone scattered.
That alone was worth
the price of the flashlight.

So rather than doing something silly or dangerous (like arming your loved
ones), instead get them an inexpensive high lumen output flashlight. If
their attacker disarms them, it isn't anything that could really be used
against them. Plus, you would be surprised how often a flashlight comes
in handy, even when you aren't being jumped by a large group of people.

MB: Which is harder, being stealthful in meat-space or being
stealthful in cyberspace?

NG: If you understand both environments, then one really isn't more
difficult to manage than the other. These are not set in stone, but I
generally think of four basic principles that apply fairly equally to
both physical and cyberspace stealth considerations.

The first principle would be disguise. In physical space, this includes
concepts, such as wearing appropriate clothing to blend in with the
rest of the population in the area, and the use of multiuse, hidden
or improvised tools/weapons. In cyberspace, this would include avoiding
the generation of network traffic that would be typically associated
with either reconnaissance or attack. Instead, the stealthy attacker
would choose to generate completely “normal” network traffic that could
also be used to glean useful information.

A good modern example of this would be the tool named FOCA, which uses
search engines to identify the location of documents that commonly
contain metadata. The tool downloads these files a few at a time and
then rips out the metadata and presents the extracted data. All of the
generated Web traffic is 100% normal. All of the documents downloaded
were intended to be downloaded. Unless a company is militant about
scrubbing metadata, it isn't unusual to find about one file out of
100 that actually contains something useful enough to be leveraged in
further attacks, such as an employee name, e-mail address or user ID.
With larger companies, it is not uncommon to discover thousands of
metadata-laden files available for download.

The second principle would be distraction (or attention control). In
physical space, there are lots of variations here, as the human mind is
very limited. We may claim to multitask, but the reality is that we
can think of only one thing at a time. All you have to do is fill the
mind with something interesting, and it will miss everything else. A
good example of this can be found on-line by Googling Daniel J. Simons'
video titled “selective attention test”. In cyberspace, the same effect
can be achieved by intentionally generating known malicious traffic to
create overt “noise” in which to hide. Tools like snot and stick
were designed to do exactly this sort of thing.

The third principle would be exhaustion (or frustration). In physical
space, a modern analog would be to repeatedly set off a building's
burglar alarm. The first time it goes off it usually will get all sorts
of attention. The police will show up first, then eventually, the building
manager will arrive on-site. After about the fifth or sixth time (in the
same night) of the alarm going off with no visible signs of attempted
forced entry, that alarm is going to be turned off until morning and
a repair call will be made to the burglar alarm vendor.

The exact same thing happens in cyberspace with intrusion detection
systems that constantly generate alarms that seem to be false positives. A
security-conscious person will put up with an interruption or two in
the middle of the night, but nobody can tolerate unwarranted nonstop
interruptions. Discover this “breaking point” limit, and you can help
“tune” remote intrusion detection systems to be more
“friendly” toward
your future stealthy endeavors.

The fourth principle I would call fu-sui (wind and water), but most
people know it as feng shui. This is intentionally selecting the
most advantageous positioning, terrain, weather or timing. In physical
space, this could be something as simple as choosing to attack at dawn or
dusk with the sun low in the sky behind you. Your enemy will be staring
directly into the sun allowing you to hide in bright glare. Another
example would be to choose a foggy night to hide in the darkness and
mist. In cyberspace, this could include performing your activities during
holidays celebrated by your target, choosing late hours when monitoring
personnel may be more scarce or choosing peak traffic hours if you are
trying to “blend in” with normal activities.

These principles are not mutually exclusive; in fact, it is often better
to blend them together when creating your plan of attack.

MB: When does the fun of Brute Force trump the righteousness of the
Elegant Solution?

NG: Wow, that definitely applies to both
penetration testing and
ninjutsu: “When it is the quickest and most direct solution to the
problem.”

MB: Who's more
elite, pirates or ninjas, and why?

NG: Wow, that is the
eternal debate. Here is my take.

If you research the origin of each of the nine martial arts that I study,
you will find that one originally was used as a naval military art, and
as a result, the body positions and movements were designed to be used
on ships that were constantly rocking and slippery. Even to this day,
techniques from this school are sometimes practiced using a boat oar
as a weapon. This means that historically, some ninjas were pirates;
however, I seriously doubt that the contrary was ever true. I'm sure this
won't surprise you, but I vote ninjas!

MB: And on that lighthearted note, I'll sadly but gratefully wrap up
what I hope, dear readers, you agree was a fun and informative
conversation. Thanks so much for playing, Ninja G!

Mick Bauer (darth.elmo@wiremonkeys.org) is Network
Security
Architect for one of the US's largest banks. He is the author of
the O'Reilly book Linux Server Security, 2nd edition
(formerly called
Building Secure Servers With Linux), an occasional
presenter at
information security conferences and composer of the “Network
Engineering Polka”.