Interview with a Ninja, Part I

It's a strange world we live in now, isn't it? We communicate through
machines as often, and with more different people, than we do in person.
Some of us have careers that didn't exist when we were in college. Some of
us even have careers that involve doing things we used to do secretly,
possibly even illegally.

Consider the professional penetration tester, who breaks into his clients'
systems and applications for the purpose of testing and improving their
security. To me, this is one of the most remarkable professions a
person can have in information security.

Some people are skeptical of the value of this line of work. What does it
prove? If penetration testers succeed, have they really learned more
than, or even as much as, they would have learned by performing a careful
analysis of system and application configurations, and maybe by analyzing
some source code? If penetration testers fail, did they simply have a bad
day, or is the system in fact secure?

Although I'm sure there is such a thing as a redundant or otherwise unhelpful
penetration test, I can assure you that in my own role as a network
security architect, there are plenty of situations where a rigorous,
professional penetration test is essential. Source code isn't always
available, configurations don't always tell the full story of system
behavior, and some things are too new for there to be any “conventional
wisdom” of how secure they are, or how they can be made secure. A good
penetration test can help me determine which of a vendor's claims hold
up, and which system administrator practices are being adhered to.

Therefore, I work closely with and proudly count as friends some
outstanding penetration testers. Luckily for you, gentle reader, this
month,
one of them, who for our purposes here I'll simply call Ninja G (for
reasons that will be revealed shortly), graciously agreed to a Linux
Journal interview, in which we discussed his singular career, what he finds
interesting in computer security, and what he sees as notable trends in the
current state of Linux/UNIX security.

MB: Thanks so much for taking the time to
chat. By way of an
introduction, can you explain what your official duties are and how you
spend a typical day?

NG: I work in the finance industry for a Fortune 500 (or less) company.
The duties of those in my group include penetration testing, enterprise-wide network security scanning and real-time security event response. My
days vary greatly from one week to the next, which is a wonderful thing
about this sort of work. One week I am testing a small embedded Linux
device, the next week it could be a remote server or Web site, corporate
Wi-Fi or VoIP network.

I believe this classifies me as a generalist, so it probably isn't
surprising that a large part of my days are spent coming up to speed on the
target platform. This can include securing logical or physical access,
reading all the vendor-supplied documentation, researching all currently
known and historical vulnerabilities and then actually using the platform
or device (as intended) to gain a holistic view of how things normally
work. Only then can I start to ask myself all of those evil little
questions concerning how the designer decided to handle situations that
(under normal circumstances) were never intended to happen.

MB: How did you arrive at this kind of a job? I'm assuming you
didn't major in penetration testing in college!

NG: College? Heh, no. I was an illicit computer/phone hacker during the
1980s and early 1990s. I don't have a criminal record, but I can't say that I
was never caught; rather, at one point I was the focus of a multi-agency
investigation. I never was prosecuted, but my “close call” officially ended
my desires to intrude illegally into systems as a hobby.

It was at that point that I started looking into jobs in the Information
Security field. I was hoping to find an outlet where I could continue the
passion of my past hobby, yet without the risk of prosecution. The problem
was trying to find just the right company that not only had an Information
Security department, but also found value in beating up proposed solutions
to look for potential security problems. I knew that large financial
organizations had an interest in this, but unfortunately, I lived in small
town, which was exactly the wrong area for this type of work.

So I “followed my bliss” and ended up in a large city, working for a
financial company whose products are offered globally. I wouldn't say that
my first job in this industry was exactly what I wanted, but often
your job is what you make it, so I slowly started revealing my skill level
in the arcane “hacking” arts. Word gets out, rumors spread, and very slowly
more and more of the work that interested me started to come my way.
Fast-forward 16 years, and I'm working for a larger financial company in an even
larger city. Now the bulk of my time is spent on penetration testing, so I
can't complain at all.

MB: What do you find most interesting in your job and your career?
What sorts of problems are the most fun, and why?

NG: Learning new things every day. Difficult problems are always the
most rewarding.

In this line of work, you are only as good as your last major find, and
people tend to forget quickly about past (now-mitigated) vulnerabilities.
It isn't uncommon to go for days without stumbling upon new vulnerabilities
of any significance, which is horribly
depressing. Perseverance at
these times often cuts deeply into my own personal time, but it almost
always pays off.

All it takes is discovering one new zero-day (vulnerability), and I'm back
on top of the world—for a day or so. Then, the process starts all over
again with something else.

As a generalist, I really don't have any preference on what types of
devices, software or systems I test. Each presents their own lessons
and potential for discovery.

MB: WLAN, as a target, seems to be especially popular in the criminal
community. Does WLAN stand out, one way or another, in your own work, or is
it just the same types of problems and vulnerabilities, only over radio
waves?

NG: No, I would say that wireless presents its own unique set of
security ramifications and even heavily trained corporate WLAN
administrators sometimes don't comprehend concepts as simple as malicious
server certificate re-use. If you're unfamiliar with this style of
“insider” attack, it can be summed up in one sentence.
An authorized Web server administrator creates a malicious (evil twin)
wireless access point that mimics the corporate WLAN (and which they load
with the valid certificate from their Web server from the same
name-domain); however, they configure it to merrily accept (and log) all
authentication credentials provided, regardless of what they are.

For those wanting to play with this sort of attack, Josh Wright's WPE
(Wireless Pwnage Edition) patch to FreeRadius is a nice shortcut to RADIUS
server impersonation. Of course, back when I was first playing with this,
that didn't exist, so I manually patched HostAPd to do the same thing. In
more sophisticated attacks, I have “Frankensteined” a modified HostAPd to
Xsupplicant, and successfully performed full man-in-the middle attacks
against WPA2/AES with hardware one-time password tokens.

There is also a bunch of older attack code floating around that doesn't
really work well anymore, as it was written before QoS was a common feature
in access points. I have updated a lot of this code for my own use, plus I
have written several of my own wireless attack tools. My favorite sort of
wireless exploit to write would probably be best categorized as “blended
surgical attacks” where the real action happens at layer 7, and forensics
after the fact is next to impossible, as the attackers aren't connected to any
WLAN, nor do they ever expose their own MAC address.

MB: I've noticed that a lot of guys on your team study ninjutsu, the
martial art practiced by real-life (not metaphorical) ninjas. Is this a
coincidence? What's the ninja-hacker connection?

NG: Oh, so you noticed that! Probably just a coincidence. To be fair,
I would say that I study a system of Japanese martial arts that contains
nine different budo taijutsu koryu (old-school combat arts), only three of
which are classified as ninjutsu.

The connection? Well, I can't speak for any other martial artist, as
everyone
has his or her own reasons for pursuing such a hobby, but I do have a few
theories on why people who are interested in computer hacking or
penetration testing may be attracted to specific martial arts, such as
ninjutsu.

First, both hackers and ninjas benefit from the mythology spun around their
mystique. Often our labels for these individuals come along with a lot of
misconceptions, which usually don't exactly hurt the people being labeled,
as often they are attributed with near super-human abilities and skills.
(Win! Unless these labels are used in a court of law, then: fail!)

Second, as my teacher often says, “We must learn the techniques and tactics
of the bad guys, if we are to defend against them.” The same holds true for
anyone who is skilled at breaking or securing systems. Knowledge of what
the other guy is likely and capable of doing at any point can make a huge
difference.

During martial training we learn to do some rather nasty things
effectively,
and then we explore the areas of space between attacker and defender,
which, if occupied, could lead to either an optimal or sub-optimal outcome for the
defender. We “play” in this way until the mind starts to form almost a
three-dimensional model of what space leads to which outcome. My teacher
often says, “Move through the 'safe shaped' space!”, which is bewildering to
those unfamiliar with this concept. A lot of budo taijutsu training is
counter-intuitive like that. Which brings me to my last point.

I once read that people who play mahjong, or other puzzle-like games can
keep their mind sharp well into their autumn years—provided that the
puzzles are hard. I'm pretty certain that those who exercise their
minds by hacking are probably naturally drawn to other activities that are
equally stimulating and challenging on a mental level.

Of course, hacking problems are way different from budo taijutsu
problems, as they use completely different sections of the brain. When
hacking, you have the luxury of being able to stop and think for a bit.
Stopping occasionally to scratch your head, ponder and plot can go a long
way to moving you forward to your end goal. Now imagine that your problem
isn't breaking into a system, but rather surviving a knife attack. There
simply is no time to stop to think; you just have to move and trust that
your mental map of interactive space, balance taking, weapon usage,
striking and so on is sound.

Training is often chaotic, with the teacher varying the conditions (such as
weapon type or length, distance, number of attackers and so on) frequently. At
higher levels of training, things start moving and changing so quickly that
there really is never a time set aside to stop mentally to comprehend your
experience. So instead of having a cognitive understanding of exactly what
happened each time, you have only a general “feeling” of what is happening.

MB: I know that you work with Linux in at least two different contexts
in your day job: as a platform from which to conduct attacks and also as a
target, since so many of the things we ask you to test nowadays seem to run
on Linux! What sorts of trends, good and bad, do you see in Linux security?

NG: On embedded Linux systems, I often find the sorts of problems that
plagued larger UNIX systems decades ago, except now they can't be blamed on
a clueless or lazy system administrator. Instead, it is often an embedded
Linux developer who doesn't understand things like file ownership and
permission bits, temporary file name prediction, proper usage of hard and
soft links, command chaining, race conditions and exactly why it may be a
good idea to update that prehistoric version of BusyBox.

One trend I see is that some embedded developers have gone to great
lengths to attempt to enhance the security of the embedded Linux
environment (to varying degrees of success). Their efforts range anywhere
from a secondary password challenge-response scheme for root access, to
neutering root completely, to only executing signed binaries, to
removing all login services and shells entirely. A lot of these same things
hold true for non-embedded Linux appliances, where users are not generally
traipsing through the filesystem with shell access.

Of course, all newer Linux platforms have benefited greatly from some of the
more recent developments in memory protection. There is still
more work to be done, but I definitely would classify things like
executable space protection as a good trend.

MB: By saying “neutering root
completely” are you referring to SELinux
and other role-based approaches to security? The “root is
omnipotent”
aspect of Linux's security model has always been its soft spot, hasn't it?

NG: SELinux is sometimes used (usually in
full-size appliances);
however, you probably would be surprised at some of the one-off, homegrown
solutions that some vendors have implemented to attempt to rein in the
almighty “root” in embedded Linux solutions.

Some vendors simply decide that root has no real purpose as a user account,
so they cut it off at the knees (by giving it an invalid shell and a
locked-out password). The system still will function perfectly, but whenever there
is a need to upgrade system files, it usually involves completely
reflashing the firmware for the device.

Other vendors choose to keep the root account, and then build their own
mechanisms where upgrades are handled by root, but practically everything
else runs as a nonprivileged user. In theory, this is a good approach, but
again, even a simple misunderstanding of how hard and symbolic links work
can spell disaster for upgrade routines that expect only vendor-provided
(properly formed) upgrade files. If you can predict ahead of time where a
temporary file used in the upgrade process is going to exist (say, /tmp),
and you get there first with a link, many times you can end up clobbering
the target of your link as root. If you get lucky, the script also may have
a more-permissive umask setting, which would end up modifying and maybe
lowering the file permissions to something less than before.