Doug Brown, former Manager of Security Resources, University of North Carolina at Chapel Hill

Stephen Northcutt - October 30th, 2008

One of the important concepts that we want to explore in
security thought leadership is the idea of group or team thought
leadership. And so we are looking for examples of teams that exhibited
security thought leadership. Doug Brown, former Manager of
Security Resources, University of North Carolina at Chapel Hill, was on a team that exhibits
many of the characteristics of security thought leadership.

Doug, can you introduce us to the team that you served on?

Thank you, Stephen. From 2001 until 2006, the University of North
Carolina at Chapel Hill had a strong team of collaborative players in
their Information Security and Networking groups. These were the early,
wild-west years for much of what has now become Information Security
canon. Inventing as they worked through each day’s new
challenges, this team happily shared their successes with colleagues
and also the few vendors who would listen.

Did this team have a Code or Guiding Principles?

As a matter of fact, we did; a few of our main guiding principles were:

Protect the University, protect the data, and protect the systems.

Security that is onerous or cumbersome is security that gets circumvented: always balance risk, efficacy, and transparency.

Clearly and frequently communicate the case, build consensus with the customers, and move forward to advance the cause.

Not having all the answers is not a mission ender; leverage all
the available resources and friends, because someone else has already
likely solved at least part of your problem.

Education and proactive measures provide the best ROI.

As I understand it, UNC was one of the early adopters, like VA Tech, of
using VLANS to increase Security and manage systems on the network. I
believe this was called SecureFast, is that right?

In 2001 as UNC's Y2K office transitioned into their Information
Security office, an essential relationship of cooperation already
existed between the Security and Networking teams. Prior to a
subsequent migration to 802.1q, the University's network relied on the
proprietary VLAN architecture of Enterasys' SecureFast. Security
awareness was central to Enterasys well before any other networking
vendors. The SecureFast VLAN technology allowed UNC to implement two
fundamental tools to support their early security efforts; these were
referred to internally as the Penalty Box and Source Blocking.

SecureFast based VLAN membership on a device's MAC address rather than
its current switch port. This simple concept allowed UNC's Networking
and Security teams to quickly move compromised or infected systems into
an isolation VLAN. This VLAN did not allow any internal or external
communications: nefarious systems could no longer speak to each other
or any one else. It was often the case that the student or departmental
administrator would try a different wall port; however, the Penalty Box
was tied to the system's MAC address and any number of ports tried
would yield the same lack of communication.

Jim Gogan was a big hockey fan and coined the name Penalty Box. Borrowing from the movie Alien
we often said "In the Penalty Box, no one can hear you scream."
Occasionally an ambitious administrator would attempt replacing network
cards, sometimes going through several, and then other safeguards would
kick in to stop bad traffic and provide protection.

Can you tell us more about the drawbacks of the Penalty Box?

Placing systems in the Penalty Box was originally a manual process. The
Security office would identify the bad actors and refer them to the
Networking group for isolation. As the size and speed of security
events began to increase, one commonality became apparent: systems
attempting to infect others would generate insanely high numbers of
unresolved ARP requests as they scanned empty IP space for new victims.
SecureFast tracked these requests and UNC's team was able to add an
automated dimension to their security tools by implementing Source
Blocking. Those systems that crossed the threshold for a reasonable
amount of unresolved ARP requests were cut off from communicating. This
also had the benefit of catching those engaged in inappropriate
experiments with security tools.

UNC was one of the early schools that had a laptop requirement
during this time. Can you tell us a bit about that helped security?

UNC implemented an educational program requiring incoming freshmen to
have laptops meeting a minimum specification. This Carolina Computing
Initiative (or CCI) offered IBM ThinkPads with a University-provided
software load as one possible source for students to meet the
requirement. More then 90% of the students purchased these IBM systems,
and the opportunity to provide students a system image that was both
supportable and secure out of the box proved invaluable to both the
Help Desk and Security teams. This system image was reviewed and
tweaked twice each year by a committee of different interest groups.
Small changes, such as the addition of a centrally managed anti-virus
client and the shut down of unnecessary services, meant that the
student networks exceeded the availability and performance seen at
similar institutions that were not providing a desirable system image.

What did you do to educate the University population about computer security?

Security training is essential; however, it’s also expensive and
problematic on a limited budget. Providing the foundation to do things
correctly, or securely, from the beginning meant fewer problems after
administrators moved systems into production. To build these
foundations on the cheap, the UNC Security team developed internal
programs to provide campus administrators a hands-on opportunity to
build and secure both Windows and RedHat server systems. These day-long
classes equipped UNC administrators with secure build experience and
resulted in repeatable results when they returned to their departments.

Sensing the desire and need for broader security training, the UNC Security team partnered with the SANS Institute and through the local mentor program I offered several sessions of Security Essentials
to the campus community. This in depth security training, offered at a
group discount and sparing the expense of travel, allowed the UNC
Security team to build a cadre of security advocates and incident
response experts in various departments around the University. Having
resources on the ground with the requisite skills to both prevent and
contain security issues multiplied the manpower of the limited Security
team staff.

The UNC Security team saw how other universities were struggling with
information security and presented at Educause on the various free
security tools that were available at the time. Providing an end-to-end
roadmap from prevention, to detection, to clean up, this look into how
the sausage was made gave the attendants at that year's conference an
opportunity to kick off their own security efforts in the absence of
any real budget priority.

For the student population, we realized that we were probably too old
and square to really connect with the right message. We engaged, and
paid, some students who were interested in video production to craft
some commercial-like messages about information security topics. We
provided the list of topics and allowed them to run with their ideas.
The results ended up being pretty catchy.

And to make your joy complete, I understand that the team also tackled some major wireless issues. How did that work out?

For campus networks that were early adopters of both Wireless and
routed VLANs, Microsoft's network bridge, initially enabled by default
in XP Home, was a disaster. Did Microsoft understand networking. Hadn't
they ever heard of spanning tree loops. Trying to work with Microsoft
was like yelling at a wall. Fortunately, most of the systems coming on
to UNC's network included UNC's image and the configuration could be
made right; however, the occasional system from outside the program
could wreck havoc on portions of the campus backbone.

What was once a little known feature buried in some vendors' switch
code eliminated this problem at UNC. BPDU blocks, or Span Guard, killed
the wired port for any system seen attempting to create a bridge. This
cleared up the network loops, but wireless management still left a lot
to be desired. The penalty box and source blocking tools available on
the Enterasys switch ports did not extended to the Cisco wireless
access points in use at the time, and the movement to 802.1q VLANs
meant the SecureFast tools were going away.

Repeated requests to the vendor for a method to drop infected systems
from wireless were met with no reply. It seemed the sales engineer and
his technical resources had never heard of such a request and really
had no interest in finding a workable answer. The UNC team took it upon
themselves to crack the puzzle and discovered that, since an AP is a
bridge in disguise, Cisco's set of bridge commands (including discard)
worked on their APs. While SNMP was generally the preferred way to
manage equipment, the only avenue available on those early APs was
management through telnet. Accordingly an "expect" script with the
necessary telnet commands was written to facilitate the speedy removal
of bad actors from the wireless network.

UNC's Security team involved themselves in various outreach efforts,
including the leadership behind a well-received white paper for
implementing secure wireless in healthcare, and extensive contributions
to the development of an Information Security Baseline for the
16-campus system of the State of North Carolina. An outside vendor was
later engaged to audit each campus against both this system baseline
and ISO 17799. For such a large University system, North Carolina was
ahead of the times.

Looking back, what advice do you have for anyone considering a major wireless rollout?

Evaluate all the equipment that you can get your hands on and really
dig into their management interface. How quickly can you find a
particular user’s system. How quickly can it be dropped or
isolated. Is their management system standards based. Will it
interoperate with other best-of-breed solutions like TippingPoint?

Can you share some more about the technological tools that helped you manage such a large and amazing network?

As more and more new Microsoft vulnerabilities were discovered, the UNC
Security and Networking teams continued to see the size and scale of
security events increase at an exponential pace. Passive intrusion
detection, like our much beloved Snort, had become a postmortem tool.
Manual efforts to identify and isolate infected systems lost their
value as hundreds of hosts fell victim to Blaster and other RPC
vulnerabilities. By now UNC's network had fully moved to 802.1q VLANs,
and many of the old SecureFast tools were gone.

Early success as the second customer of TippingPoint showed both the
value of Intrusion Prevention and the uniqueness of TippingPoint's
approach. UNC's Security and Networking teams soon recognized the
opportunities of this powerful tool, and beyond their sub-division of
the campus network into multiple "attack domains," the UNC team home
grew the pieces to combine TippingPoint's IPS with their powerful
network management tools from Enterasys.

We are always interested in tools, do you know if they are
still running TippingPoint and Enterasys today? Any thoughts about
these tools for someone implementing a network today?

As I understand it, the core of what we built is still in use and the
Networking group is focused on a process of constant improvement. When
you find good vendors who listen to customers, and they quickly
implement requested changes, then the opportunities to really make some
gains open up for everyone. Vendors who maintain the flexibility to fit
your needs will sell plenty of product.

Thank you for that, Doug; so, you were able to achieve sanity through the proper use of technology?

All these pieces led to a completely automated, standards-based,
vendor-neutral, response to several IPS signatures that were known to
be 100% accurate to identify infected systems. Through various methods
this environment grew to the automated moving systems to a limited
self-help VLAN, the forcing a DHCP release and renew regardless of
lease times, and the creation of a detailed Remedy ticket so the Help
Desk could know immediately why a system had been isolated. The UNC
Help Desk often contacted the user before they realized there was a
problem.

And I understand that the world noticed your achievement, can you tell us about that?

The Information Security environment at UNC remained a transparent part
of an open network meant to encourage and support education and
research. No pre-authorization checks stood in the way of systems
joining the network, and yet infected systems were removed with enough
speed that their bad traffic could not impact others. The self-help
VLAN provided access to tools like fresh anti-virus definitions,
Windows updates, and a security scan by the Whole Security Confidence
Online behavioral anomaly detector. Through creative subnetting,
systems in self-help could not attack each other, or anyone else, but
they could access some rudimentary tools to get them on the road to
recovery.

UNC’s extensive program, ranging from their secure system images,
to their education efforts, their excellent network management, and
their willingness to become early adopters and supporters of highly
effective new technologies such as TippingPoint's IPS, made the group
of people accomplishing things at UNC a team of Information Security
thought leaders. They were recognized in 2005 as Network World
All-Stars for their efforts to automate security response.[1]

Doug, I really appreciate your taking the time to share, this
group clearly achieved thought leadership, who were the members of this
team?

Many folks contributed, but the core members of this successful UNC Security and Networking team were:

Can you tell us anything about the team dynamics. Can you talk about a
time you were under extreme stress, maybe when blaster hit. Who took a
leadership role?

We happy band of brothers (and sisters) grew to know each other well
enough to complete each others’ sentences. During crisis or
extreme stress the team member with the current answer took the
leadership role, until the answer changed or was further refined. We
were always focused on solving the problem; personalities rarely got in
the way, especially when an argument with technical merit was to be
had. We kept meetings small and short. Sometimes quick decisions had to
be made, but even those usually had the quick and concise input of
three or more people - often a sanity check: dude, are you seeing what
I’m seeing?

One official once commented that "those guys from Security roll their
own"; we weren’t rolling anything, and weren’t entirely
sure what he meant, but we took it as a compliment anyway.

Thanks, can you talk about a time the team was divided on an issue. How did you sort that out and develop consensus?

There were times when more then one solution would solve the problem;
when that happened it often became a divide and conquer situation of
"you try your way with that one and I’ll try my way with this
one." We’d then circle back and come to consensus on what was
cleanest, quickest, and best for the customers.

Do you still think of yourself as a team-oriented person or
does your current job position cause you to be in more of an individual
contributor role?

I consider myself to still be very team oriented; a new team with new
challenges but still very much a team that works together to find what
is best for the customers. I firmly believe that a strong team will
almost always out-perform any individual.

If there was one thing you really wanted to share with our
readers, a rant, a tip, an observation, even a charge to action, what
would that be?

UNC’s success came from cherry picking the best-of-breed
solutions to fit each need. Do not buy into the marketing hype that any
single vendor can supply all of your security needs, you’ll need
to work with several different security vendors.

Thanks so much Doug, can I ask you to tell us a bit about yourself, what do you do when you are not in front of a computer?

When I’m not spending time with my wife and kids, I’m
usually studying for my graduate classes or tinkering with
non-electronic things. On occasion I’ll dress for 1776 and join a
group of folks who reenact the Department of the Army Geographer from the American Revolution.
This group of hobbyists focuses on the technologies of surveying and
cartography from the days before GPS made our lives so easy. It’s
a great way to get away from the machines for awhile - sleeping in
canvas tents around a campfire and cooking salt pork.