At TS/SCI Security, we like to focus on the positive instead of the
negative. Pen-testing, especially application pen-testing/review is an
important topic to us. So here's our rundown of what we feel is
important to know about pen-testing, as a response to what Matasano
wrote.

Seven things you can do to improve your pen-test results

1. Time Management. If you're not already using
GCalendar,
RTM (with
GCal),
Twitter, Sandy,
and Jott (thanks to Dennis Groves for some of
these!) -- take a look at how these can help you manage your time
better.

Good pen-testing takes time. If you haven't gone through everything
you need in #2, then you might need more time. What I've found is
that pen-testers need to frame their pen-tests after good strategy
consulting (1-2 days of a
SWOT or similar
analysis). If, like mentioned in The New School, the "basics" are
not taken care of -- then help the client work on asset,
configuration, and change management before doing a vulnerability
assessment.

2. Utilize applicable MITRE resources.
CAPEC (for runtime testing) and
CWE (for static analysis testing, which may
include reversing) should be utilized throughout. Be careful to only
choose the relevant aspects of each (relevant to the application, not
your skill or other criteria).

My favorite MITRE resource is the Introduction to Vulnerability
Theory,
which Marcin and I spoke about at ShmooCon in our talk on Path
X. It may take some
extra time to spell out exactly how the vulnerabilities were found
using this method, but it will help future assessment work, including
repeat client business.

Standard C and assembly code is difficult to reverse the design from,
but they are becoming more easy to reverse. In the case of Java, C#,
and C++ -- UML can be extracted to elicit the design of the
application. Using other tools such as Klocwork K7, Fujaba, or IBM
Rational Rose on the UML diagrams will possibly provide faster
program understanding. Left with the source code, GrammaTech
CodeSurfer also aids understanding, but open-source tools such as
Doxygen can also be of use.

4. Automate development-testing. While web application security
scanners (or fat application-based fuzzers) collectively only typically
find at-most 30% of the bugs with a large amount of false positives,
there are plenty of developer tools that don't have these problems.

Developers should be made aware of integration testing with Canoo
WebTest. Business analysts, customers, and test managers can submit
table-driven test cases using FitNesse (a wiki that allows for
collaborative test case design), along with additional tools such as
HtmlFixture. I've spoke
to the benefits of Dependency
Injection
before, as well as automating Continuous Integration both inside of
the IDE as well as during the application builds.

Automation is good for developers and the modern tools surrounding
test automation are extreme improvements over last decade's
technology. However, test cases and test-first strategies still find
only as much as half (possibly up to 70% or more) of the bugs. It's
good to combine exploratory
testing with
scripted (i.e. from a test case) testing. Allot some time with a
buddy and explore the application as it wasn't meant to be run after
learning as much as you can from the developers and their automated
tests.

5. Peer review your work. This one is easy. Find a friend and have
him/her check your work. The more eyes you get on your pen-test and
assessment work, the less likely you're going to miss something.

6. Stay up-to-date on both process and technology. Read
forums/blogs/mailing-lists, trade journals/magazines, and books. Attend
conferences. Keep it cheap and continuous if possible.

I try to read every security-related book that pops up on Safari
Library
and ACM's access to
Books24x7.
Audible/Amazon are certainly other
resources, especially for non-technical solution learning. Checking
yourself at least once a year on a certification or re-certification
program will make strides in your professional development.

Using GReader and sharing daily
using it's sharing features and
OPML with your colleagues is
important. Get involved in conversations on forums, IRC channels,
blogs, and mailing-lists is likely the largest win here.

7. Go back to basics. Set a minimum bar of maturity with clients
before you pen-test or do an assessment. The BITS Shared Assessments
Standardized Information Gathering questionnaire is a nice start,
especially combined with Visible Ops, ITIL, COBIT, ISO27K, PCI-DSS,
PABP/PA-DSS, ISECOM, and OWASP information on process.

All of the above is going to go a long way towards improvements to what
pen-testers and vulnerability assessors do. There's very few good
certifications out there for what we do, and it's difficult to measure
what we do. Put the cards on the table up front with your clients and
teach them your methodology and approach.