CERT/CC Blog

CERT Linux Triage Tools 1.0 Released

As part of the vulnerability discovery work at CERT, we have developed a GNU Debugger (GDB) extension called "exploitable" that classifies Linux application bugs by severity. Version 1.0 of the extension is available for public download here. This blog post contains an overview of the extension and how it works.

Background

CERT recently released version 1.0 of the Failure Observation Engine (FOE), a fuzz testing framework for Microsoft Windows platforms, for public download. You can learn more about FOE in David Warren's recent blog post. As FOE accumulates crashing test cases for the application-under-test, it uses MSEC's !exploitable debugger extension to classify the associated application errors by severity. This allows the auditor or developer who is using the tool to determine which bugs to investigate first.

The CERT Basic Fuzzing Framework (BFF), which was initially released for public download in May 2010, is a fuzzing framework designed for use in Linux and Apple OS X. Support for Apple OS X was added in version 2.5 and will be described in an upcoming blog post. When running in OS X, the BFF uses Apple's CrashWrangler tool to assign crash severity in a manner similar to how FOE uses !exploitable on Windows. We were unaware of a similar, lightweight application error classification tool on the Linux platform, so we developed the CERT exploitable GDB extension for this purpose. While the GDB extension is designed to be integrated with future versions of BFF, it can be used as a standalone tool as well.

Inspired by !exploitable and CrashWrangler

MSEC's !exploitable and Apple's CrashWrangler are simple, but powerful, application error classification tools that have enjoyed widespread adoption on their respective platforms. Our team has found their lightweight execution and simple heuristics amenable to our black box fuzzing approach. In order to foster adoption, promote ease of use, and to allow for orthogonal design in our fuzzing frameworks, the CERT exploitable extension is modeled closely after the MSEC and Apple products. Just take a look at some respective output from the tools:

Explanation: The target crashed on an access violation at an address matching the destination operand of the instruction. This likely indicates a write access violation, which means the attacker may control the write address and/or value.

Crash accessing invalid address. Consider running it again with libgmalloc(3) to see if the log changes.

Test case was (null)

While the output from these tools may appear to be similar, the CERT exploitable extension includes some key differences. For example, where MSEC's !exploitable command performs taint analysis of the basic block containing the instruction that caused the application error, the CERT exploitable extension does not look beyond the faulting instruction. The CERT extension is similar to Apple's CrashWrangler in this respect. Additionally, the CERT exploitable extension is written entirely in Python, whereas MSEC's !exploitable is written in C++, and CrashWrangler is written in a mix of Ruby and C. We chose the Python language for the CERT tool to promote modification and enhancement of the code by third parties. It is also important to note that the CERT exploitable extension is designed to work specifically with user space applications on the x86 and x86_64 platforms. For more information on how the code works and some tips on modifying it, see the readme files.

Using the CERT exploitable GDB extension and triage script

The CERT exploitable GDB extension is designed to be relatively simple to invoke. If you have experience using MSEC's !exploitable extension, then using the CERT exploitable GDB extension may seem familiar. First, download the CERT triage tools package and extract it to a directory on a Linux host. Be sure to check the readme files for system requirements.

Once you have the tools extracted, you should be able to run GDB and load the exploitable extension:

$ gdb

(gdb) source exploitable/exploitable.py

(gdb) help exploitable

A GDB Command that determines how exploitable the current state of the

Inferior (the program being debugged) is. Either prints the result to

GDB's STDOUT or pickles the result to a file.

This command is designed to be run just after the Inferior stops on

a signal, before any commands that might change the underlying state

of GDB have been issued. WARNING: This command may change the underlying

Explanation: The target crashed on an access violation at an address matching the destination operand of the instruction. This likely indicates a write access violation, which means the attacker may control the write address and/or value.

Other tags: AccessViolation (20/21)

(gdb)

For more details on installation and usage of the exploitable extension, check out the readme file. In addition to the GDB extension, the CERT triage tools package also includes an example wrapper script called "triage." While the GDB extension classifies a single application error, the triage script runs an application with a set of test cases and classifies each with the GDB extension. This script was developed as proof-of-concept for integration with later versions of the CERT BFF and as an example for integration into other testing frameworks by other developers; however, it can used as-is to classify a set of test cases.

To use the script, you will need to download the CERT triage tools package and extract it to a directory on a Linux host if you haven't already. Again, be sure to check the readme files for system requirements. Once the tools are extracted, run a properly formatted triage command. Here is an example of running the tool against the set of Apple CrashWrangler test cases that are included in the package.

$ python triage.py \$sub `find ./exploitable/tests/bin -type f`

The input syntax for the triage script is somewhat tricky, and the tool isn't particularly robust. This is a side effect of the tool's primary purpose as a proof-of-concept and example wrapper script. Regardless, once you have executed the script correctly you should see this:

... (libc output) ...

EXPLOITABLE: StackBufferOverflow

./exploitable/tests/bin/stack_buffer_overflow.test

EXPLOITABLE: PossibleStackCorruption

./exploitable/tests/bin/variable_length_stack_buffer.test

EXPLOITABLE: DestAv

./exploitable/tests/bin/cpp_crash.test

./exploitable/tests/bin/crashwrite.test

./exploitable/tests/bin/fastMalloc.test

./exploitable/tests/bin/invalid_address_64.test

./exploitable/tests/bin/recursive_write.test

EXPLOITABLE: BadInstruction

./exploitable/tests/bin/illegalinstruction.test

EXPLOITABLE: HeapError

./exploitable/tests/bin/malloc_abort.test

PROBABLY_EXPLOITABLE: BranchAvNearNull

./exploitable/tests/bin/bad_func_call.test

./exploitable/tests/bin/crashexec.test

PROBABLY_EXPLOITABLE: BlockMoveAv

./exploitable/tests/bin/read_and_write_instruction.test

PROBABLY_EXPLOITABLE: DestAvNearNull

./exploitable/tests/bin/nullderef.test

PROBABLY_NOT_EXPLOITABLE: SourceAvNearNull

./exploitable/tests/bin/uninit_heap.test

PROBABLY_NOT_EXPLOITABLE: FloatingPointException

./exploitable/tests/bin/divzero.test

UNKNOWN: SourceAv

./exploitable/tests/bin/crashread.test

UNKNOWN: AbortSignal

./exploitable/tests/bin/abort.test

Failed to triage:

./exploitable/tests/bin/nocrash.test

Customization

While we kept third-party adoption and development in mind, we developed these tools primarily for some specific testing purposes in the CERT Program. The tools have not been exhaustively tested, and have not been tested at all on many Linux distributions! Further, the nuances of various Linux distributions and GDB versions may change how the exploitable extension classifies application errors. If you choose to use the GDB extension, I encourage you to become familiar with the source code and edit it to suit your needs. It is designed to be simple and easy to modify--check out the readme files to get started.

That's all for now--if you have any comments, questions, patches, or other feedback please drop me a line.

SEI LIBRARY

LATEST POST

According to DevSecOps: Early, Everywhere, at Scale, a survey published by Sonatype, "Mature DevOps organizations are able to perform automated security analysis on each phase (design, develop, test) more often than non-DevOps organizations." Since DevOps enables strong collaboration and automation of the process and enforces traceability, mature DevOps organizations are more likely to perform automated security analysis than non DevOps organizations. My previous blog post, Microcosm: A Secure DevOps Pipeline as Code, helped address the problem that most organizations do not have a complete deployment pipeline in place (and are therefore not considered to be DevOps mature) by automating penetration tests of software applications and generating HTML reports as part of the build process through the Jenkins CI service. In this follow-up blog post, I explore the use of a service evolution of Microcosm as a simple one-stop shop for anyone interested in learning how to implement a DevSecOps pipeline.