How can we know if our patching behavior is scientific and effective? When adjusting defenses to protect web programs in a small shop, sometimes we will be unable to immediately observe if the defensive changes we've made will work to protect our assets. Following the examples of Benjamin Dean and William Shadish, we can design quasi-natural experiments that will allow us to reasonably assess the effectiveness of our treatments.
In most situations, conducting a traditionally controlled scientific experiment for simulating an injection attack will require too many resources and too much time. Quasi-natural experimental designs, when chosen carefully, can help us conserve testing and experimentation. They can help us step closer to proving that a patch or treatment will work to defend against a style of attack. By tying
our choices about experimental design back to a standard NIST model of risk assessments, we can support reasonable plans for evaluating injection attack tools and their defenses.
Injection attacks have long been at the forefront of our most frequently observed attack styles. I would suggest that the more automated an attack suite is and the less technical knowledge an operator needs to use it: the more dangerous that tool is to websites. As part of my studies, I will show a basic pattern of analyzing the effects of SQL Injection Attack tools by using quasi-natural experimentation. Using those trials, we will be able to see concrete examples of how the experimental design process can inform the practical planning of defensive tests.
In this talk we will examine the relationship of some popular attack tools, experimental design techniques, and risk assessments. We'll also cover experimental design and risk analysis for policy change in response to attacks involving: buffer overflows, code injection, network scripting attacks, WiFi replays, LAN wiretaps, phishing campaigns, RFID cloning, and mechanical lockpicking. We'll cover some experimental design patterns and their strengths and weaknesses. We'll focus on how the type of quasi-natural experiment we choose, based on a NIST-style risk analysis, can help us direct our attention toward evaluating the effectiveness of solutions to attack problems.