Studijní program

Instituce přidělující hodnost

Metadata

Abstrakt

Intrusion detection systems (IDS) used in network security are complex solutions that require
precise tuning prior to their deployment. Such tuning, however, is a problem. If done statically,
the fixed configuration fails to follow the dynamic trends in the network traffic. On the other
hand, configuration which is dynamically optimized using the complete traffic of the monitored
network (background traffic) is infeasible due to the lack of ground-truth. To tackle these issues,
researchers recently proposed to mix prerecorded static traces of labeled network traffic (i.e.
challenges) into the background traffic, where they serve as evaluation data, and the IDS is
dynamically adapted with respect to these challenges.
This thesis extends the challenge-based approach in two steps. In the first step, we adopt
techniques from game theory to model the interactions between IDS (defender) and an attacker
to make the adaptation process robust against the rational adversaries. We propose
a dynamically-defined two-player single stage game with complex utility function to precisely
capture incentives of both attacker and defender. Next, we combine the game definition with
the challenge-based principle so we can estimate the parameters of the security game online, use
traditional game-theoretical solution concept to solve the game, and immediately reconfigure
the IDS accordingly. The experimental evaluation proves that this approach outperforms the
trust-based baseline solution and thus allows us to improve the performance of the IDS against
rational attacker.
However, using fixed database of static challenges for dynamic adaptation of the IDS is still
far from optimal as it provides data with only limited variability, and manual updates of the
database cannot provide new data fast enough as new trends and techniques used by malware
authors emerge literally every day.
To solve these problems, we propose to replace legitimate challenges with dynamic simulation
of network behavior based on probabilistic generative model. We experimentally verified that
the proposed model generates network traffic similar to the traffic of real users. Next, we
automate the updating the database of malicious challenges via emulation of malicious behavior
with network traffic observed during execution of malware binaries in controlled environment
(sandbox). In order to address the lack of labeled malware binaries, we propose novel approach
for classification and clustering of unknown binaries based on their interactions with system
resources (files, network traffic, mutexes, registry keys and error messages generated by the
operating system). Moreover, the proposed model prioritizes the generated clusters to further
aid the manual analysis of the threat level required in the definition of the security game. The
performance of the classification and clustering of malware binaries is verified on large real-world
dataset.