Detecting Malicious Java Code
Using Virtual Machine Auditing

Abstract:

The Java language and its execution environment, the Java Virtual Machine
(JVM), has evolved from a technology that supports active web pages into an
environment for the development and execution
of large-scale, network-based applications. Java provides
extensive support for authentication and access control but it lacks
support for intrusion detection.

Existing operating system auditing facilities and
host-based intrusion detection systems operate at the process level, with the
assumption that one application is mapped onto one process. However, in many
cases, multiple Java-based applications are executed as threads within a single
JVM process. This makes it difficult to analyze the behavior of Java
applications using the corresponding OS-level audit trail. In addition, the
malicious actions of a single Java application may trigger a response that
disables an entire execution environment. To overcome this limitation, we
developed an auditing facility for the Java Virtual Machine and an intrusion
detection tool that uses audit data generated by this facility to detect
attacks by malicious Java code. This paper describes the JVM
auditing mechanisms, the intrusion detection tool, and the
quantitative evaluation of their
performance.