Using Aggregation Technology to Improve System Call Based Malware Behavior Detection

Date of Defense

2011-07-19

Page Count

63

Keyword

Behavioral detection of malware

self-replication

survival intent

system call

Abstract

Malware is one kind of software which has intention to attack computer systems. In recent years there has significant increase in the number of malware, in addition malware also use polymorphism, obfuscation and packing technologies to protect itself. For the above reason, the effect of traditional static malware detection technology is restricted, as a result in recent years many studies focused on dynamic malware detection technology. However most of the previous studies are process center oriented, which mean these studies only monitor one process’s behavior, ignoring the possibility of malware using multiple process to complete malicious intent, or control legal process to hide their malicious behavior. In this paper we propose the use of dependency structure matrix to record the behavior of all process in user’s system and also propose an algorithm to detect multiple process’s self-replication and survival behavior, find the relations of the system processes by using the aggregation technology to improve the detection rate of traditional dynamic malware detection. As an evaluation of our proposes system. We execute the malware samples in the virtual machine and using process monitor tool to recorded system processes, and then detect whether our system can detect the malware or not. Experimental results show that we can detect 11% malware used the multiple processes to complete malicious intent in the 140 malware samples, and improve the weakness of previous studies which must used white list to avoid false positive.