A platform called ExecScent can seek and block bot calls to command-and-control servers

Attackers launch malware that automatically alters itself to avoid detection, and they constantly create new domains where their command-and-control servers can hide, but researchers have come up with security software that detects the presence of attack code even if it has morphed and tracks down domains that infected client machines report to.

A platform called ExecScent can detect C&C traffic and ID the malware family involved or even the specific criminal group behind it, a team of researchers will tell Usenix Security 2013 this week.

The software carries out a two-pronged attack on malware. First it defines protocol characteristics of HTTP/HTTPS requests between infected clients and C&C servers as a way to recognize malicious traffic, its source and its destination, say the researchers. They are led by Terry Nelms, a Ph.D student at Georgia Institute of Technology and also director of research at Damballa, Inc.

Damballa has incorporated ExecScent into its Failsafe network security appliance as a feature called Request Profiler. Failsafe is usually attached to a TAP port on an Internet router and sometimes to a switch port in order to monitor traffic.

The protocol characteristics ExecScent examines are rolled up into control protocol templates (CPT) for each known type of malware. When actual communications traffic matches a CPT, it is considered suspect. The templates include the family of malware involved and, if available, the criminal groups behind the malware.

Often when attackers craft morphing malware they alter the code to avoid being detected by signature based defenses. But the devices involved in the attacks still have to communicate, and the means they use can be used to identify them, the researchers say.

The second prong of their approach is designed to reduce false positives that may result from matching traffic to a CPT. ExecScent tunes itself to each network it runs on, figuring out what potentially suspect traffic components are actually legitimate. So a particular HTTP user agent header may be commonly used on a given network and so would carry less weight as a factor in determining whether the traffic it enables is malicious.

The value of these components as determining factors varies network to network. For example, a domestic bank network might never send legitimate requests to China, so request to URLs in China would carry a heavy weight in determining that such requests are to C&C servers. On the other hand, a manufacturer with suppliers in China would generate HTTP requests to China URLs frequently, lowering the significance of such requests as a determining factor, the researchers say.

This evaluation and weighting as well as blocking the illicit traffic is done automatically via TCP resets, Nelms says. The system analyzes enough traffic over the course of two or three business days to adjust to each network, he says.

Users can set a detection threshold on the evaluation, that, if exceeded will mean traffic is deemed a C&C request. The higher the threshold, the fewer false positives are returned, but at the same time fewer new C&C domains are caught and fewer infected hosts are discovered, he says.

For the research Nelms and his team are reporting on, prototype of ExecScent was deployed in three large networks for two weeks. Compared to an alternative method of using a domain black list to find command and control servers, ExecScent found more new C&C servers as well as hundreds of previously undetected infected machines.

Nelms says Damballa is working on improvements to Request Profiler that will keep false positives lower and at the same time increase the number of C&C domains and infected client machines that it detects.

If the product generates a false positive, it has to be discovered by end users who can't reach a legitimate domain or by security techs who recognize a blocked domain as legitimate. Then customers have to get Damballa to resolve it, he says.