Sandbox Architectures — Which One is Best at Detecting Malware?

Sandbox Architectures — Which One is Best at Detecting Malware?

Sandboxes are very good at detecting malware. However, some sandbox architectures are significantly more effective than others when it comes to identifying the more advanced strains of malware.

The idea behind a sandbox is simple—it’s an isolated, secure environment to open a file and determine if it is either benign or malicious by monitoring and analyzing its behavior. The sandbox allows the program to execute and perform all of its operations, which are monitored and recorded by the sandbox. After a specific period of time, the sandbox stops the program and analyzes its behaviors for malicious activities and patterns.

Since sandboxes do not rely on signatures, it is even possible to detect zero-day or highly targeted malware that security researchers and AV tools have not yet seen or evaluated.

Sandbox Architectures

However, not all sandboxes are created equal. When developing a sandbox, vendors use one of three architectures: virtualization, operating system (OS) emulation, and hardware (full system) emulation. Here’s a quick look at each.

Virtualization

Most sandbox products use virtualization. These products typically run on virtual machines to optimize the volume of files a single piece of hardware can analyze, using a “hypervisor” to control the execution of different virtual environments. During the timeframe when the hypervisor relinquishes control to the malware for execution, the malware and operating system run directly on the system hardware. Performance is essentially the same as when executed on a prospective victim’s machine.

However, criminals have adapted malware to include evasion techniques designed to discover a virtualized system and then alter its behavior to avoid detection. We are seeing an increasing percentage of malware with evasion capabilities.

Virtualization also dramatically limits what the sandbox can see. While the sandbox can observe calls to the OS, it can’t see what the malware does internally with those calls. Since a virtualization-based sandbox can only view part of the threat, one might liken it to a museum that has a security camera on the front door but no camera for the back door, and no cameras inside the museum.

While effective for basic forms of malware, virtualization-based sandboxes are ill-equipped to detect modern, evasive malware.

OS Emulation

Some sandboxes emulate the operating system. The idea is that by emulating the operating system, the sandbox has greater visibility into what the malware is doing. This provides a number of advantages over the limited view of emulation based technologies. So, in theory, this seems like a sound approach.

However, trying to emulate an entire OS would require duplicating Windows and every other OS in the enterprise. That, of course, is not possible. So, developers emulate just portions of the OS—the most common and applicable system calls. The problem is that cybercriminals have picked up on this approach, and advanced malware evokes infrequently used system calls that the sandbox doesn’t support. This makes it very easy for the malware to detect and evade sandboxes based on OS emulation.

Hardware or Full System Emulation

The third and most advanced sandboxing approach emulates the entire hardware system, including the CPU, memory, and I/O devices. Because this method doesn’t introduce any artifacts, it is much harder to detect than either virtualization or OS emulation. It’s also the only approach that provides Deep Content Inspection, which allows the sandbox to view everything that the malware does, including its use of the CPU, memory, and I/O devices.

Full system emulation also allows the sandbox to interact with the malware. When malware makes a call to the operating system, the sandbox can return different results and evaluate the malware’s reaction, or cause different code modules to execute.

Independent studies, such as the 2017 NSS Labs Breach Detection System Test, have repeatedly shown the advantages of full system emulation. Lastline, which uses full system emulation, is the only sandbox tested by NSS Labs to achieve 100% detection of advanced, evasive malware without generating false positives.

As mentioned earlier, emulation does introduce an additional layer that can impact performance, especially if the implementation is poor. Fortunately, we’ve developed a number of innovations to speed up emulation. It’s now at a level where it is (almost) as fast as native execution.

The Architecture Matters

As malware continues to evolve in sophistication, the type of sandbox an organization deploys becomes all the more important. While conventional sandboxes based on virtualization technologies may have sufficed when they were initially developed, modern malware that has been programmed to evade detection by sandboxes mandates the use of a sandbox based on full system emulation, which malware is unable to detect and thereby evade.

Bert Rankin has been leading technology innovation for over 25 years including over 5 years in security solutions that prevent cybercrime. He is a frequent blogger and is often quoted in security-related articles. Bert earned his BA from Harvard University and an MBA at Stanford University.