How to discover hidden rootkits

Unearth and remove rootkits using BitDefender's RescueDisk

Once upon a time, viruses were about chaos, destruction and loss of data, but that was before criminal gangs realised that computers could be used to extort and defraud, and could even be used as cyber weapons.

For the past decade or so, online crime has continued to evolve faster than the industry that has sprung up to protect us from it. Malware of all kinds is becoming stealthier as the rewards become more lucrative, and today even the most basic botnet client can cover itself in a shroud of invisibility.

So how do you detect such an infection and give your network a clean bill of health? This requires deep scanning - far deeper than your normal antivirus software can provide.

Rooting around

The name 'rootkit' derives from 'root', which is the system administrator's account name on UNIX and Linux-based operating systems, and 'kit', simply meaning a toolkit. Therefore, a rootkit is a toolkit designed to give privileged access to a computer.

To understand rootkits properly, it's necessary to see an operating system as a series of concentric security rings. At the centre is the kernel; this is usually called ring zero, and has the highest level of privilege over the operating system and the information it processes. Ring zero is also often referred to as kernel mode.

Rings one and two are usually reserved for less privileged processes. If these rings fail, they will only affect any ring three processes that rely on them.

Ring three is where user processes reside, and is usually referred to as user mode. Ring three is always subject to a strict hierarchy of privileges. It's interesting to note, however, that debuggers usually run in ring two because they need to be able to pause and inspect the state of user mode processes.

Importantly, a process running in a higher privileged ring is able to lower its privileges and run in an outer ring, but this can't work the other way around without the explicit consent of the operating system's security mechanisms. This is known as the principle of least privilege.

In cases where such security mechanisms can be avoided, a privilege escalation vulnerability is said to exist. Ring zero (kernel mode) processes, along with the modules that make them up, are responsible for managing the system's resources, CPU, I/O, and modules such as low-level device drivers. Many rootkits are therefore designed to resemble device drivers or other kernel modules.

If you want to spy on a computer, or intercept and modify data that doesn't belong to you, the kernel is the place to be. If you want to see everything that's typed into a keyboard, a rootkit that masquerades as the keyboard driver is what you need. To see everything sent to and from the network, a network card driver is the thing to replace.

Protection

If kernels were simply lumps of code that were compiled by the developer and then never changed until another was released, rootkits would be easier to detect. However, modern operating systems are extensible; they can take advantage of optionally loadable modules.

At system bootup, a typical operating system might scan the hardware and only load the modules it needs in order to control that hardware. These modules are therefore very lucrative targets for malicious code writers. If a module can be replaced with one containing a rootkit, it will then be loaded into the kernel and will run in ring zero.

To prevent poisoned kernel code from being loaded in 64-bit Windows 7, Microsoft now insists on cryptographic code signing for all loadable modules. A 'hash value' is generated for the module by running its code through an algorithm. Only if the code produces the same hash value as the original code compiled by Microsoft is it loaded and run. Any deviation from the hash value means that the code must have been modified and therefore will not load.

However, because some older hardware still uses device drivers that don't support signing (as does some custom hardware), it is possible to bypass device driver signing by pressing [F8] at boot time and selecting the option on the boot menu to disable it.

When installing programs in Windows, the user access control prompt that appears is also a potential infection vector. Once you say 'Yes', you're giving privileged access to the operating system - but do you always know what you're installing? If a hacker can convince you to click 'Yes' when you should be saying 'No', your antivirus software can't always save you.

This is why it's dangerous to simply install software because a friend sends it to you, and why installing cracked, pirated versions of software can be extremely dangerous.

Flexible malware

Today, rootkits are an epidemic. As malware, their purpose is not usually directly malicious, but instead they are used to hide malicious code from your operating system and your defences.

Being so flexible, rootkits find many uses. For example, rootkits can be used to create and open back doors to operating systems for privileged access, either by command line or via a GUI. Such access allows a potential attacker to browse, steal and modify information at will by subverting and even bypassing existing account authorisation mechanisms.

If a rootkit stays on a PC after reboot, it will also allow hackers back into that system with privileged access at a later date. To prevent discovery, once running, rootkits can also actively cloak their presence.

How they do this is quite ingenious. Programs such as the Windows Task Manager or Microsoft's alternative Process Explorer both need access to the operating system to report on what's happening. They are user processes, running in ring three with no direct access to the kernel's activities. Instead, they request information via authorised function calls.

However, if a rootkit has replaced the part of the kernel servicing those calls, it can return all the information the system monitor wants - except for anything relating to the rootkit.

Antivirus programs also use standard system calls, and this is why they are very poor at detecting rootkits on a running system. This ability to operate invisibly within the OS means that a major use of rootkits is to conceal other malware, which might in turn run in the outer rings of operating systems.

Some rootkits can even disable antivirus software. It's not unusual to find a highly sophisticated rootkit protecting a fairly simple piece of malware. So, how can they be discovered?

Detection time

Because a rootkit can actively defend against detection on a running operating system, the only way to be sure that it's not doing so is to prevent it from running. The best way of doing this is to shut down the operating system itself and examine the disk upon which it is installed.

Though this is specialised work, many antivirus vendors have woken up to the need for such tools, and now supply them free of charge. We're going to use BitDefender's free RescueDisk, which is supplied as a bootable ISO image ready to be burned onto a bootable DVD.

Based on Linux, this boots in place of the computer's operating system and scans the hard disk without any rootkit activity getting in the way. You can download BitDefender's RescueDisk from http://bit.ly/coqNmL. Click the 'BitDefenderRescue CD_v2.0.0_5_10_2010.iso' file to download it, then burn to a DVD.

Once this is done, place the DVD in the drive and reboot the computer. After a few seconds, the BitDefender boot menu will appear. Press [Enter], and after a few minutes a graphical desktop will load. BitDefender's software runs automatically from here. Click 'Continue' to start and the software will download and install the latest updates.

BitDefender then sets to work examining the disk. The software will run for 15 minutes or more depending on the size of your disk. It scans not only the operating system files but also the boot loader and other files, looking for signs of infection.

Provided that any rootkits are listed in the downloaded definition files, this should identify any stealth malware it encounters - including the malware that the rootkit was shielding.

Note that as it runs, BitDefender will refer to the C: drive as /media/LocalDisk-0. This is a convention in Linux and refers to the fact that the software mounts the system disk as it would any other storage device.

Once you have completed the scan and (hopefully) found nothing, you can switch the system off and on again, remove the disk and boot into Windows.

Within Windows

Though it's harder to determine whether a running Windows system is infected with a rootkit, it can be done. One solution to this problem is the free utility GMER, which you can download from www.gmer.net.

To do so, click 'Files' and then the 'Download EXE' button. This randomises the filename. In theory, any lurking rootkit might be ready to block the GMER executable, but if the filename is random, it will be harder for this to happen.

You'll then download a zip file, which Windows Explorer should be able to open. Drag and drop the GMER.exe file to a convenient directory (a USB memory stick is a good option) and then double click it to run. Click 'Scan' and GMER will scan the list of ticked OS items in the right-hand column.

This can take a while, but don't be concerned about the long list that appears unless you see something red, which indicates a possible infection.

As it scans, GMER also produces information about the running system. To see this information, click the tab marked '> > >'. This opens up several other tabs with the various types of information. Perhaps the most useful of these is the Processes tab.

As with other forms of malware, the success of rootkit detection depends on the technology used and the definitions provided by the security vendor, but this isn't always 100 per cent effective. It's therefore highly recommended that you scan your system using the free rescue disks provided by more than one vendor, as a mix of technologies and scanning methods is much more likely to detect rootkits.