The Doomsday Machines of Malicious Software

Nothing unusual running on your system? Don't bet on it. A small piece of code called a rootkit may be lurking in the recesses of your system.

Every week I write about evil people and the evil software they write. But nothing scares me like rootkits.
Ever hear of a rootkit? Its a surreptitious program--not necessarily malicious (but if not, then whats the point?)--that goes to great lengths to conceal its presence on your system. A rootkit intercepts file system and other calls to prevent you from seeing it in directory listings, or lists of running programs. And once it has accomplished this, it can go about its business, whatever that may be. Rootkits have been around on UNIX for many years. They started showing up for Windows about a year ago, and theres every reason to think that Windows will be where the action is from now on.
With so much going for them, of course this weeks BlackHat USA 2003 Briefings & Training has a two-day course on"Aspects of Offensive Root-kit Technology." The course is filled up. Gulp!

Before we all go running in circles while screaming, remember that for a rootkit to run, it needs to find its way onto your system and then be executed. It also needs a fairly high level of privilege, which presumes a fairly serious breach of security.
If youre not already being infected by viruses and Trojan horses all the time, you probably already have the sort of measures in place that would block most attempts to place a rootkit on your systems. But its also true that some vulnerabilities can go undetected for a while, and once a rootkit is installed, with sufficient rights on a trusted system, it can become a vector to compromise anything else on the network.
It also occurs to me, despite my recent skeptical treatment of heuristic virus scanners, that rootkit installation is a natural target for heuristic analysis.
Surely, the operating system or some other trusted component could track processes that hook the appropriate calls and interrupts, and then flag them. I have to think that an operating system on the lookout for such behavior would be much harder for a rootkit to slip past. In the meantime its possible for conventional pattern-based antivirus programs to detect known rootkits, but only known ones.
At the same time, everyone should realize that it is possible to detect a rootkit, but its just not easy. Discovery will require that you to suspect its presence and have a decent instinct for forensic examination of your system.
For instance, a rootkit may hide its presence in Windowss Task Manager, but it will still consume memory. Perhaps you might notice that theres less memory available than there should be. Another symptom might be a machine operating slower than one with a supposedly identical configuration.
On the other hand, the rootkit could be installed as a service and utilize Windowss practice of glomming many services together in single process blocks. But once you have a strong enough suspicion, you can boot into the Windows Recovery Console or to boot CD-based tools to examine the system without running the infected installation.
Then again, you cant just block all software that looks and acts like a rootkit. Antivirus software, for example, is likely to hook most of the same system calls as rootkits.
The answer is a trust relationship: Probably no such application or driver should be allowed to run without a digital signature from a trusted source, and the user should be explicitly warned in detail what is happening.
Like a scene out of Invasion of the Body Snatchers, rootkits inspire paranoia and fear, and yet there is something there. Have they taken over your computer? Things may not be entirely what they seem.
Security Supersite Editor Larry Seltzer has worked in and written about the computer industry since 1983.

Larry Seltzer has been writing software for and English about computers ever since,much to his own amazement,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.