The secret life of a rootkit

Website: http://www.computerworld.com/printthis/2005/0,4814,106842,00.html

DECEMBER 07, 2005 (COMPUTERWORLD) –

Much like in the movie The Matrix, where the world presented to the computer user is not a true representation of what is really going on, a rootkit is a program that can be used to hide files, registry settings, network connections, processes and other information from computer users.

For example, a rootkit can make processes that run but are hidden from Windows Task Manager, registry keys that can't be seen with Regedit, and network connections that are not viewable by Netstat.

Rootkit technology allows malicious software (malware) to be stealthier, and that in general makes it more effective. This is not about just evading detection by a casual user; rootkit technology allows malware to evade many antivirus and antispyware programs. There are few legitimate uses for this kind of technology, although some companies do market "hidden folders" that enable users to hide sensitive or embarrassing information from other users of the same computer.

All rootkits rely on the ability to manipulate the results of the function calls made by programs. For example, in order for the Task Manager program in Windows to show a list of running processes, it calls a Windows API function (EnumProcesses) that returns a list of identifiers (or process IDs), which are obtained from a data structure in the kernel.

A rootkit works by intercepting the call and filtering out the processes that it is trying to hide. They can be implemented either in user space or in the kernel, with the kernel rootkits being the most dangerous.

Files, registry entries and network connections can all be hidden in analogous ways by altering the results of the appropriate function calls. Because most antivirus and antispyware programs rely on these calls (for example, to find files to scan), files hidden by rootkits are invisible to an antivirus program. The machine could be infected, but an antivirus program would be unable to detect it.

Kernel-mode rootkits require some code to be loaded into the kernel (normally a device driver or .sys file). They can do this by following the legitimate route that low-level device drivers use (using the service control manager services .exe), or there are a few undocumented ways to insert code into the kernel. Once inside, the code can modify the results of functions calls made into the kernel or modify kernel structures.

How to spot a rootkit

There are two main ways to detect the presence of a rootkit on an infected machine: scanning and event monitoring. The scanning technique involves comparing a view of the system using user-space tools and a view from inside the kernel. If anything is hidden, it should be visible in the kernel, but not in user space. Recently, a variety of programs have been released that do these scans.

This technique is good in principle -- rootkits hide resources, so the best way to detect them is to look for things that are hidden. There are, however, a couple of weaknesses with this approach. The first is that if the kernel itself has been compromised, then the scan from kernel space may be tainted by the rootkit. Whether this happens or not depends on the details of exactly how the scan and rootkit are implemented. Since the Windows kernel is largely undocumented, it is hard to be sure that the scan is giving the correct results. Also, rootkits can evade detection by hiding from all processes except the rootkit detector.

The alternative approach is to use an event-based system that monitors continuously to catch the rootkit in the act of installation. These programs are often called intrusion-prevention systems (IPS). It is important that the program behavior is monitored from the kernel. IPS systems that monitor in user space are just as vulnerable to rootkits as any other user-space program.

These systems can detect and block the loading of kernel modules. However, blocking all kernel modules is impractical -- many legitimate programs install kernel modules. For example, some antivirus programs use kernel modules to perform on-demand scanning.

It is possible to make a better decision regarding whether the loading of the module is malicious by looking at other properties of the installer and other associated programs. While a rootkit and an antivirus program might have actions in common (e.g., installing a kernel module), there are many other characteristics that they would be less likely to share.

For example, a rootkit might try to be stealthy by not having a visible window, while the antivirus program will likely advertise its presence to reassure the user. The rootkit program may also install a keylogger, which one would not expect a well-behaved antivirus program to do. By combining various behavioral characteristics (carefully chosen so that they capture the common behaviors associated with malicious code), it is possible to reliably detect rootkit programs. In fact, this general approach, termed "behavioral heuristics," can be applied more broadly, to detect other classes of malicious code such as Trojans, bots and spyware.

Being based on heuristics, this type of system can make mistakes (classifying normal programs as malicious). The normal technique used to deal with this is to have exclusion lists for the common errors, which would then have to be maintained.

Undoubtedly, rootkits are serious and increasingly problematic for computer users. They enable malicious code to remain undetected by most security software. They work by manipulating the results of function calls, which they gain access to by a variety of mechanisms. The good news is that they are detectable, with two mechanisms currently available -- a scanning technology that can detect them after they are installed, and an event-based system that can catch them in the act of installation (by observing their behavior) -- and thus prevent compromise in the first place.