The Pentium II introduced enhanced support for transitioning between user-mode and kernel-mode. This support was provided through the introduction of two new instructions: sysenter and sysexit3. When a user-mode application wishes to transition to kernel-mode, it issues the sysenter instruction. When the kernel is ready to return to user-mode, it issues the sysexit instruction. Unlike the the call instruction, the sysenter instruction takes no operands. Instead, this instruction uses three specific MSRs that are initialized by the operating system as the target for control transfers[23].

The IA32_SYSENTER_CS (0x174) MSR is used by the processor to set the kernel-mode CS. The IA32_SYSENTER_EIP (0x176) MSR contains the virtual address of the kernel-mode entry point that code should begin executing at once the transition has completed. The third MSR, IA32_SYSENTER_ESP (0x175), contains the virtual address that the stack pointer should be set to. Of these three MSRs, IA32_SYSENTER_EIP is the most interesting in terms of its potential for use in the context of a backdoor. Setting this MSR to the address of a function controlled by the backdoor makes it possible for the backdoor to intercept all system calls after they have trapped into kernel-mode. This provides a very powerful vantage point.

For more information on the behavior of the sysenter and sysexit instructions, the reader should consult both the Intel manuals and John Gulbrandsen's article[23,15].

Category: Type I

Origin: This feature is provided for the explicit purpose of allowing an operating system to control the behavior of the sysenter instruction. As such, it is only logical that it can also be applied in the context of a backdoor. Kimmo Kasslin mentions a virus from December, 2005 that made use of MSR hooks[25]. Earlier that year in February, fuzen_op from rootkit.com released a proof of concept[12].

Capabilities: Kernel-mode code execution

Considerations: This technique is restricted by the fact that not all processors support this MSR. Furthermore, user-mode processes are not necessarily required to use it in order to transition into kernel-mode when performing a system call. These facts limit the effectiveness of this technique as it is not guaranteed to work on all machines.

Covertness: Changing the value of the IA32_SYSENTER_EIP MSR can be detected. For example, PatchGuard currently checks to see if the equivalent AMD64 MSR has been modified as a part of its polling checks[36]. It is more difficult for third party vendors to perform this check due to the simple fact that the default value for this MSR is an unexported symbol named nt!KiFastCallEntry: