I'm making a sticky post for this, as some users may get bit by a recent necessary change to PaX's UDEREF (made a few days ago). Previously, UDEREF would detect the VMWare I/O backdoor, and if detected, would disable UDEREF for the guest. This was to avoid a conflict with VMWare that would cause incredibly poor performance on the guest. There now exist options available through the main interface of VMWare to choose to specifically use hardware-based virtualization for VM guests. The use of these options have no conflicts with UDEREF, so disabling UDEREF at startup in these cases was unnecessary.

If you're booting a VM guest with UDEREF enabled and the "Preferred mode" under Virtual Machine Settings -> Hardware -> Processors -> Execution Mode is set to "Automatic" or "Binary Translation", you need to:Append "pax_nouderef" to your kernel command lineCheck the "Disable acceleration for binary translation" optionIf the second operation listed here is not done, there's some bug in VMWare's binary translation that causes the same poor performance even with UDEREF enabled but pax_nouderef used

Alternatively, you choose one of the two below options at some performance cost to keep UDEREF without the significant impact of it with binary translation:Choose "Intel VT-x or AMD-v" as the "Preferred mode"Choose "Intel VT-x/EPT or AMD-V/RVI" as the Preferred mode"

elazar wrote:I noticed that the kernel won't boot if paravirtualization is enabled on the VM(and in the kernel).

do you have CONFIG_CC_STACKPROTECTOR enabled in your .config? i get a very early boot failure with it, but paravirt/VMI works otherwise (minus the usual features like KERNEXEC/UDEREF). can you also test a vanilla kernel and report this upstream if it is indeed a problem with SSP and paravirt/VMI?

I am getting looping reboots with 2.6.31.1 using grsecurity-2.1.14-2.6.31.1-200910012153, though nothing useful is displayed on the screen. System.map, bzImage etc. are at http://drop.io/qi0g5ga/asset/2-6-31-1-grsec-gz. This is a virtual machine running on ESXi 4, build 181792 on a PowerEdge T300(12gb RAM, Core2 Duo E6305), the VM has both processors, NX, and 768MB allocated, paravirtualization is enabled and CPU/MMU Virtualization is set to automatic. I will test a vanilla kernel in the morning.

Vanilla 2.6.31.3 works fine, and I get looping reboots with the latest 2.6.31.3 patch.

Edit:I'm seeing this in the virtual machine's event log:

Message from virtualhost: *** Virtual machine kernel stack fault (hardware reset) *** The virtual machine just suffered a stack fault inkernel mode. On a real computer, this would amount to a reset of the processor. It can be caused by an incorrect configuration of the virtual machine, a bug in the operating system, or a problem in the VMware ESX software. Press OK to reboot virtual machine or Cancel to shut itdown. info10/12/2009 5:22:31 PMUser

Disabling uderef via no_paxuderef=1 allows 2.6.31 to boot fine when paravirt is enabled on the VM itself, though 2.6.31.3 still triggers a kernel mode stack fault no matter how the VM is configured, it will only boot if I compile without paravirt enabled in the kernel.

it's pax_nouderef and it's an 'off-only' switch, it doesn't take arguments.

though 2.6.31.3 still triggers a kernel mode stack fault no matter how the VM is configured, it will only boot if I compile without paravirt enabled in the kernel.

i've been working on getting KERNEXEC to work with VMI for the past few days and it can boot into userland here albeit still dies randomly on certain page table operations. without KERNEXEC it should have always been fine though.

it's pax_nouderef and it's an 'off-only' switch, it doesn't take arguments.

Woops, forgot the no part. Actually, pax_nouderef on its own did not work, pax_nouderef=1 worked.

i've been working on getting KERNEXEC to work with VMI for the past few days and it can boot into userland here albeit still dies randomly on certain page table operations. without KERNEXEC it should have always been fine though.

I can boot 2.6.31.4 with pax_nouderef, paravirt compiled in and enabled on the VM but various processes(udev, bash, syslogd etc) die with general protection faults at boot time. It does not work any other way. 2.6.31 works, but with paravirt disabled on the VM itself.

I really appreciate the time that you have been putting in to this. If you need an ESXi box to test on, please PM me.

elazar wrote:Woops, forgot the no part. Actually, pax_nouderef on its own did not work, pax_nouderef=1 worked.

hmm, that must be a bug, the code doesn't need or check for any extra arguments and i've been using it under vmware workstation as such.

I can boot 2.6.31.4 with pax_nouderef, paravirt compiled in and enabled on the VM but various processes(udev, bash, syslogd etc) die with general protection faults at boot time. It does not work any other way. 2.6.31 works, but with paravirt disabled on the VM itself.

was this with KERNEXEC on or off? if it was on, can you try without?

I really appreciate the time that you have been putting in to this. If you need an ESXi box to test on, please PM me.

i don't think i'll need it as i get the GPFs here as well, although their random nature makes me think that it may no longer be my fault but something else, but it's very hard to debug as it's the hypervisor that simulates a GPF on certain page table operations.