PAGE_TABLE_ISOLATION needs to switch to a different CR3 value when itenters the kernel and switch back when it exits. This essentially needs tobe done before leaving assembly code.

This is extra challenging because the switching context is tricky: theregisters that can be clobbered can vary. It is also hard to store thingson the stack because there is an established ABI (ptregs) or the stack isentirely unsafe to use.

Establish a set of macros that allow changing to the user and kernel CR3values.

Interactions with SWAPGS:

Previous versions of the PAGE_TABLE_ISOLATION code relied on having per-CPU scratch space to save/restore a register that can be used for the CR3 MOV. The %GS register is used to index into our per-CPU space, so SWAPGS *had* to be done before the CR3 switch. That scratch space is gone now, but the semantic that SWAPGS must be done before the CR3 MOV is retained. This is good to keep because it is not that hard to do and it allows to do things like add per-CPU debugging information.

What this does in the NMI code is worth pointing out. NMIs can interrupt*any* context and they can also be nested with NMIs interrupting otherNMIs. The comments below ".Lnmi_from_kernel" explain the format of thestack during this situation. Changing the format of this stack is hard.Instead of storing the old CR3 value on the stack, this depends on the*regular* register save/restore mechanism and then uses %r14 to keep CR3during the NMI. It is callee-saved and will not be clobbered by the C NMIhandlers that get called.

swapgs+ /*+ * This path is not taken when PAGE_TABLE_ISOLATION is disabled so it+ * is not required to switch CR3.+ */ movq %rsp, PER_CPU_VAR(rsp_scratch) movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp

@@ -399,6 +406,7 @@ syscall_return_via_sysret: * We are on the trampoline stack. All regs except RDI are live. * We can do future final exit work right here. */+ SWITCH_TO_USER_CR3 scratch_reg=%rdi

popq %rdi popq %rsp@@ -736,6 +744,8 @@ GLOBAL(swapgs_restore_regs_and_return_to * We can do future final exit work right here. */

.Lerror_bad_iret: /*- * We came from an IRET to user mode, so we have user gsbase.- * Switch to kernel gsbase:+ * We came from an IRET to user mode, so we have user+ * gsbase and CR3. Switch to kernel gsbase and CR3: */ SWAPGS+ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax

/* * Pretend that the exception came from user mode: set up pt_regs@@ -1383,6 +1409,10 @@ END(error_exit) /* * Runs on exception stack. Xen PV does not go through this path at all, * so we can use real assembly here.+ *+ * Registers:+ * %r14: Used to save/restore the CR3 of the interrupted context+ * when PAGE_TABLE_ISOLATION is in use. Do not clobber. */ ENTRY(nmi) UNWIND_HINT_IRET_REGS@@ -1446,6 +1476,7 @@ ENTRY(nmi)

/*+ * We just saved %rdi so it is safe to clobber. It is not+ * preserved during the C calls inside TRACE_IRQS_OFF anyway.+ */+ SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi++ /* * User mode is traced as though IRQs are on, and SYSENTER * turned them off. */@@ -256,10 +266,22 @@ sysret32_from_system_call: * when the system call started, which is already known to user * code. We zero R8-R10 to avoid info leaks. */+ movq RSP-ORIG_RAX(%rsp), %rsp++ /*+ * The original userspace %rsp (RSP-ORIG_RAX(%rsp)) is stored+ * on the process stack which is not mapped to userspace and+ * not readable after we SWITCH_TO_USER_CR3. Delay the CR3+ * switch until after after the last reference to the process+ * stack.+ *+ * %r8 is zeroed before the sysret, thus safe to clobber.+ */+ SWITCH_TO_USER_CR3 scratch_reg=%r8+ xorq %r8, %r8 xorq %r9, %r9 xorq %r10, %r10- movq RSP-ORIG_RAX(%rsp), %rsp swapgs sysretl END(entry_SYSCALL_compat)