*[PATCH, RFC 00/62] Intel MKTME enabling@ 2019-05-08 14:43 Kirill A. Shutemov
2019-05-08 14:43 ` [PATCH, RFC 01/62] mm: Do no merge VMAs with different encryption KeyIDs Kirill A. Shutemov
` (63 more replies)0 siblings, 64 replies; 153+ messages in thread
From: Kirill A. Shutemov @ 2019-05-08 14:43 UTC (permalink / raw)
To: Andrew Morton, x86, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Borislav Petkov, Peter Zijlstra, Andy Lutomirski, David Howells
Cc: Kees Cook, Dave Hansen, Kai Huang, Jacob Pan, Alison Schofield,
linux-mm, kvm, keyrings, linux-kernel, Kirill A. Shutemov
= Intro =
The patchset brings enabling of Intel Multi-Key Total Memory Encryption.
It consists of changes into multiple subsystems:
* Core MM: infrastructure for allocation pages, dealing with encrypted VMAs
and providing API setup encrypted mappings.
* arch/x86: feature enumeration, program keys into hardware, setup
page table entries for encrypted pages and more.
* Key management service: setup and management of encryption keys.
* DMA/IOMMU: dealing with encrypted memory on IO side.
* KVM: interaction with virtualization side.
* Documentation: description of APIs and usage examples.
The patchset is huge. This submission aims to give view to the full picture and
get feedback on the overall design. The patchset will be split into more
digestible pieces later.
Please review. Any feedback is welcome.
= Overview =
Multi-Key Total Memory Encryption (MKTME)[1] is a technology that allows
transparent memory encryption in upcoming Intel platforms. It uses a new
instruction (PCONFIG) for key setup and selects a key for individual pages by
repurposing physical address bits in the page tables.
These patches add support for MKTME into the existing kernel keyring subsystem
and add a new mprotect_encrypt() system call that can be used by applications
to encrypt anonymous memory with keys obtained from the keyring.
This architecture supports encrypting both normal, volatile DRAM and persistent
memory. However, these patches do not implement persistent memory support. We
anticipate adding that support next.
== Hardware Background ==
MKTME is built on top of an existing single-key technology called TME. TME
encrypts all system memory using a single key generated by the CPU on every
boot of the system. TME provides mitigation against physical attacks, such as
physically removing a DIMM or watching memory bus traffic.
MKTME enables the use of multiple encryption keys[2], allowing selection of the
encryption key per-page using the page tables. Encryption keys are programmed
into each memory controller and the same set of keys is available to all
entities on the system with access to that memory (all cores, DMA engines,
etc...).
MKTME inherits many of the mitigations against hardware attacks from TME. Like
TME, MKTME does not mitigate vulnerable or malicious operating systems or
virtual machine managers. MKTME offers additional mitigations when compared to
TME.
TME and MKTME use the AES encryption algorithm in the AES-XTS mode. This mode,
typically used for block-based storage devices, takes the physical address of
the data into account when encrypting each block. This ensures that the
effective key is different for each block of memory. Moving encrypted content
across physical address results in garbage on read, mitigating block-relocation
attacks. This property is the reason many of the discussed attacks require
control of a shared physical page to be handed from the victim to the attacker.
== MKTME-Provided Mitigations ==
MKTME adds a few mitigations against attacks that are not mitigated when using
TME alone. The first set are mitigations against software attacks that are
familiar today:
* Kernel Mapping Attacks: information disclosures that leverage the
kernel direct map are mitigated against disclosing user data.
* Freed Data Leak Attacks: removing an encryption key from the
hardware mitigates future user information disclosure.
The next set are attacks that depend on specialized hardware, such as an “evil
DIMM” or a DDR interposer:
* Cross-Domain Replay Attack: data is captured from one domain
(guest) and replayed to another at a later time.
* Cross-Domain Capture and Delayed Compare Attack: data is captured
and later analyzed to discover secrets.
* Key Wear-out Attack: data is captured and analyzed in order to
Weaken the AES encryption itself.
More details on these attacks are below.
=== Kernel Mapping Attacks ===
Information disclosure vulnerabilities leverage the kernel direct map because
many vulnerabilities involve manipulation of kernel data structures (examples:
CVE-2017-7277, CVE-2017-9605). We normally think of these bugs as leaking
valuable *kernel* data, but they can leak application data when application
pages are recycled for kernel use.
With this MKTME implementation, there is a direct map created for each MKTME
KeyID which is used whenever the kernel needs to access plaintext. But, all
kernel data structures are accessed via the direct map for KeyID-0. Thus,
memory reads which are not coordinated with the KeyID get garbage (for example,
accessing KeyID-4 data with the KeyID-0 mapping).
This means that if sensitive data encrypted using MKTME is leaked via the
KeyID-0 direct map, ciphertext decrypted with the wrong key will be disclosed.
To disclose plaintext, an attacker must “pivot” to the correct direct mapping,
which is non-trivial because there are no kernel data structures in the
KeyID!=0 direct mapping.
=== Freed Data Leak Attack ===
The kernel has a history of bugs around uninitialized data. Usually, we think
of these bugs as leaking sensitive kernel data, but they can also be used to
leak application secrets.
MKTME can help mitigate the case where application secrets are leaked:
* App (or VM) places a secret in a page
* App exits or frees memory to kernel allocator
* Page added to allocator free list
* Attacker reallocates page to a purpose where it can read the page
Now, imagine MKTME was in use on the memory being leaked. The data can only be
leaked as long as the key is programmed in the hardware. If the key is
de-programmed, like after all pages are freed after a guest is shut down, any
future reads will just see ciphertext.
Basically, the key is a convenient choke-point: you can be more confident that
data encrypted with it is inaccessible once the key is removed.
=== Cross-Domain Replay Attack ===
MKTME mitigates cross-domain replay attacks where an attacker replaces an
encrypted block owned by one domain with a block owned by another domain.
MKTME does not prevent this replacement from occurring, but it does mitigate
plaintext from being disclosed if the domains use different keys.
With TME, the attack could be executed by:
* A victim places secret in memory, at a given physical address.
Note: AES-XTS is what restricts the attack to being performed at a
single physical address instead of across different physical
addresses
* Attacker captures victim secret’s ciphertext
* Later on, after victim frees the physical address, attacker gains
ownership
* Attacker puts the ciphertext at the address and get the secret
plaintext
But, due to the presumably different keys used by the attacker and the victim,
the attacker can not successfully decrypt old ciphertext.
=== Cross-Domain Capture and Delayed Compare Attack ===
This is also referred to as a kind of dictionary attack.
Similarly, MKTME protects against cross-domain capture-and-compare attacks.
Consider the following scenario:
* A victim places a secret in memory, at a known physical address
* Attacker captures victim’s ciphertext
* Attacker gains control of the target physical address, perhaps
after the victim’s VM is shut down or its memory reclaimed.
* Attacker computes and writes many possible plaintexts until new
ciphertext matches content captured previously.
Secrets which have low (plaintext) entropy are more vulnerable to this attack
because they reduce the number of possible plaintexts an attacker has to
compute and write.
The attack will not work if attacker and victim uses different keys.
=== Key Wear-out Attack ===
Repeated use of an encryption key might be used by an attacker to infer
information about the key or the plaintext, weakening the encryption. The
higher the bandwidth of the encryption engine, the more vulnerable the key is
to wear-out. The MKTME memory encryption hardware works at the speed of the
memory bus, which has high bandwidth.
Such a weakness has been demonstrated[3] on a theoretical cipher with similar
properties as AES-XTS.
An attack would take the following steps:
* Victim system is using TME with AES-XTS-128
* Attacker repeatedly captures ciphertext/plaintext pairs (can be
Performed with online hardware attack like an interposer).
* Attacker compels repeated use of the key under attack for a
sustained time period without a system reboot[4].
* Attacker discovers a cipertext collision (two plaintexts
translating to the same ciphertext)
* Attacker can induce controlled modifications to the targeted
plaintext by modifying the colliding ciphertext
MKTME mitigates key wear-out in two ways:
* Keys can be rotated periodically to mitigate wear-out. Since TME
keys are generated at boot, rotation of TME keys requires a
reboot. In contrast, MKTME allows rotation while the system is
booted. An application could implement a policy to rotate keys at
a frequency which is not feasible to attack.
* In the case that MKTME is used to encrypt two guests’ memory with
two different keys, an attack on one guest’s key would not weaken
the key used in the second guest.
--
[1] https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
[2] The MKTME architecture supports up to 16 bits of KeyIDs, so a
maximum of 65535 keys on top of the “TME key” at KeyID-0. The
first implementation is expected to support 5 bits, making 63 keys
available to applications. However, this is not guaranteed. The
number of available keys could be reduced if, for instance,
additional physical address space is desired over additional
KeyIDs.
[3] http://web.cs.ucdavis.edu/~rogaway/papers/offsets.pdf
[4] This sustained time required for an attack could vary from days
to years depending on the attacker’s goals.
Alison Schofield (33):
x86/pconfig: Set a valid encryption algorithm for all MKTME commands
keys/mktme: Introduce a Kernel Key Service for MKTME
keys/mktme: Preparse the MKTME key payload
keys/mktme: Instantiate and destroy MKTME keys
keys/mktme: Move the MKTME payload into a cache aligned structure
keys/mktme: Strengthen the entropy of CPU generated MKTME keys
keys/mktme: Set up PCONFIG programming targets for MKTME keys
keys/mktme: Program MKTME keys into the platform hardware
keys/mktme: Set up a percpu_ref_count for MKTME keys
keys/mktme: Require CAP_SYS_RESOURCE capability for MKTME keys
keys/mktme: Store MKTME payloads if cmdline parameter allows
acpi: Remove __init from acpi table parsing functions
acpi/hmat: Determine existence of an ACPI HMAT
keys/mktme: Require ACPI HMAT to register the MKTME Key Service
acpi/hmat: Evaluate topology presented in ACPI HMAT for MKTME
keys/mktme: Do not allow key creation in unsafe topologies
keys/mktme: Support CPU hotplug for MKTME key service
keys/mktme: Find new PCONFIG targets during memory hotplug
keys/mktme: Program new PCONFIG targets with MKTME keys
keys/mktme: Support memory hotplug for MKTME keys
mm: Generalize the mprotect implementation to support extensions
syscall/x86: Wire up a system call for MKTME encryption keys
x86/mm: Set KeyIDs in encrypted VMAs for MKTME
mm: Add the encrypt_mprotect() system call for MKTME
x86/mm: Keep reference counts on encrypted VMAs for MKTME
mm: Restrict MKTME memory encryption to anonymous VMAs
selftests/x86/mktme: Test the MKTME APIs
x86/mktme: Overview of Multi-Key Total Memory Encryption
x86/mktme: Document the MKTME provided security mitigations
x86/mktme: Document the MKTME kernel configuration requirements
x86/mktme: Document the MKTME Key Service API
x86/mktme: Document the MKTME API for anonymous memory encryption
x86/mktme: Demonstration program using the MKTME APIs
Jacob Pan (3):
iommu/vt-d: Support MKTME in DMA remapping
x86/mm: introduce common code for mem encryption
x86/mm: Use common code for DMA memory encryption
Kai Huang (2):
mm, x86: export several MKTME variables
kvm, x86, mmu: setup MKTME keyID to spte for given PFN
Kirill A. Shutemov (24):
mm: Do no merge VMAs with different encryption KeyIDs
mm: Add helpers to setup zero page mappings
mm/ksm: Do not merge pages with different KeyIDs
mm/page_alloc: Unify alloc_hugepage_vma()
mm/page_alloc: Handle allocation for encrypted memory
mm/khugepaged: Handle encrypted pages
x86/mm: Mask out KeyID bits from page table entry pfn
x86/mm: Introduce variables to store number, shift and mask of KeyIDs
x86/mm: Preserve KeyID on pte_modify() and pgprot_modify()
x86/mm: Detect MKTME early
x86/mm: Add a helper to retrieve KeyID for a page
x86/mm: Add a helper to retrieve KeyID for a VMA
x86/mm: Add hooks to allocate and free encrypted pages
x86/mm: Map zero pages into encrypted mappings correctly
x86/mm: Rename CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING
x86/mm: Allow to disable MKTME after enumeration
x86/mm: Calculate direct mapping size
x86/mm: Implement syncing per-KeyID direct mappings
x86/mm: Handle encrypted memory in page_to_virt() and __pa()
mm/page_ext: Export lookup_page_ext() symbol
mm/rmap: Clear vma->anon_vma on unlink_anon_vmas()
x86/mm: Disable MKTME on incompatible platform configurations
x86/mm: Disable MKTME if not all system memory supports encryption
x86: Introduce CONFIG_X86_INTEL_MKTME
.../admin-guide/kernel-parameters.rst | 1 +
.../admin-guide/kernel-parameters.txt | 11 +
Documentation/x86/mktme/index.rst | 13 +
.../x86/mktme/mktme_configuration.rst | 17 +
Documentation/x86/mktme/mktme_demo.rst | 53 ++
Documentation/x86/mktme/mktme_encrypt.rst | 57 ++
Documentation/x86/mktme/mktme_keys.rst | 96 +++
Documentation/x86/mktme/mktme_mitigations.rst | 150 ++++
Documentation/x86/mktme/mktme_overview.rst | 57 ++
Documentation/x86/x86_64/mm.txt | 4 +
arch/alpha/include/asm/page.h | 2 +-
arch/x86/Kconfig | 29 +-
arch/x86/entry/syscalls/syscall_32.tbl | 1 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/x86/include/asm/intel-family.h | 2 +
arch/x86/include/asm/intel_pconfig.h | 14 +-
arch/x86/include/asm/mem_encrypt.h | 29 +
arch/x86/include/asm/mktme.h | 93 +++
arch/x86/include/asm/page.h | 4 +
arch/x86/include/asm/page_32.h | 1 +
arch/x86/include/asm/page_64.h | 4 +-
arch/x86/include/asm/pgtable.h | 19 +
arch/x86/include/asm/pgtable_types.h | 23 +-
arch/x86/include/asm/setup.h | 6 +
arch/x86/kernel/cpu/intel.c | 58 +-
arch/x86/kernel/head64.c | 4 +
arch/x86/kernel/setup.c | 3 +
arch/x86/kvm/mmu.c | 18 +-
arch/x86/mm/Makefile | 3 +
arch/x86/mm/init_64.c | 68 ++
arch/x86/mm/kaslr.c | 11 +-
arch/x86/mm/mem_encrypt_common.c | 28 +
arch/x86/mm/mktme.c | 630 ++++++++++++++
drivers/acpi/hmat/hmat.c | 67 ++
drivers/acpi/tables.c | 10 +-
drivers/firmware/efi/efi.c | 25 +-
drivers/iommu/intel-iommu.c | 29 +-
fs/dax.c | 3 +-
fs/exec.c | 4 +-
fs/userfaultfd.c | 7 +-
include/asm-generic/pgtable.h | 8 +
include/keys/mktme-type.h | 39 +
include/linux/acpi.h | 9 +-
include/linux/dma-direct.h | 4 +-
include/linux/efi.h | 1 +
include/linux/gfp.h | 51 +-
include/linux/intel-iommu.h | 9 +-
include/linux/mem_encrypt.h | 23 +-
include/linux/migrate.h | 14 +-
include/linux/mm.h | 27 +-
include/linux/page_ext.h | 11 +-
include/linux/syscalls.h | 2 +
include/uapi/asm-generic/unistd.h | 4 +-
kernel/fork.c | 2 +
kernel/sys_ni.c | 2 +
mm/compaction.c | 3 +
mm/huge_memory.c | 6 +-
mm/khugepaged.c | 10 +
mm/ksm.c | 17 +
mm/madvise.c | 2 +-
mm/memory.c | 3 +-
mm/mempolicy.c | 30 +-
mm/migrate.c | 4 +-
mm/mlock.c | 2 +-
mm/mmap.c | 31 +-
mm/mprotect.c | 98 ++-
mm/page_alloc.c | 50 ++
mm/page_ext.c | 5 +
mm/rmap.c | 4 +-
mm/userfaultfd.c | 3 +-
security/keys/Makefile | 1 +
security/keys/mktme_keys.c | 768 ++++++++++++++++++
.../selftests/x86/mktme/encrypt_tests.c | 433 ++++++++++
.../testing/selftests/x86/mktme/flow_tests.c | 266 ++++++
tools/testing/selftests/x86/mktme/key_tests.c | 526 ++++++++++++
.../testing/selftests/x86/mktme/mktme_test.c | 300 +++++++
76 files changed, 4301 insertions(+), 122 deletions(-)
create mode 100644 Documentation/x86/mktme/index.rst
create mode 100644 Documentation/x86/mktme/mktme_configuration.rst
create mode 100644 Documentation/x86/mktme/mktme_demo.rst
create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
create mode 100644 Documentation/x86/mktme/mktme_keys.rst
create mode 100644 Documentation/x86/mktme/mktme_mitigations.rst
create mode 100644 Documentation/x86/mktme/mktme_overview.rst
create mode 100644 arch/x86/include/asm/mktme.h
create mode 100644 arch/x86/mm/mem_encrypt_common.c
create mode 100644 arch/x86/mm/mktme.c
create mode 100644 include/keys/mktme-type.h
create mode 100644 security/keys/mktme_keys.c
create mode 100644 tools/testing/selftests/x86/mktme/encrypt_tests.c
create mode 100644 tools/testing/selftests/x86/mktme/flow_tests.c
create mode 100644 tools/testing/selftests/x86/mktme/key_tests.c
create mode 100644 tools/testing/selftests/x86/mktme/mktme_test.c
--
2.20.1
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 03/62] mm/ksm: Do not merge pages with different KeyIDs
2019-05-08 14:43 ` [PATCH, RFC 03/62] mm/ksm: Do not merge pages with different KeyIDs Kirill A. Shutemov
@ 2019-05-10 18:07 ` Dave Hansen
2019-05-13 14:27 ` Kirill A. Shutemov0 siblings, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-05-10 18:07 UTC (permalink / raw)
To: Kirill A. Shutemov, Andrew Morton, x86, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
Andy Lutomirski, David Howells
Cc: Kees Cook, Kai Huang, Jacob Pan, Alison Schofield, linux-mm, kvm,
keyrings, linux-kernel
On 5/8/19 7:43 AM, Kirill A. Shutemov wrote:
> KeyID indicates what key to use to encrypt and decrypt page's content.
> Depending on the implementation a cipher text may be tied to physical
> address of the page. It means that pages with an identical plain text
> would appear different if KSM would look at a cipher text. It effectively
> disables KSM for encrypted pages.
>
> In addition, some implementations may not allow to read cipher text at all.
>
> KSM compares plain text instead (transparently to KSM code).
>
> But we still need to make sure that pages with identical plain text will
> not be merged together if they are encrypted with different keys.
>
> To make it work kernel only allows merging pages with the same KeyID.
> The approach guarantees that the merged page can be read by all users.
I can't really parse this description. Can I suggest replacement text?
Problem: KSM compares plain text. It might try to merge two pages that
have the same plain text but different ciphertext and possibly different
encryption keys. When the kernel encrypted the page, it promised that
it would keep it encrypted with _that_ key. That makes it impossible to
merge two pages encrypted with different keys.
Solution: Never merge encrypted pages with different KeyIDs.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 44/62] x86/mm: Set KeyIDs in encrypted VMAs for MKTME
2019-06-14 11:44 ` Peter Zijlstra@ 2019-06-14 17:33 ` Alison Schofield
2019-06-14 18:26 ` Dave Hansen0 siblings, 1 reply; 153+ messages in thread
From: Alison Schofield @ 2019-06-14 17:33 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Kirill A. Shutemov, Andrew Morton, x86, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Andy Lutomirski,
David Howells, Kees Cook, Dave Hansen, Kai Huang, Jacob Pan,
linux-mm, kvm, keyrings, linux-kernel
On Fri, Jun 14, 2019 at 01:44:08PM +0200, Peter Zijlstra wrote:
> On Wed, May 08, 2019 at 05:44:04PM +0300, Kirill A. Shutemov wrote:
> > From: Alison Schofield <alison.schofield@intel.com>
> >
> > MKTME architecture requires the KeyID to be placed in PTE bits 51:46.
> > To create an encrypted VMA, place the KeyID in the upper bits of
> > vm_page_prot that matches the position of those PTE bits.
> >
> > When the VMA is assigned a KeyID it is always considered a KeyID
> > change. The VMA is either going from not encrypted to encrypted,
> > or from encrypted with any KeyID to encrypted with any other KeyID.
> > To make the change safely, remove the user pages held by the VMA
> > and unlink the VMA's anonymous chain.
>
> This does not look like a transformation that preserves content; is
> mprotect() still a suitable name?
Data is not preserved across KeyID changes, by design.
Background:
We chose to implement encrypt_mprotect() as an extension
to the legacy mprotect so that memory allocated in any
method could be encrypted. ie. we didn't want to be tied
to mmap. As an mprotect extension, encrypt_mprotect also
supports the changing of access flags.
The usage we suggest is:
1) alloc the memory w PROT_NONE to prevent any usage before
encryption
2) use encrypt_mprotect() to add the key and change the
access to PROT_WRITE|PROT_READ.
Preserving the data across encryption key changes has not
been a requirement. I'm not clear if it was ever considered
and rejected. I believe that copying in order to preserve
the data was never considered.
Back to your naming question:
Since it is an mprotect extension, it seems we need to keep
the mprotect in the name.
Thanks for bringing it up. It would be good to hear more
thoughts on this.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 44/62] x86/mm: Set KeyIDs in encrypted VMAs for MKTME
2019-06-14 17:33 ` Alison Schofield@ 2019-06-14 18:26 ` Dave Hansen
2019-06-14 18:46 ` Alison Schofield0 siblings, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-06-14 18:26 UTC (permalink / raw)
To: Alison Schofield, Peter Zijlstra
Cc: Kirill A. Shutemov, Andrew Morton, x86, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Andy Lutomirski,
David Howells, Kees Cook, Kai Huang, Jacob Pan, linux-mm, kvm,
keyrings, linux-kernel
On 6/14/19 10:33 AM, Alison Schofield wrote:
> Preserving the data across encryption key changes has not
> been a requirement. I'm not clear if it was ever considered
> and rejected. I believe that copying in order to preserve
> the data was never considered.
We could preserve the data pretty easily. It's just annoying, though.
Right now, our only KeyID conversions happen in the page allocator. If
we were to convert in-place, we'd need something along the lines of:
1. Allocate a scratch page
2. Unmap target page, or at least make it entirely read-only
3. Copy plaintext into scratch page
4. Do cache KeyID conversion of page being converted:
Flush caches, change page_ext metadata
5. Copy plaintext back into target page from scratch area
6. Re-establish PTEs with new KeyID
#2 is *really* hard. It's similar to the problems that the poor
filesystem guys are having with RDMA these days when RDMA is doing writes.
What we have here (destroying existing data) is certainly the _simplest_
semantic. We can certainly give it a different name, or even non-PROT_*
semantics where it shares none of mprotect()'s functionality.
Doesn't really matter to me at all.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-05-08 14:44 ` [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call " Kirill A. Shutemov
2019-06-14 11:47 ` Peter Zijlstra
2019-06-14 11:51 ` Peter Zijlstra@ 2019-06-17 15:07 ` Andy Lutomirski
2019-06-17 15:28 ` Dave Hansen2 siblings, 1 reply; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-17 15:07 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, X86 ML, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Borislav Petkov, Peter Zijlstra, David Howells,
Kees Cook, Dave Hansen, Kai Huang, Jacob Pan, Alison Schofield,
Linux-MM, kvm list, keyrings, LKML
On Wed, May 8, 2019 at 7:44 AM Kirill A. Shutemov
<kirill.shutemov@linux.intel.com> wrote:
>
> From: Alison Schofield <alison.schofield@intel.com>
>
> Implement memory encryption for MKTME (Multi-Key Total Memory
> Encryption) with a new system call that is an extension of the
> legacy mprotect() system call.
>
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed MKTME encryption key. The key can be
> obtained through the kernel key service type "mktme". The caller
> must have KEY_NEED_VIEW permission on the key.
>
> MKTME places an additional restriction on the protected data:
> The length of the data must be page aligned. This is in addition
> to the existing mprotect restriction that the addr must be page
> aligned.
I still find it bizarre that this is conflated with mprotect().
I also remain entirely unconvinced that MKTME on anonymous memory is
useful in the long run. There will inevitably be all kinds of fancy
new CPU features that make the underlying MKTME mechanisms much more
useful. For example, some way to bind a key to a VM, or a way to
*sanely* encrypt persistent memory. By making this thing a syscall
that does more than just MKTME, you're adding combinatorial complexity
(you forget pkey!) and you're tying other functionality (change of
protection) to this likely-to-be-deprecated interface.
This is part of why I much prefer the idea of making this style of
MKTME a driver or some other non-intrusive interface. Then, once
everyone gets tired of it, the driver can just get turned off with no
side effects.
--Andy
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 15:07 ` Andy Lutomirski@ 2019-06-17 15:28 ` Dave Hansen
2019-06-17 15:46 ` Andy Lutomirski0 siblings, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-06-17 15:28 UTC (permalink / raw)
To: Andy Lutomirski, Kirill A. Shutemov
Cc: Andrew Morton, X86 ML, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Borislav Petkov, Peter Zijlstra, David Howells,
Kees Cook, Kai Huang, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML
On 6/17/19 8:07 AM, Andy Lutomirski wrote:
> I still find it bizarre that this is conflated with mprotect().
This needs to be in the changelog. But, for better or worse, it's
following the mprotect_pkey() pattern.
Other than the obvious "set the key on this memory", we're looking for
two other properties: atomicity (ensuring there is no transient state
where the memory is usable without the desired properties) and that it
is usable on existing allocations.
For atomicity, we have a model where we can allocate things with
PROT_NONE, then do mprotect_pkey() and mprotect_encrypt() (plus any
future features), then the last mprotect_*() call takes us from
PROT_NONE to the desired end permisions. We could just require a plain
old mprotect() to do that instead of embedding mprotect()-like behavior
in these, of course, but that isn't the path we're on at the moment with
mprotect_pkey().
So, for this series it's just a matter of whether we do this:
ptr = mmap(..., PROT_NONE);
mprotect_pkey(protect_key, ptr, PROT_NONE);
mprotect_encrypt(encr_key, ptr, PROT_READ|PROT_WRITE);
// good to go
or this:
ptr = mmap(..., PROT_NONE);
mprotect_pkey(protect_key, ptr, PROT_NONE);
sys_encrypt(key, ptr);
mprotect(ptr, PROT_READ|PROT_WRITE);
// good to go
I actually don't care all that much which one we end up with. It's not
like the extra syscall in the second options means much.
> This is part of why I much prefer the idea of making this style of
> MKTME a driver or some other non-intrusive interface. Then, once
> everyone gets tired of it, the driver can just get turned off with no
> side effects.
I like the concept, but not where it leads. I'd call it the 'hugetlbfs
approach". :) Hugetblfs certainly go us huge pages, but it's continued
to be a parallel set of code with parallel bugs and parallel
implementations of many VM features. It's not that you can't implement
new things on hugetlbfs, it's that you *need* to. You never get them
for free.
For instance, if we do a driver, how do we get large pages? How do we
swap/reclaim the pages? How do we do NUMA affinity? How do we
eventually stack it on top of persistent memory filesystems or Device
DAX? With a driver approach, I think we're stuck basically
reimplementing things or gluing them back together. Nothing comes for free.
With this approach, we basically start with our normal, full feature set
(modulo weirdo interactions like with KSM).
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 15:28 ` Dave Hansen@ 2019-06-17 15:46 ` Andy Lutomirski
2019-06-17 18:27 ` Dave Hansen0 siblings, 1 reply; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-17 15:46 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Kirill A. Shutemov, Andrew Morton, X86 ML,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Borislav Petkov,
Peter Zijlstra, David Howells, Kees Cook, Kai Huang, Jacob Pan,
Alison Schofield, Linux-MM, kvm list, keyrings, LKML
On Mon, Jun 17, 2019 at 8:28 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 6/17/19 8:07 AM, Andy Lutomirski wrote:
> > I still find it bizarre that this is conflated with mprotect().
>
> This needs to be in the changelog. But, for better or worse, it's
> following the mprotect_pkey() pattern.
>
> Other than the obvious "set the key on this memory", we're looking for
> two other properties: atomicity (ensuring there is no transient state
> where the memory is usable without the desired properties) and that it
> is usable on existing allocations.
>
> For atomicity, we have a model where we can allocate things with
> PROT_NONE, then do mprotect_pkey() and mprotect_encrypt() (plus any
> future features), then the last mprotect_*() call takes us from
> PROT_NONE to the desired end permisions. We could just require a plain
> old mprotect() to do that instead of embedding mprotect()-like behavior
> in these, of course, but that isn't the path we're on at the moment with
> mprotect_pkey().
>
> So, for this series it's just a matter of whether we do this:
>
> ptr = mmap(..., PROT_NONE);
> mprotect_pkey(protect_key, ptr, PROT_NONE);
> mprotect_encrypt(encr_key, ptr, PROT_READ|PROT_WRITE);
> // good to go
>
> or this:
>
> ptr = mmap(..., PROT_NONE);
> mprotect_pkey(protect_key, ptr, PROT_NONE);
> sys_encrypt(key, ptr);
> mprotect(ptr, PROT_READ|PROT_WRITE);
> // good to go
>
> I actually don't care all that much which one we end up with. It's not
> like the extra syscall in the second options means much.
The benefit of the second one is that, if sys_encrypt is absent, it
just works. In the first model, programs need a fallback because
they'll segfault of mprotect_encrypt() gets ENOSYS.
>
> > This is part of why I much prefer the idea of making this style of
> > MKTME a driver or some other non-intrusive interface. Then, once
> > everyone gets tired of it, the driver can just get turned off with no
> > side effects.
>
> I like the concept, but not where it leads. I'd call it the 'hugetlbfs
> approach". :) Hugetblfs certainly go us huge pages, but it's continued
> to be a parallel set of code with parallel bugs and parallel
> implementations of many VM features. It's not that you can't implement
> new things on hugetlbfs, it's that you *need* to. You never get them
> for free.
Fair enough, but...
>
> For instance, if we do a driver, how do we get large pages? How do we
> swap/reclaim the pages? How do we do NUMA affinity?
Those all make sense.
> How do we
> eventually stack it on top of persistent memory filesystems or Device
> DAX?
How do we stack anonymous memory on top of persistent memory or Device
DAX? I'm confused.
Just to throw this out there, what if we had a new device /dev/xpfo
and MKTME were one of its features. You open /dev/xpfo, optionally do
an ioctl to set a key, and them map it. The pages you get are
unmapped entirely from the direct map, and you get a PFNMAP VMA with
all its limitations. This seems much more useful -- it's limited, but
it's limited *because the kernel can't accidentally read it*.
I think that, in the long run, we're going to have to either expand
the core mm's concept of what "memory" is or just have a whole
parallel set of mechanisms for memory that doesn't work like memory.
We're already accumulating a set of things that are backed by memory
but aren't usable as memory. SGX EPC pages and SEV pages come to mind.
They are faster when they're in big contiguous chunks (well, not SGX
AFAIK, but maybe some day), they have NUMA node affinity, and they
show up in page tables, but the hardware restricts who can read and
write them. If Intel isn't planning to do something like this with
the MKTME hardware, I'll eat my hat.
I expect that some day normal memory will be able to be repurposed as
SGX pages on the fly, and that will also look a lot more like SEV or
XPFO than like the this model of MKTME.
So, if we upstream MKTME as anonymous memory with a magic config
syscall, I predict that, in a few years, it will be end up inheriting
all downsides of both approaches with few of the upsides. Programs
like QEMU will need to learn to manipulate pages that can't be
accessed outside the VM without special VM buy-in, so the fact that
MKTME pages are fully functional and can be GUP-ed won't be very
useful. And the VM will learn about all these things, but MKTME won't
really fit in.
And, one of these days, someone will come up with a version of XPFO
that could actually be upstreamed, and it seems entirely plausible
that it will be totally incompatible with MKTME-as-anonymous-memory
and that users of MKTME will actually get *worse* security.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 15:46 ` Andy Lutomirski@ 2019-06-17 18:27 ` Dave Hansen
2019-06-17 19:12 ` Andy Lutomirski
2019-06-17 23:59 ` Kai Huang0 siblings, 2 replies; 153+ messages in thread
From: Dave Hansen @ 2019-06-17 18:27 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Kai Huang, Jacob Pan, Alison Schofield,
Linux-MM, kvm list, keyrings, LKML, Tom Lendacky
Tom Lendacky, could you take a look down in the message to the talk of
SEV? I want to make sure I'm not misrepresenting what it does today.
...
>> I actually don't care all that much which one we end up with. It's not
>> like the extra syscall in the second options means much.
>
> The benefit of the second one is that, if sys_encrypt is absent, it
> just works. In the first model, programs need a fallback because
> they'll segfault of mprotect_encrypt() gets ENOSYS.
Well, by the time they get here, they would have already had to allocate
and set up the encryption key. I don't think this would really be the
"normal" malloc() path, for instance.
>> How do we
>> eventually stack it on top of persistent memory filesystems or Device
>> DAX?
>
> How do we stack anonymous memory on top of persistent memory or Device
> DAX? I'm confused.
If our interface to MKTME is:
fd = open("/dev/mktme");
ptr = mmap(fd);
Then it's hard to combine with an interface which is:
fd = open("/dev/dax123");
ptr = mmap(fd);
Where if we have something like mprotect() (or madvise() or something
else taking pointer), we can just do:
fd = open("/dev/anything987");
ptr = mmap(fd);
sys_encrypt(ptr);
Now, we might not *do* it that way for dax, for instance, but I'm just
saying that if we go the /dev/mktme route, we never get a choice.
> I think that, in the long run, we're going to have to either expand
> the core mm's concept of what "memory" is or just have a whole
> parallel set of mechanisms for memory that doesn't work like memory.
...
> I expect that some day normal memory will be able to be repurposed as
> SGX pages on the fly, and that will also look a lot more like SEV or
> XPFO than like the this model of MKTME.
I think you're drawing the line at pages where the kernel can manage
contents vs. not manage contents. I'm not sure that's the right
distinction to make, though. The thing that is important is whether the
kernel can manage the lifetime and location of the data in the page.
Basically: Can the kernel choose where the page comes from and get the
page back when it wants?
I really don't like the current state of things like with SEV or with
KVM direct device assignment where the physical location is quite locked
down and the kernel really can't manage the memory. I'm trying really
hard to make sure future hardware is more permissive about such things.
My hope is that these are a temporary blip and not the new normal.
> So, if we upstream MKTME as anonymous memory with a magic config
> syscall, I predict that, in a few years, it will be end up inheriting
> all downsides of both approaches with few of the upsides. Programs
> like QEMU will need to learn to manipulate pages that can't be
> accessed outside the VM without special VM buy-in, so the fact that
> MKTME pages are fully functional and can be GUP-ed won't be very
> useful. And the VM will learn about all these things, but MKTME won't
> really fit in.
Kai Huang (who is on cc) has been doing the QEMU enabling and might want
to weigh in. I'd also love to hear from the AMD folks in case I'm not
grokking some aspect of SEV.
But, my understanding is that, even today, neither QEMU nor the kernel
can see SEV-encrypted guest memory. So QEMU should already understand
how to not interact with guest memory. I _assume_ it's also already
doing this with anonymous memory, without needing /dev/sme or something.
> And, one of these days, someone will come up with a version of XPFO
> that could actually be upstreamed, and it seems entirely plausible
> that it will be totally incompatible with MKTME-as-anonymous-memory
> and that users of MKTME will actually get *worse* security.
I'm not following here. XPFO just means that we don't keep the direct
map around all the time for all memory. If XPFO and
MKTME-as-anonymous-memory were both in play, I think we'd just be
creating/destroying the MKTME-enlightened direct map instead of a
vanilla one.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 18:27 ` Dave Hansen@ 2019-06-17 19:12 ` Andy Lutomirski
2019-06-17 21:36 ` Dave Hansen
2019-06-18 0:05 ` Kai Huang
2019-06-17 23:59 ` Kai Huang1 sibling, 2 replies; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-17 19:12 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Kirill A. Shutemov, Andrew Morton, X86 ML,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Borislav Petkov,
Peter Zijlstra, David Howells, Kees Cook, Kai Huang, Jacob Pan,
Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On Mon, Jun 17, 2019 at 11:37 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> Tom Lendacky, could you take a look down in the message to the talk of
> SEV? I want to make sure I'm not misrepresenting what it does today.
> ...
>
>
> >> I actually don't care all that much which one we end up with. It's not
> >> like the extra syscall in the second options means much.
> >
> > The benefit of the second one is that, if sys_encrypt is absent, it
> > just works. In the first model, programs need a fallback because
> > they'll segfault of mprotect_encrypt() gets ENOSYS.
>
> Well, by the time they get here, they would have already had to allocate
> and set up the encryption key. I don't think this would really be the
> "normal" malloc() path, for instance.
>
> >> How do we
> >> eventually stack it on top of persistent memory filesystems or Device
> >> DAX?
> >
> > How do we stack anonymous memory on top of persistent memory or Device
> > DAX? I'm confused.
>
> If our interface to MKTME is:
>
> fd = open("/dev/mktme");
> ptr = mmap(fd);
>
> Then it's hard to combine with an interface which is:
>
> fd = open("/dev/dax123");
> ptr = mmap(fd);
>
> Where if we have something like mprotect() (or madvise() or something
> else taking pointer), we can just do:
>
> fd = open("/dev/anything987");
> ptr = mmap(fd);
> sys_encrypt(ptr);
I'm having a hard time imagining that ever working -- wouldn't it blow
up if someone did:
fd = open("/dev/anything987");
ptr1 = mmap(fd);
ptr2 = mmap(fd);
sys_encrypt(ptr1);
So I think it really has to be:
fd = open("/dev/anything987");
ioctl(fd, ENCRYPT_ME);
mmap(fd);
But I really expect that the encryption of a DAX device will actually
be a block device setting and won't look like this at all. It'll be
more like dm-crypt except without device mapper.
>
> Now, we might not *do* it that way for dax, for instance, but I'm just
> saying that if we go the /dev/mktme route, we never get a choice.
>
> > I think that, in the long run, we're going to have to either expand
> > the core mm's concept of what "memory" is or just have a whole
> > parallel set of mechanisms for memory that doesn't work like memory.
> ...
> > I expect that some day normal memory will be able to be repurposed as
> > SGX pages on the fly, and that will also look a lot more like SEV or
> > XPFO than like the this model of MKTME.
>
> I think you're drawing the line at pages where the kernel can manage
> contents vs. not manage contents. I'm not sure that's the right
> distinction to make, though. The thing that is important is whether the
> kernel can manage the lifetime and location of the data in the page.
The kernel can manage the location of EPC pages, for example, but only
under extreme constraints right now. The draft SGX driver can and
does swap them out and swap them back in, potentially at a different
address.
>
> Basically: Can the kernel choose where the page comes from and get the
> page back when it wants?
>
> I really don't like the current state of things like with SEV or with
> KVM direct device assignment where the physical location is quite locked
> down and the kernel really can't manage the memory. I'm trying really
> hard to make sure future hardware is more permissive about such things.
> My hope is that these are a temporary blip and not the new normal.
>
> > So, if we upstream MKTME as anonymous memory with a magic config
> > syscall, I predict that, in a few years, it will be end up inheriting
> > all downsides of both approaches with few of the upsides. Programs
> > like QEMU will need to learn to manipulate pages that can't be
> > accessed outside the VM without special VM buy-in, so the fact that
> > MKTME pages are fully functional and can be GUP-ed won't be very
> > useful. And the VM will learn about all these things, but MKTME won't
> > really fit in.
>
> Kai Huang (who is on cc) has been doing the QEMU enabling and might want
> to weigh in. I'd also love to hear from the AMD folks in case I'm not
> grokking some aspect of SEV.
>
> But, my understanding is that, even today, neither QEMU nor the kernel
> can see SEV-encrypted guest memory. So QEMU should already understand
> how to not interact with guest memory. I _assume_ it's also already
> doing this with anonymous memory, without needing /dev/sme or something.
Let's find out!
If it's using anonymous memory, it must be very strange, since it
would more or less have to be mmapped PROT_NONE. The thing that makes
anonymous memory in particular seem so awkward to me is that it's
fundamentally identified by it's *address*, which means it makes no
sense if it's not mapped.
>
> > And, one of these days, someone will come up with a version of XPFO
> > that could actually be upstreamed, and it seems entirely plausible
> > that it will be totally incompatible with MKTME-as-anonymous-memory
> > and that users of MKTME will actually get *worse* security.
>
> I'm not following here. XPFO just means that we don't keep the direct
> map around all the time for all memory. If XPFO and
> MKTME-as-anonymous-memory were both in play, I think we'd just be
> creating/destroying the MKTME-enlightened direct map instead of a
> vanilla one.
What I'm saying is that I can imagine XPFO also wanting to be
something other than anonymous memory. I don't think we'll ever want
regular MAP_ANONYMOUS to enable XPFO by default because the
performance will suck. Doing this seems odd:
ptr = mmap(MAP_ANONYMOUS);
sys_xpfo_a_pointer(ptr);
So I could imagine:
ptr = mmap(MAP_ANONYMOUS | MAP_XPFO);
or
fd = open("/dev/xpfo"); (or fd = memfd_create(..., XPFO);
ptr = mmap(fd);
I'm thinking that XPFO is a *lot* simpler under the hood if we just
straight-up don't support GUP on it. Maybe we should call this
"strong XPFO". Similarly, the kinds of things that want MKTME may
also want the memory to be entirely absent from the direct map. And
the things that use SEV (as I understand it) *can't* usefully use the
memory for normal IO via GUP or copy_to/from_user(), so these things
all have a decent amount in common.
Another down side of anonymous memory (in my head, anyway -- QEMU
people should chime in) is that it seems awkward to use it for IO
techniques in which the back-end isn't in the QEMU process. If
there's an fd involved, you can pass it around, feed it to things like
vfio, etc. If there's no fd, it's stuck in the creating process.
And another silly argument: if we had /dev/mktme, then we could
possibly get away with avoiding all the keyring stuff entirely.
Instead, you open /dev/mktme and you get your own key under the hook.
If you want two keys, you open /dev/mktme twice. If you want some
other program to be able to see your memory, you pass it the fd.
I hope this email isn't too rambling :)
--Andy
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 19:12 ` Andy Lutomirski@ 2019-06-17 21:36 ` Dave Hansen
2019-06-18 0:48 ` Kai Huang
2019-06-18 0:05 ` Kai Huang1 sibling, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-06-17 21:36 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Kai Huang, Jacob Pan, Alison Schofield,
Linux-MM, kvm list, keyrings, LKML, Tom Lendacky
>> Where if we have something like mprotect() (or madvise() or something
>> else taking pointer), we can just do:
>>
>> fd = open("/dev/anything987");
>> ptr = mmap(fd);
>> sys_encrypt(ptr);
>
> I'm having a hard time imagining that ever working -- wouldn't it blow
> up if someone did:
>
> fd = open("/dev/anything987");
> ptr1 = mmap(fd);
> ptr2 = mmap(fd);
> sys_encrypt(ptr1);
>
> So I think it really has to be:
> fd = open("/dev/anything987");
> ioctl(fd, ENCRYPT_ME);
> mmap(fd);
Yeah, shared mappings are annoying. :)
But, let's face it, nobody is going to do what you suggest in the
ptr1/ptr2 example. It doesn't make any logical sense because it's
effectively asking to read the memory with two different keys. I
_believe_ fscrypt has similar issues and just punts on them by saying
"don't do that".
We can also quite easily figure out what's going on. It's a very simple
rule to kill a process that tries to fault a page in whose KeyID doesn't
match the VMA under which it is faulted in, and also require that no
pages are faulted in under VMAs which have their key changed.
>> Now, we might not *do* it that way for dax, for instance, but I'm just
>> saying that if we go the /dev/mktme route, we never get a choice.
>>
>>> I think that, in the long run, we're going to have to either expand
>>> the core mm's concept of what "memory" is or just have a whole
>>> parallel set of mechanisms for memory that doesn't work like memory.
>> ...
>>> I expect that some day normal memory will be able to be repurposed as
>>> SGX pages on the fly, and that will also look a lot more like SEV or
>>> XPFO than like the this model of MKTME.
>>
>> I think you're drawing the line at pages where the kernel can manage
>> contents vs. not manage contents. I'm not sure that's the right
>> distinction to make, though. The thing that is important is whether the
>> kernel can manage the lifetime and location of the data in the page.
>
> The kernel can manage the location of EPC pages, for example, but only
> under extreme constraints right now. The draft SGX driver can and
> does swap them out and swap them back in, potentially at a different
> address.
The kernel can't put arbitrary data in EPC pages and can't use normal
memory for EPC. To me, that puts them clearly on the side of being
unmanageable by the core mm code.
For instance, there's no way we could mix EPC pages in the same 'struct
zone' with non-EPC pages. Not only are they not in the direct map, but
they never *can* be, even for a second.
>>> And, one of these days, someone will come up with a version of XPFO
>>> that could actually be upstreamed, and it seems entirely plausible
>>> that it will be totally incompatible with MKTME-as-anonymous-memory
>>> and that users of MKTME will actually get *worse* security.
>>
>> I'm not following here. XPFO just means that we don't keep the direct
>> map around all the time for all memory. If XPFO and
>> MKTME-as-anonymous-memory were both in play, I think we'd just be
>> creating/destroying the MKTME-enlightened direct map instead of a
>> vanilla one.
>
> What I'm saying is that I can imagine XPFO also wanting to be
> something other than anonymous memory. I don't think we'll ever want
> regular MAP_ANONYMOUS to enable XPFO by default because the
> performance will suck.
It will certainly suck for some things. But, does it suck if the kernel
never uses the direct map for the XPFO memory? If it were for KVM guest
memory for a guest using direct device assignment, we might not even
ever notice.
> I'm thinking that XPFO is a *lot* simpler under the hood if we just
> straight-up don't support GUP on it. Maybe we should call this
> "strong XPFO". Similarly, the kinds of things that want MKTME may
> also want the memory to be entirely absent from the direct map. And
> the things that use SEV (as I understand it) *can't* usefully use the
> memory for normal IO via GUP or copy_to/from_user(), so these things
> all have a decent amount in common.
OK, so basically, you're thinking about new memory management
infrastructure that a memory-allocating-app can opt into where they get
a reduced kernel feature set, but also increased security guarantees?
The main insight thought is that some hardware features *already*
impose (some of) this reduced feature set?
FWIW, I don't think many folks will go for the no-GUP rule. It's one
thing to say no-GUPs for SGX pages which can't have I/O done on them in
the first place, but it's quite another to tell folks that sendfile() no
longer works without bounce buffers.
MKTME's security guarantees are very different than something like SEV.
Since the kernel is in the trust boundary, it *can* do fun stuff like
RDMA which is a heck of a lot faster than bounce buffering. Let's say a
franken-system existed with SEV and MKTME. It isn't even clear to me
that *everyone* would pick SEV over MKTME. IOW, I'm not sure the MKTME
model necessarily goes away given the presence of SEV.
> And another silly argument: if we had /dev/mktme, then we could
> possibly get away with avoiding all the keyring stuff entirely.
> Instead, you open /dev/mktme and you get your own key under the hook.
> If you want two keys, you open /dev/mktme twice. If you want some
> other program to be able to see your memory, you pass it the fd.
We still like the keyring because it's one-stop-shopping as the place
that *owns* the hardware KeyID slots. Those are global resources and
scream for a single global place to allocate and manage them. The
hardware slots also need to be shared between any anonymous and
file-based users, no matter what the APIs for the anonymous side.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 21:36 ` Dave Hansen@ 2019-06-18 0:48 ` Kai Huang
2019-06-18 1:50 ` Andy Lutomirski0 siblings, 1 reply; 153+ messages in thread
From: Kai Huang @ 2019-06-18 0:48 UTC (permalink / raw)
To: Dave Hansen, Andy Lutomirski
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML, Tom Lendacky
>
> > And another silly argument: if we had /dev/mktme, then we could
> > possibly get away with avoiding all the keyring stuff entirely.
> > Instead, you open /dev/mktme and you get your own key under the hook.
> > If you want two keys, you open /dev/mktme twice. If you want some
> > other program to be able to see your memory, you pass it the fd.
>
> We still like the keyring because it's one-stop-shopping as the place
> that *owns* the hardware KeyID slots. Those are global resources and
> scream for a single global place to allocate and manage them. The
> hardware slots also need to be shared between any anonymous and
> file-based users, no matter what the APIs for the anonymous side.
MKTME driver (who creates /dev/mktme) can also be the one-stop-shopping. I think whether to choose
keyring to manage MKTME key should be based on whether we need/should take advantage of existing key
retention service functionalities. For example, with key retention service we can
revoke/invalidate/set expiry for a key (not sure whether MKTME needs those although), and we have
several keyrings -- thread specific keyring, process specific keyring, user specific keyring, etc,
thus we can control who can/cannot find the key, etc. I think managing MKTME key in MKTME driver
doesn't have those advantages.
Thanks,
-Kai
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 0:48 ` Kai Huang@ 2019-06-18 1:50 ` Andy Lutomirski
2019-06-18 2:11 ` Kai Huang
2019-06-18 14:19 ` Dave Hansen0 siblings, 2 replies; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-18 1:50 UTC (permalink / raw)
To: Kai Huang
Cc: Dave Hansen, Andy Lutomirski, Kirill A. Shutemov, Andrew Morton,
X86 ML, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Borislav Petkov, Peter Zijlstra, David Howells, Kees Cook,
Jacob Pan, Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On Mon, Jun 17, 2019 at 5:48 PM Kai Huang <kai.huang@linux.intel.com> wrote:
>
>
> >
> > > And another silly argument: if we had /dev/mktme, then we could
> > > possibly get away with avoiding all the keyring stuff entirely.
> > > Instead, you open /dev/mktme and you get your own key under the hook.
> > > If you want two keys, you open /dev/mktme twice. If you want some
> > > other program to be able to see your memory, you pass it the fd.
> >
> > We still like the keyring because it's one-stop-shopping as the place
> > that *owns* the hardware KeyID slots. Those are global resources and
> > scream for a single global place to allocate and manage them. The
> > hardware slots also need to be shared between any anonymous and
> > file-based users, no matter what the APIs for the anonymous side.
>
> MKTME driver (who creates /dev/mktme) can also be the one-stop-shopping. I think whether to choose
> keyring to manage MKTME key should be based on whether we need/should take advantage of existing key
> retention service functionalities. For example, with key retention service we can
> revoke/invalidate/set expiry for a key (not sure whether MKTME needs those although), and we have
> several keyrings -- thread specific keyring, process specific keyring, user specific keyring, etc,
> thus we can control who can/cannot find the key, etc. I think managing MKTME key in MKTME driver
> doesn't have those advantages.
>
Trying to evaluate this with the current proposed code is a bit odd, I
think. Suppose you create a thread-specific key and then fork(). The
child can presumably still use the key regardless of whether the child
can nominally access the key in the keyring because the PTEs are still
there.
More fundamentally, in some sense, the current code has no semantics.
Associating a key with memory and "encrypting" it doesn't actually do
anything unless you are attacking the memory bus but you haven't
compromised the kernel. There's no protection against a guest that
can corrupt its EPT tables, there's no protection against kernel bugs
(*especially* if the duplicate direct map design stays), and there
isn't even any fd or other object around by which you can only access
the data if you can see the key.
I'm also wondering whether the kernel will always be able to be a
one-stop shop for key allocation -- if the MKTME hardware gains
interesting new uses down the road, who knows how key allocation will
work?
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 1:50 ` Andy Lutomirski@ 2019-06-18 2:11 ` Kai Huang
2019-06-18 4:24 ` Andy Lutomirski
2019-06-18 14:19 ` Dave Hansen1 sibling, 1 reply; 153+ messages in thread
From: Kai Huang @ 2019-06-18 2:11 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Dave Hansen, Kirill A. Shutemov, Andrew Morton, X86 ML,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Borislav Petkov,
Peter Zijlstra, David Howells, Kees Cook, Jacob Pan,
Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On Mon, 2019-06-17 at 18:50 -0700, Andy Lutomirski wrote:
> On Mon, Jun 17, 2019 at 5:48 PM Kai Huang <kai.huang@linux.intel.com> wrote:
> >
> >
> > >
> > > > And another silly argument: if we had /dev/mktme, then we could
> > > > possibly get away with avoiding all the keyring stuff entirely.
> > > > Instead, you open /dev/mktme and you get your own key under the hook.
> > > > If you want two keys, you open /dev/mktme twice. If you want some
> > > > other program to be able to see your memory, you pass it the fd.
> > >
> > > We still like the keyring because it's one-stop-shopping as the place
> > > that *owns* the hardware KeyID slots. Those are global resources and
> > > scream for a single global place to allocate and manage them. The
> > > hardware slots also need to be shared between any anonymous and
> > > file-based users, no matter what the APIs for the anonymous side.
> >
> > MKTME driver (who creates /dev/mktme) can also be the one-stop-shopping. I think whether to
> > choose
> > keyring to manage MKTME key should be based on whether we need/should take advantage of existing
> > key
> > retention service functionalities. For example, with key retention service we can
> > revoke/invalidate/set expiry for a key (not sure whether MKTME needs those although), and we
> > have
> > several keyrings -- thread specific keyring, process specific keyring, user specific keyring,
> > etc,
> > thus we can control who can/cannot find the key, etc. I think managing MKTME key in MKTME driver
> > doesn't have those advantages.
> >
>
> Trying to evaluate this with the current proposed code is a bit odd, I
> think. Suppose you create a thread-specific key and then fork(). The
> child can presumably still use the key regardless of whether the child
> can nominally access the key in the keyring because the PTEs are still
> there.
Right. This is a little bit odd, although virtualization (Qemu, which is the main use case of MKTME
at least so far) doesn't use fork().
>
> More fundamentally, in some sense, the current code has no semantics.
> Associating a key with memory and "encrypting" it doesn't actually do
> anything unless you are attacking the memory bus but you haven't
> compromised the kernel. There's no protection against a guest that
> can corrupt its EPT tables, there's no protection against kernel bugs
> (*especially* if the duplicate direct map design stays), and there
> isn't even any fd or other object around by which you can only access
> the data if you can see the key.
I am not saying managing MKTME key/keyID in key retention service is definitely better, but it seems
all those you mentioned are not related to whether to choose key retention service to manage MKTME
key/keyID? Or you are saying it doesn't matter we manage key/keyID in key retention service or in
MKTME driver, since MKTME barely have any security benefits (besides physical attack)?
>
> I'm also wondering whether the kernel will always be able to be a
> one-stop shop for key allocation -- if the MKTME hardware gains
> interesting new uses down the road, who knows how key allocation will
> work?
I by now don't have any use case which requires to manage key/keyID specifically for its own use,
rather than letting kernel to manage keyID allocation. Please inspire us if you have any potential.
Thanks,
-Kai
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 2:11 ` Kai Huang@ 2019-06-18 4:24 ` Andy Lutomirski0 siblings, 0 replies; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-18 4:24 UTC (permalink / raw)
To: Kai Huang
Cc: Andy Lutomirski, Dave Hansen, Kirill A. Shutemov, Andrew Morton,
X86 ML, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Borislav Petkov, Peter Zijlstra, David Howells, Kees Cook,
Jacob Pan, Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On Mon, Jun 17, 2019 at 7:11 PM Kai Huang <kai.huang@linux.intel.com> wrote:
>
> On Mon, 2019-06-17 at 18:50 -0700, Andy Lutomirski wrote:
> > On Mon, Jun 17, 2019 at 5:48 PM Kai Huang <kai.huang@linux.intel.com> wrote:
> > >
> > >
> > > >
> > > > > And another silly argument: if we had /dev/mktme, then we could
> > > > > possibly get away with avoiding all the keyring stuff entirely.
> > > > > Instead, you open /dev/mktme and you get your own key under the hook.
> > > > > If you want two keys, you open /dev/mktme twice. If you want some
> > > > > other program to be able to see your memory, you pass it the fd.
> > > >
> > > > We still like the keyring because it's one-stop-shopping as the place
> > > > that *owns* the hardware KeyID slots. Those are global resources and
> > > > scream for a single global place to allocate and manage them. The
> > > > hardware slots also need to be shared between any anonymous and
> > > > file-based users, no matter what the APIs for the anonymous side.
> > >
> > > MKTME driver (who creates /dev/mktme) can also be the one-stop-shopping. I think whether to
> > > choose
> > > keyring to manage MKTME key should be based on whether we need/should take advantage of existing
> > > key
> > > retention service functionalities. For example, with key retention service we can
> > > revoke/invalidate/set expiry for a key (not sure whether MKTME needs those although), and we
> > > have
> > > several keyrings -- thread specific keyring, process specific keyring, user specific keyring,
> > > etc,
> > > thus we can control who can/cannot find the key, etc. I think managing MKTME key in MKTME driver
> > > doesn't have those advantages.
> > >
> >
> > Trying to evaluate this with the current proposed code is a bit odd, I
> > think. Suppose you create a thread-specific key and then fork(). The
> > child can presumably still use the key regardless of whether the child
> > can nominally access the key in the keyring because the PTEs are still
> > there.
>
> Right. This is a little bit odd, although virtualization (Qemu, which is the main use case of MKTME
> at least so far) doesn't use fork().
>
> >
> > More fundamentally, in some sense, the current code has no semantics.
> > Associating a key with memory and "encrypting" it doesn't actually do
> > anything unless you are attacking the memory bus but you haven't
> > compromised the kernel. There's no protection against a guest that
> > can corrupt its EPT tables, there's no protection against kernel bugs
> > (*especially* if the duplicate direct map design stays), and there
> > isn't even any fd or other object around by which you can only access
> > the data if you can see the key.
>
> I am not saying managing MKTME key/keyID in key retention service is definitely better, but it seems
> all those you mentioned are not related to whether to choose key retention service to manage MKTME
> key/keyID? Or you are saying it doesn't matter we manage key/keyID in key retention service or in
> MKTME driver, since MKTME barely have any security benefits (besides physical attack)?
Mostly the latter. I think it's very hard to evaluate whether a given
key allocation model makes sense given that MKTME provides such weak
security benefits. TME has obvious security benefits, as does
encryption of persistent memory, but this giant patch set isn't needed
for plain TME and it doesn't help with persistent memory.
>
> >
> > I'm also wondering whether the kernel will always be able to be a
> > one-stop shop for key allocation -- if the MKTME hardware gains
> > interesting new uses down the road, who knows how key allocation will
> > work?
>
> I by now don't have any use case which requires to manage key/keyID specifically for its own use,
> rather than letting kernel to manage keyID allocation. Please inspire us if you have any potential.
>
Other than compliance, I can't think of much reason that using
multiple keys is useful, regardless of how their allocated. The only
thing I've thought of is that, with multiple keys, you can use PCONFIG
to remove one and flush caches and the data is most definitely gone.
On the other hand, you can just zero the memory and the data is just
as gone even without any encryption.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 9:12 ` Peter Zijlstra@ 2019-06-18 14:09 ` Dave Hansen
2019-06-18 16:15 ` Kirill A. Shutemov0 siblings, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-06-18 14:09 UTC (permalink / raw)
To: Peter Zijlstra, Kai Huang
Cc: Andy Lutomirski, Kirill A. Shutemov, Andrew Morton, X86 ML,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Borislav Petkov,
David Howells, Kees Cook, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML, Tom Lendacky
On 6/18/19 2:12 AM, Peter Zijlstra wrote:
> On Tue, Jun 18, 2019 at 02:23:31PM +1200, Kai Huang wrote:
>> Assuming I am understanding the context correctly, yes from this perspective it seems having
>> sys_encrypt is annoying, and having ENCRYPT_ME should be better. But Dave said "nobody is going to
>> do what you suggest in the ptr1/ptr2 example"?
>
> You have to phrase that as: 'nobody who knows what he's doing is going
> to do that', which leaves lots of people and fuzzers.
>
> Murphy states that if it is possible, someone _will_ do it. And this
> being something that causes severe data corruption on persistent
> storage,...
I actually think it's not a big deal at all to avoid the corruption that
would occur if it were allowed. But, if you're even asking to map the
same data with two different keys, you're *asking* for data corruption.
What we're doing here is continuing to preserve cache coherency and
ensuring an early failure.
We'd need two rules:
1. A page must not be faulted into a VMA if the page's page_keyid()
is not consistent with the VMA's
2. Upon changing the VMA's KeyID, all underlying PTEs must either be
checked or zapped.
If the rules are broken, we SIGBUS. Andy's suggestion has the same
basic requirements. But, with his scheme, the error can be to the
ioctl() instead of in the form of a SIGBUS. I guess that makes the
fuzzers' lives a bit easier.
BTW, note that we don't have any problems with the current anonymous
implementation and fork() because we zap at the encryption syscall.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 14:09 ` Dave Hansen@ 2019-06-18 16:15 ` Kirill A. Shutemov
2019-06-18 16:22 ` Dave Hansen0 siblings, 1 reply; 153+ messages in thread
From: Kirill A. Shutemov @ 2019-06-18 16:15 UTC (permalink / raw)
To: Dave Hansen
Cc: Peter Zijlstra, Kai Huang, Andy Lutomirski, Kirill A. Shutemov,
Andrew Morton, X86 ML, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Borislav Petkov, David Howells, Kees Cook,
Jacob Pan, Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On Tue, Jun 18, 2019 at 07:09:36AM -0700, Dave Hansen wrote:
> On 6/18/19 2:12 AM, Peter Zijlstra wrote:
> > On Tue, Jun 18, 2019 at 02:23:31PM +1200, Kai Huang wrote:
> >> Assuming I am understanding the context correctly, yes from this perspective it seems having
> >> sys_encrypt is annoying, and having ENCRYPT_ME should be better. But Dave said "nobody is going to
> >> do what you suggest in the ptr1/ptr2 example"?
> >
> > You have to phrase that as: 'nobody who knows what he's doing is going
> > to do that', which leaves lots of people and fuzzers.
> >
> > Murphy states that if it is possible, someone _will_ do it. And this
> > being something that causes severe data corruption on persistent
> > storage,...
>
> I actually think it's not a big deal at all to avoid the corruption that
> would occur if it were allowed. But, if you're even asking to map the
> same data with two different keys, you're *asking* for data corruption.
> What we're doing here is continuing to preserve cache coherency and
> ensuring an early failure.
>
> We'd need two rules:
> 1. A page must not be faulted into a VMA if the page's page_keyid()
> is not consistent with the VMA's
> 2. Upon changing the VMA's KeyID, all underlying PTEs must either be
> checked or zapped.
>
> If the rules are broken, we SIGBUS. Andy's suggestion has the same
> basic requirements. But, with his scheme, the error can be to the
> ioctl() instead of in the form of a SIGBUS. I guess that makes the
> fuzzers' lives a bit easier.
I see a problem with the scheme: if we don't have a way to decide if the
key is right for the file, user without access to the right key is able to
prevent legitimate user from accessing the file. Attacker just need read
access to the encrypted file to prevent any legitimate use to access it.
The problem applies to ioctl() too.
To make sense of it we must have a way to distinguish right key from
wrong. I don't see obvious solution with the current hardware design.
--
Kirill A. Shutemov
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 16:15 ` Kirill A. Shutemov@ 2019-06-18 16:22 ` Dave Hansen
2019-06-18 16:36 ` Andy Lutomirski0 siblings, 1 reply; 153+ messages in thread
From: Dave Hansen @ 2019-06-18 16:22 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Peter Zijlstra, Kai Huang, Andy Lutomirski, Kirill A. Shutemov,
Andrew Morton, X86 ML, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Borislav Petkov, David Howells, Kees Cook,
Jacob Pan, Alison Schofield, Linux-MM, kvm list, keyrings, LKML,
Tom Lendacky
On 6/18/19 9:15 AM, Kirill A. Shutemov wrote:
>> We'd need two rules:
>> 1. A page must not be faulted into a VMA if the page's page_keyid()
>> is not consistent with the VMA's
>> 2. Upon changing the VMA's KeyID, all underlying PTEs must either be
>> checked or zapped.
>>
>> If the rules are broken, we SIGBUS. Andy's suggestion has the same
>> basic requirements. But, with his scheme, the error can be to the
>> ioctl() instead of in the form of a SIGBUS. I guess that makes the
>> fuzzers' lives a bit easier.
> I see a problem with the scheme: if we don't have a way to decide if the
> key is right for the file, user without access to the right key is able to
> prevent legitimate user from accessing the file. Attacker just need read
> access to the encrypted file to prevent any legitimate use to access it.
I think you're bringing up a separate issue.
We were talking about how you resolve a conflict when someone attempts
to use two *different* keyids to decrypt the data in the API and what
the resulting API interaction looks like.
You're describing the situation where one of those is the wrong *key*
(not keyid). That's a subtly different scenario and requires different
handling (or no handling IMNHO).
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 16:22 ` Dave Hansen@ 2019-06-18 16:36 ` Andy Lutomirski
2019-06-18 16:48 ` Dave Hansen0 siblings, 1 reply; 153+ messages in thread
From: Andy Lutomirski @ 2019-06-18 16:36 UTC (permalink / raw)
To: Dave Hansen
Cc: Kirill A. Shutemov, Peter Zijlstra, Kai Huang, Andy Lutomirski,
Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, David Howells,
Kees Cook, Jacob Pan, Alison Schofield, Linux-MM, kvm list,
keyrings, LKML, Tom Lendacky
> On Jun 18, 2019, at 9:22 AM, Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 6/18/19 9:15 AM, Kirill A. Shutemov wrote:
>>> We'd need two rules:
>>> 1. A page must not be faulted into a VMA if the page's page_keyid()
>>> is not consistent with the VMA's
>>> 2. Upon changing the VMA's KeyID, all underlying PTEs must either be
>>> checked or zapped.
>>>
>>> If the rules are broken, we SIGBUS. Andy's suggestion has the same
>>> basic requirements. But, with his scheme, the error can be to the
>>> ioctl() instead of in the form of a SIGBUS. I guess that makes the
>>> fuzzers' lives a bit easier.
>> I see a problem with the scheme: if we don't have a way to decide if the
>> key is right for the file, user without access to the right key is able to
>> prevent legitimate user from accessing the file. Attacker just need read
>> access to the encrypted file to prevent any legitimate use to access it.
>
> I think you're bringing up a separate issue.
>
> We were talking about how you resolve a conflict when someone attempts
> to use two *different* keyids to decrypt the data in the API and what
> the resulting API interaction looks like.
>
> You're describing the situation where one of those is the wrong *key*
> (not keyid). That's a subtly different scenario and requires different
> handling (or no handling IMNHO).
I think we’re quibbling over details before we look at the big questions:
Should MKTME+DAX encrypt the entire volume or should it encrypt individual files? Or both?
If it encrypts individual files, should the fs be involved at all? Should there be metadata that can check whether a given key is the correct key?
If it encrypts individual files, is it even conceptually possible to avoid corruption if the fs is not involved? After all, many filesystems think that they can move data blocks, compute checksums, journal data, etc.
I think Dave is right that there should at least be a somewhat credible proposal for how this could fit together.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 16:36 ` Andy Lutomirski@ 2019-06-18 16:48 ` Dave Hansen0 siblings, 0 replies; 153+ messages in thread
From: Dave Hansen @ 2019-06-18 16:48 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Kirill A. Shutemov, Peter Zijlstra, Kai Huang, Andy Lutomirski,
Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, David Howells,
Kees Cook, Jacob Pan, Alison Schofield, Linux-MM, kvm list,
keyrings, LKML, Tom Lendacky
On 6/18/19 9:36 AM, Andy Lutomirski wrote:
> Should MKTME+DAX encrypt the entire volume or should it encrypt individual files? Or both?
Our current thought is that there should be two modes: One for entire
DAX namespaces and one for filesystem DAX that would allow it to be at
the file level. More likely, we would mirror fscrypt and do it at the
directory level.
> If it encrypts individual files, should the fs be involved at all?
> Should there be metadata that can check whether a given key is the
> correct key?
FWIW, this is a question for the fs guys. Their guidance so far has
been "do what fscrypt does", and fscrypt does not protect against
incorrect keys being specified. See:
https://www.kernel.org/doc/html/v5.1/filesystems/fscrypt.html
Which says:
> Currently, fscrypt does not prevent a user from maliciously providing
> an incorrect key for another user’s existing encrypted files. A
> protection against this is planned.
> If it encrypts individual files, is it even conceptually possible to
> avoid corruption if the fs is not involved? After all, many
> filesystems think that they can move data blocks, compute checksums,
> journal data, etc.
Yes, exactly. Thankfully, fscrypt has thought about this already and
has infrastructure for this. For instance:
> Online defragmentation of encrypted files is not supported. The
> EXT4_IOC_MOVE_EXT and F2FS_IOC_MOVE_RANGE ioctls will fail with
> EOPNOTSUPP.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-18 0:15 ` Andy Lutomirski
2019-06-18 1:35 ` Kai Huang@ 2019-06-18 14:13 ` Dave Hansen1 sibling, 0 replies; 153+ messages in thread
From: Dave Hansen @ 2019-06-18 14:13 UTC (permalink / raw)
To: Andy Lutomirski, Kai Huang
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML, Tom Lendacky
On 6/17/19 5:15 PM, Andy Lutomirski wrote:
>>> But I really expect that the encryption of a DAX device will actually
>>> be a block device setting and won't look like this at all. It'll be
>>> more like dm-crypt except without device mapper.
>> Are you suggesting not to support MKTME for DAX, or adding MKTME support to dm-crypt?
> I'm proposing exposing it by an interface that looks somewhat like
> dm-crypt. Either we could have a way to create a device layered on
> top of the DAX devices that exposes a decrypted view or we add a way
> to tell the DAX device to kindly use MKTME with such-and-such key.
I think this basically implies that we need to settle (or at least
present) on an interface for storage (FS-DAX, Device DAX, page cache)
before we merge one for anonymous memory.
That sounds like a reasonable exercise.
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 18:27 ` Dave Hansen
2019-06-17 19:12 ` Andy Lutomirski@ 2019-06-17 23:59 ` Kai Huang
2019-06-18 1:34 ` Lendacky, Thomas1 sibling, 1 reply; 153+ messages in thread
From: Kai Huang @ 2019-06-17 23:59 UTC (permalink / raw)
To: Dave Hansen, Andy Lutomirski
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML, Tom Lendacky
On Mon, 2019-06-17 at 11:27 -0700, Dave Hansen wrote:
> Tom Lendacky, could you take a look down in the message to the talk of
> SEV? I want to make sure I'm not misrepresenting what it does today.
> ...
>
>
> > > I actually don't care all that much which one we end up with. It's not
> > > like the extra syscall in the second options means much.
> >
> > The benefit of the second one is that, if sys_encrypt is absent, it
> > just works. In the first model, programs need a fallback because
> > they'll segfault of mprotect_encrypt() gets ENOSYS.
>
> Well, by the time they get here, they would have already had to allocate
> and set up the encryption key. I don't think this would really be the
> "normal" malloc() path, for instance.
>
> > > How do we
> > > eventually stack it on top of persistent memory filesystems or Device
> > > DAX?
> >
> > How do we stack anonymous memory on top of persistent memory or Device
> > DAX? I'm confused.
>
> If our interface to MKTME is:
>
> fd = open("/dev/mktme");
> ptr = mmap(fd);
>
> Then it's hard to combine with an interface which is:
>
> fd = open("/dev/dax123");
> ptr = mmap(fd);
>
> Where if we have something like mprotect() (or madvise() or something
> else taking pointer), we can just do:
>
> fd = open("/dev/anything987");
> ptr = mmap(fd);
> sys_encrypt(ptr);
>
> Now, we might not *do* it that way for dax, for instance, but I'm just
> saying that if we go the /dev/mktme route, we never get a choice.
>
> > I think that, in the long run, we're going to have to either expand
> > the core mm's concept of what "memory" is or just have a whole
> > parallel set of mechanisms for memory that doesn't work like memory.
>
> ...
> > I expect that some day normal memory will be able to be repurposed as
> > SGX pages on the fly, and that will also look a lot more like SEV or
> > XPFO than like the this model of MKTME.
>
> I think you're drawing the line at pages where the kernel can manage
> contents vs. not manage contents. I'm not sure that's the right
> distinction to make, though. The thing that is important is whether the
> kernel can manage the lifetime and location of the data in the page.
>
> Basically: Can the kernel choose where the page comes from and get the
> page back when it wants?
>
> I really don't like the current state of things like with SEV or with
> KVM direct device assignment where the physical location is quite locked
> down and the kernel really can't manage the memory. I'm trying really
> hard to make sure future hardware is more permissive about such things.
> My hope is that these are a temporary blip and not the new normal.
>
> > So, if we upstream MKTME as anonymous memory with a magic config
> > syscall, I predict that, in a few years, it will be end up inheriting
> > all downsides of both approaches with few of the upsides. Programs
> > like QEMU will need to learn to manipulate pages that can't be
> > accessed outside the VM without special VM buy-in, so the fact that
> > MKTME pages are fully functional and can be GUP-ed won't be very
> > useful. And the VM will learn about all these things, but MKTME won't
> > really fit in.
>
> Kai Huang (who is on cc) has been doing the QEMU enabling and might want
> to weigh in. I'd also love to hear from the AMD folks in case I'm not
> grokking some aspect of SEV.
>
> But, my understanding is that, even today, neither QEMU nor the kernel
> can see SEV-encrypted guest memory. So QEMU should already understand
> how to not interact with guest memory. I _assume_ it's also already
> doing this with anonymous memory, without needing /dev/sme or something.
Correct neither Qemu nor kernel can see SEV-encrypted guest memory. Qemu requires guest's
cooperation when it needs to interacts with guest, i.e. to support virtual DMA (of virtual devices
in SEV-guest), qemu requires SEV-guest to setup bounce buffer (which will not be SEV-encrypted
memory, but shared memory can be accessed from host side too), so that guest kernel can copy DMA
data from bounce buffer to its own SEV-encrypted memory after qemu/host kernel puts DMA data to
bounce buffer.
And yes from my reading (better to have AMD guys to confirm) SEV guest uses anonymous memory, but it
also pins all guest memory (by calling GUP from KVM -- SEV specifically introduced 2 KVM ioctls for
this purpose), since SEV architecturally cannot support swapping, migraiton of SEV-encrypted guest
memory, because SME/SEV also uses physical address as "tweak", and there's no way that kernel can
get or use SEV-guest's memory encryption key. In order to swap/migrate SEV-guest memory, we need SGX
EPC eviction/reload similar thing, which SEV doesn't have today.
From this perspective, I think driver proposal kinda makes sense since we already have security
feature which uses normal memory some kind like "device memory" (no swap, no migration, etc), so it
makes sense that MKTME just follows that (although from HW MKTME can support swap, page migration,
etc). The downside of driver proposal for MKTME I think is, like Dave mentioned, it's hard (or not
sure whether it is possible) to extend to support NVDIMM (and file backed guest memory), since for
virtual NVDIMM, Qemu needs to call mmap against fd of NVDIMM.
Thanks,
-Kai
^permalinkrawreply [flat|nested] 153+ messages in thread

*Re: [PATCH, RFC 45/62] mm: Add the encrypt_mprotect() system call for MKTME
2019-06-17 23:59 ` Kai Huang@ 2019-06-18 1:34 ` Lendacky, Thomas
2019-06-18 1:40 ` Andy Lutomirski0 siblings, 1 reply; 153+ messages in thread
From: Lendacky, Thomas @ 2019-06-18 1:34 UTC (permalink / raw)
To: Kai Huang, Dave Hansen, Andy Lutomirski
Cc: Kirill A. Shutemov, Andrew Morton, X86 ML, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Borislav Petkov, Peter Zijlstra,
David Howells, Kees Cook, Jacob Pan, Alison Schofield, Linux-MM,
kvm list, keyrings, LKML
On 6/17/19 6:59 PM, Kai Huang wrote:
> On Mon, 2019-06-17 at 11:27 -0700, Dave Hansen wrote:
>> Tom Lendacky, could you take a look down in the message to the talk of
>> SEV? I want to make sure I'm not misrepresenting what it does today.
Sorry, I'm traveling this week, so responses will be delayed...
>> ...
>>
>>
>>>> I actually don't care all that much which one we end up with. It's not
>>>> like the extra syscall in the second options means much.
>>>
>>> The benefit of the second one is that, if sys_encrypt is absent, it
>>> just works. In the first model, programs need a fallback because
>>> they'll segfault of mprotect_encrypt() gets ENOSYS.
>>
>> Well, by the time they get here, they would have already had to allocate
>> and set up the encryption key. I don't think this would really be the
>> "normal" malloc() path, for instance.
>>
>>>> How do we
>>>> eventually stack it on top of persistent memory filesystems or Device
>>>> DAX?
>>>
>>> How do we stack anonymous memory on top of persistent memory or Device
>>> DAX? I'm confused.
>>
>> If our interface to MKTME is:
>>
>> fd = open("/dev/mktme");
>> ptr = mmap(fd);
>>
>> Then it's hard to combine with an interface which is:
>>
>> fd = open("/dev/dax123");
>> ptr = mmap(fd);
>>
>> Where if we have something like mprotect() (or madvise() or something
>> else taking pointer), we can just do:
>>
>> fd = open("/dev/anything987");
>> ptr = mmap(fd);
>> sys_encrypt(ptr);
>>
>> Now, we might not *do* it that way for dax, for instance, but I'm just
>> saying that if we go the /dev/mktme route, we never get a choice.
>>
>>> I think that, in the long run, we're going to have to either expand
>>> the core mm's concept of what "memory" is or just have a whole
>>> parallel set of mechanisms for memory that doesn't work like memory.
>>
>> ...
>>> I expect that some day normal memory will be able to be repurposed as
>>> SGX pages on the fly, and that will also look a lot more like SEV or
>>> XPFO than like the this model of MKTME.
>>
>> I think you're drawing the line at pages where the kernel can manage
>> contents vs. not manage contents. I'm not sure that's the right
>> distinction to make, though. The thing that is important is whether the
>> kernel can manage the lifetime and location of the data in the page.
>>
>> Basically: Can the kernel choose where the page comes from and get the
>> page back when it wants?
>>
>> I really don't like the current state of things like with SEV or with
>> KVM direct device assignment where the physical location is quite locked
>> down and the kernel really can't manage the memory. I'm trying really
>> hard to make sure future hardware is more permissive about such things.
>> My hope is that these are a temporary blip and not the new normal.
>>
>>> So, if we upstream MKTME as anonymous memory with a magic config
>>> syscall, I predict that, in a few years, it will be end up inheriting
>>> all downsides of both approaches with few of the upsides. Programs
>>> like QEMU will need to learn to manipulate pages that can't be
>>> accessed outside the VM without special VM buy-in, so the fact that
>>> MKTME pages are fully functional and can be GUP-ed won't be very
>>> useful. And the VM will learn about all these things, but MKTME won't
>>> really fit in.
>>
>> Kai Huang (who is on cc) has been doing the QEMU enabling and might want
>> to weigh in. I'd also love to hear from the AMD folks in case I'm not
>> grokking some aspect of SEV.
>>
>> But, my understanding is that, even today, neither QEMU nor the kernel
>> can see SEV-encrypted guest memory. So QEMU should already understand
>> how to not interact with guest memory. I _assume_ it's also already
>> doing this with anonymous memory, without needing /dev/sme or something.
>
> Correct neither Qemu nor kernel can see SEV-encrypted guest memory. Qemu requires guest's
> cooperation when it needs to interacts with guest, i.e. to support virtual DMA (of virtual devices
> in SEV-guest), qemu requires SEV-guest to setup bounce buffer (which will not be SEV-encrypted
> memory, but shared memory can be accessed from host side too), so that guest kernel can copy DMA
> data from bounce buffer to its own SEV-encrypted memory after qemu/host kernel puts DMA data to
> bounce buffer.
That is correct. an SEV guest must use un-encrypted memory if it wishes
for the hypervisor to be able to see it. Also, to support DMA into the
guest, the target memory must be un-encrypted, which SEV does through the
DMA api and SWIOTLB.
SME must also use bounce buffers if the device does not support 48-bit
DMA, since the encryption bit is bit 47. Any device supporting DMA above
the encryption bit position can perform DMA without bounce buffers under
SME.
>
> And yes from my reading (better to have AMD guys to confirm) SEV guest uses anonymous memory, but it
> also pins all guest memory (by calling GUP from KVM -- SEV specifically introduced 2 KVM ioctls for
> this purpose), since SEV architecturally cannot support swapping, migraiton of SEV-encrypted guest
> memory, because SME/SEV also uses physical address as "tweak", and there's no way that kernel can
> get or use SEV-guest's memory encryption key. In order to swap/migrate SEV-guest memory, we need SGX
> EPC eviction/reload similar thing, which SEV doesn't have today.
Yes, all the guest memory is currently pinned by calling GUP when creating
an SEV guest. This is to prevent page migration by the kernel. However,
the support to do page migration is available in the 0.17 version of the
SEV API via the COPY commmand. The COPY commannd allows for copying of
an encrypted page from one location to another via the PSP. The support
for this is not yet in the Linux kernel, but we are working on it. This
would remove the requirement of having to pin the guest memory.
The SEV API also supports migration of memory via the SEND_* and RECEIVE_*
APIs to support live migration.
Swapping, however, is not yet supported.
Thanks,
Tom
>
> From this perspective, I think driver proposal kinda makes sense since we already have security
> feature which uses normal memory some kind like "device memory" (no swap, no migration, etc), so it
> makes sense that MKTME just follows that (although from HW MKTME can support swap, page migration,
> etc). The downside of driver proposal for MKTME I think is, like Dave mentioned, it's hard (or not
> sure whether it is possible) to extend to support NVDIMM (and file backed guest memory), since for
> virtual NVDIMM, Qemu needs to call mmap against fd of NVDIMM.
>
> Thanks,
> -Kai
>
^permalinkrawreply [flat|nested] 153+ messages in thread

*[PATCH, RFC 58/62] x86/mktme: Document the MKTME provided security mitigations
2019-05-08 14:43 [PATCH, RFC 00/62] Intel MKTME enabling Kirill A. Shutemov
` (56 preceding siblings ...)
2019-05-08 14:44 ` [PATCH, RFC 57/62] x86/mktme: Overview of Multi-Key Total Memory Encryption Kirill A. Shutemov
@ 2019-05-08 14:44 ` Kirill A. Shutemov
2019-05-08 14:44 ` [PATCH, RFC 59/62] x86/mktme: Document the MKTME kernel configuration requirements Kirill A. Shutemov
` (5 subsequent siblings)63 siblings, 0 replies; 153+ messages in thread
From: Kirill A. Shutemov @ 2019-05-08 14:44 UTC (permalink / raw)
To: Andrew Morton, x86, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Borislav Petkov, Peter Zijlstra, Andy Lutomirski, David Howells
Cc: Kees Cook, Dave Hansen, Kai Huang, Jacob Pan, Alison Schofield,
linux-mm, kvm, keyrings, linux-kernel, Kirill A . Shutemov
From: Alison Schofield <alison.schofield@intel.com>
Describe the security benefits of Multi-Key Total Memory
Encryption (MKTME) over Total Memory Encryption (TME) alone.
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
Documentation/x86/mktme/index.rst | 1 +
Documentation/x86/mktme/mktme_mitigations.rst | 150 ++++++++++++++++++
2 files changed, 151 insertions(+)
create mode 100644 Documentation/x86/mktme/mktme_mitigations.rst
diff --git a/Documentation/x86/mktme/index.rst b/Documentation/x86/mktme/index.rst
index 1614b52dd3e9..a3a29577b013 100644
--- a/Documentation/x86/mktme/index.rst
+++ b/Documentation/x86/mktme/index.rst
@@ -6,3 +6,4 @@ Multi-Key Total Memory Encryption (MKTME)
.. toctree::
mktme_overview
+ mktme_mitigations
diff --git a/Documentation/x86/mktme/mktme_mitigations.rst b/Documentation/x86/mktme/mktme_mitigations.rst
new file mode 100644
index 000000000000..90699c38750a
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_mitigations.rst
@@ -0,0 +1,150 @@+MKTME-Provided Mitigations
+==========================
+
+MKTME adds a few mitigations against attacks that are not
+mitigated when using TME alone. The first set are mitigations
+against software attacks that are familiar today:
+
+ * Kernel Mapping Attacks: information disclosures that leverage
+ the kernel direct map are mitigated against disclosing user
+ data.
+ * Freed Data Leak Attacks: removing an encryption key from the
+ hardware mitigates future user information disclosure.
+
+The next set are attacks that depend on specialized hardware,
+such as an “evil DIMM” or a DDR interposer:
+
+ * Cross-Domain Replay Attack: data is captured from one domain
+(guest) and replayed to another at a later time.
+ * Cross-Domain Capture and Delayed Compare Attack: data is
+ captured and later analyzed to discover secrets.
+ * Key Wear-out Attack: data is captured and analyzed in order
+ to Weaken the AES encryption itself.
+
+More details on these attacks are below.
+
+Kernel Mapping Attacks
+----------------------
+Information disclosure vulnerabilities leverage the kernel direct
+map because many vulnerabilities involve manipulation of kernel
+data structures (examples: CVE-2017-7277, CVE-2017-9605). We
+normally think of these bugs as leaking valuable *kernel* data,
+but they can leak application data when application pages are
+recycled for kernel use.
+
+With this MKTME implementation, there is a direct map created for
+each MKTME KeyID which is used whenever the kernel needs to
+access plaintext. But, all kernel data structures are accessed
+via the direct map for KeyID-0. Thus, memory reads which are not
+coordinated with the KeyID get garbage (for example, accessing
+KeyID-4 data with the KeyID-0 mapping).
+
+This means that if sensitive data encrypted using MKTME is leaked
+via the KeyID-0 direct map, ciphertext decrypted with the wrong
+key will be disclosed. To disclose plaintext, an attacker must
+“pivot” to the correct direct mapping, which is non-trivial
+because there are no kernel data structures in the KeyID!=0
+direct mapping.
+
+Freed Data Leak Attack
+----------------------
+The kernel has a history of bugs around uninitialized data.
+Usually, we think of these bugs as leaking sensitive kernel data,
+but they can also be used to leak application secrets.
+
+MKTME can help mitigate the case where application secrets are
+leaked:
+
+ * App (or VM) places a secret in a page * App exits or frees
+memory to kernel allocator * Page added to allocator free list *
+Attacker reallocates page to a purpose where it can read the page
+
+Now, imagine MKTME was in use on the memory being leaked. The
+data can only be leaked as long as the key is programmed in the
+hardware. If the key is de-programmed, like after all pages are
+freed after a guest is shut down, any future reads will just see
+ciphertext.
+
+Basically, the key is a convenient choke-point: you can be more
+confident that data encrypted with it is inaccessible once the
+key is removed.
+
+Cross-Domain Replay Attack
+--------------------------
+MKTME mitigates cross-domain replay attacks where an attacker
+replaces an encrypted block owned by one domain with a block
+owned by another domain. MKTME does not prevent this replacement
+from occurring, but it does mitigate plaintext from being
+disclosed if the domains use different keys.
+
+With TME, the attack could be executed by:
+ * A victim places secret in memory, at a given physical address.
+ Note: AES-XTS is what restricts the attack to being performed
+ at a single physical address instead of across different
+ physical addresses
+ * Attacker captures victim secret’s ciphertext * Later on, after
+ victim frees the physical address, attacker gains ownership
+ * Attacker puts the ciphertext at the address and get the secret
+ plaintext
+
+But, due to the presumably different keys used by the attacker
+and the victim, the attacker can not successfully decrypt old
+ciphertext.
+
+Cross-Domain Capture and Delayed Compare Attack
+-----------------------------------------------
+This is also referred to as a kind of dictionary attack.
+
+Similarly, MKTME protects against cross-domain capture-and-compare
+attacks. Consider the following scenario:
+ * A victim places a secret in memory, at a known physical address
+ * Attacker captures victim’s ciphertext
+ * Attacker gains control of the target physical address, perhaps
+ after the victim’s VM is shut down or its memory reclaimed.
+ * Attacker computes and writes many possible plaintexts until new
+ ciphertext matches content captured previously.
+
+Secrets which have low (plaintext) entropy are more vulnerable to
+this attack because they reduce the number of possible plaintexts
+an attacker has to compute and write.
+
+The attack will not work if attacker and victim uses different
+keys.
+
+Key Wear-out Attack
+-------------------
+Repeated use of an encryption key might be used by an attacker to
+infer information about the key or the plaintext, weakening the
+encryption. The higher the bandwidth of the encryption engine,
+the more vulnerable the key is to wear-out. The MKTME memory
+encryption hardware works at the speed of the memory bus, which
+has high bandwidth.
+
+Such a weakness has been demonstrated[1] on a theoretical cipher
+with similar properties as AES-XTS.
+
+An attack would take the following steps:
+ * Victim system is using TME with AES-XTS-128
+ * Attacker repeatedly captures ciphertext/plaintext pairs (can
+ be Performed with online hardware attack like an interposer).
+ * Attacker compels repeated use of the key under attack for a
+ sustained time period without a system reboot[2].
+ * Attacker discovers a cipertext collision (two plaintexts
+ translating to the same ciphertext)
+ * Attacker can induce controlled modifications to the targeted
+ plaintext by modifying the colliding ciphertext
+
+MKTME mitigates key wear-out in two ways:
+ * Keys can be rotated periodically to mitigate wear-out. Since
+ TME keys are generated at boot, rotation of TME keys requires a
+ reboot. In contrast, MKTME allows rotation while the system is
+ booted. An application could implement a policy to rotate keys
+ at a frequency which is not feasible to attack.
+ * In the case that MKTME is used to encrypt two guests’ memory
+ with two different keys, an attack on one guest’s key would not
+ weaken the key used in the second guest.
+
+--
+1. http://web.cs.ucdavis.edu/~rogaway/papers/offsets.pdf
+2. This sustained time required for an attack could vary from days
+ to years depending on the attacker’s goals.
--
2.20.1
^permalinkrawreply [flat|nested] 153+ messages in thread