Are there any applications, JIT frameworks or operating systems that focus on encrypted virtual memory, or perhaps virtual machines that do something similar? I know there are processors (albeit old, slow and weak) that allow for systems that are entirely encrypted, but I've yet to see any OS or application come up with a decent solution for x86 or one of its descendants.

I understand that L1/L2/L3 caches, DMA buffers, etc. cannot be encrypted without hardware support, but surely at least certain kernel structures and most of the user-mode memory (i.e. stuff allocated by processes) could be encrypted given support at compile-time?

I also would like to know if it would be theoretically possible (whilst obviously difficult) to translate an existing JIT compiler (e.g. .NET CLR) that produces code that automatically encrypts its user-mode memory.

Any sort of existing solutions along these lines would be a lot of fun to take a look at.

What are you referring to when you say "virtual memory"? As a Windows user, when I hear "virtual memory" I think of the swap or page file - a portion of the HDD that the OS treats as extra RAM. Certainly, this could be encrypted along with the rest of the HDD with whole-disk encryption solutions.
–
IsziOct 20 '11 at 19:26

3

I'm talking about virtual memory in the x86 architecture sense, i.e. memory that may be in physical or swap memory, mapped into virtual memory space.
–
PolynomialOct 20 '11 at 20:07

1

A reason why it's not done might be the very small benefit. Encrypted RAM reduces the trusted base to the CPU, but unless the CPU has been specifically designed with this in mind, it's probably leaking sensitive data through DPA anyway.
–
GillesOct 21 '11 at 8:33

By the way, there are processors that aren't “old, slow and weak” by any count with hardware-assisted encryption for off-chip RAM. Some modern ARM processors such as OMAP processors with M-Shield have a small amount (think L2 cache) of on-chip “secure RAM” (only accessible to the TrustZone secure world) with secure demand paging (SDP), i.e. encrypted paging in ordinary RAM.
–
GillesNov 18 '11 at 17:46

4 Answers
4

I also would like to know if it would be theoretically possible (whilst obviously difficult) to translate an existing JIT compiler (e.g. .NET CLR) that produces code that automatically encrypts its user-mode memory.

This might be quite difficult for the OS to do on behalf of a program. The reason for this is that the virtual pages allocated by the operating system are in fact controlled by the CPU - the OS instructs the CPU as to what it wants in terms of storage areas and the CPU then computes the offsets on that basis for each program and applies the result directly. Have a read up on the MMU and Paging on x86.

In short, when your program makes the computation lea eax, [edi+something], the MMU/paging handles the address translation and issues a page fault when the page is not found. That allows you to then load the page out of storage, should you need to.

So the memory access doesn't go through the kernel per se and as such can't be hooked and processed as a file read or write can (your read and writes are translated through the appropriate system calls into the appropriate file system structure. As such, the OS sees the data as it passes through. You don't need a system call to write to RAM. You could make one, but it would only work for programs that call it and so not most programs).

However, that doesn't mean to say a JIT process couldn't do this on behalf of an application, or the application itself couldn't keep data encrypted until it needs to load it into registers, decrypting it as it passes over the data.

In that case, you're into the issue SteveS covers well - you have a key storage problem. The key itself must also be in RAM somewhere. This is a turtles all the way down problem - fundamentally it's impossible to keep that key "secure".

Finally, since being able to read another application's RAM requires supervisor mode access to the CPU (i.e. kernel space) you probably have bigger issues if your concern is software interception. If your concern is hardware, I think physical security might be a better way of mitigating the risk.

Edit I had a look for papers. Here's one called CryptKeeper. Their technique is to have a large encrypted "RAM disk" as the swap file and swap all their pages out to that when not used:

We mitigate this vulnerability with Cryptkeeper (CK), a
software-encrypted virtual memory manager. Traditional pro-
cessors cannot operate on encrypted data, so CK segments
RAM into a smaller working set called the Clear, and a
larger encrypted RAM device called the Crypt. As the working
space fills, pages are automatically swapped into the encrypted
portion of memory, and are decrypted on demand.

Apparently the performance isn't too bad, but I don't think there are any implementations of it yet.

Edit 2 So it turns out CryptKeeper and the OpenBSD Swap Encryption mechanisms work on a very similar level; it does not actually encrypt physical memory but uses physical memory as a swap structure, forcing page faults and encrypting/decrypting data in resolution.

Good answer! With the CryptKeeper couldn't you stick a kernel process (driver) and intercept cleartext as it hits the processor or swapped back into the ram disk?
–
SteveOct 21 '11 at 16:08

@SteveS I don't think so - at least not on x86. CryptKeeper I believe basically allocates most of the physical memory to the "Crypt" i.e. a swapfile structure. A tiny bit is left for stuff you're working on, but most of the time you'd generate page faults, fetching the data from encrypted memory, when you allocate memory, shift something else out of the clear into the swap. The swap being in ram it should be pretty quick, although not as quick as just normal ram, because you've got all those page faults/swapping and the slight overhead from encryption taking up a few extra clock cycles.
–
user2213Oct 21 '11 at 16:21

The OpenBSD operating system includes automatic encryption of virtual memory; it is enabled by default since version 3.9.

Without a CPU with embedded encryption hardware, the data in caches and physical RAM cannot be really encrypted, because the CPU could then not use it; but "virtual memory" as in "RAM blocks copied to and from the disk" is under the total control of the operating system, and thus the OS can encrypt it at will, which is what OpenBSD does. To some extent, most of the RAM could be kept encrypted as well, by using it as if it was a disk (i.e. a swap file on a RAM-based virtual disk -- it would work !).

RAM contents are supposed to disappear automatically when power is cut (it actually takes about one minute or so, depending on temperature).

I was not really referencing the swap file, but rather the physical memory itself. See the comments on my question. I know about the temporary non-volatility of RAM after power-off, which is partly the reason I'm interested in the idea of encrypted physical memory.
–
PolynomialOct 21 '11 at 8:13

does it store the key unencrypted in RAM?
–
user24408May 30 '13 at 15:47

I'd be curious if something exists too, but I'm not entirely sure one can exist without hardware intervention.

My logic is that in order for the memory to be encrypted, the key has to be known by whatever process is accessing it directly, which for the sake of argument would just be at the kernel level. At some point that key needs to be moved into unencrypted memory so things can be encrypted or decrypted, since the processor can't manipulate encrypted data directly. Once this key is in unencrypted memory you can access it through a couple ways: memory dump, kernel driver, physical probing, etc.

In theory you could prevent memory dumps from being written to disk and could prevent drivers from being installed, and you could lock the device in a safe and drop it into the ocean (a waterproof safe preferably) making it physically inaccessible, but that changes the usage of the system dramatically.

Well, let me rephrase my first statement then -- one could exist without hardware intervention, but I'm not sure how secure it would be.

As for your other question, it would be possible, but you still run into the same basic problem. You would have to modify the compiler and modify the JIT interpreter, and the exe would store the key in its own memory space. The exe would have to hand the key off to the interpreter to handle the crypto. The problem with this is that you still have to store the key somewhere in unencrypted memory. In fact, if the key is hard coded in the exe, then you could just crack open the file and go looking for it.

That's what I originally thought. It'd be interesting to see something along the lines of a HSM that stored the keys securely, allowing for hardware level security at the memory bus level. That way you'd get the extra security with the benefit of being able to support DMA, as long as you can consider the bus and peripheral devices as safe.
–
PolynomialOct 21 '11 at 8:18

+1 "problem with this is that you still have to store the key somewhere in unencrypted memory" cryptkeeper's problem
–
user24408May 30 '13 at 15:47

@Polynomial And if you store the key in a external hardware? Like tokens flash drives or smartcards? And then you tell the application to read the key on the device and then decrypt ram ? The ideia here is that you can only use your RAM with physical access to an external hardware containing an key
–
FreedomFeb 1 at 22:26

The problem with the cryptkeeper approach is that it still leaves critical data in the clear (albeit much less than a normal system). I have been looking into this question for the past year and I don't believe there is anything doing what you're asking (something I'm working on remedying to some degree). I do believe that INTEL will be releasing a chip in the next several years that will automatically encrypt all of the contents of RAM.