LWN.net Logo

Sanitizing kernel memory

By Jake Edge
May 27, 2009

The contents of memory consist of vast quantities of useless—to an attacker at least—data, along with a small amount that would be of interest. Cryptographic keys, passwords, and the like are probable targets of those with malicious intent. Normally, the kernel guards memory from access by unprivileged processes, but, various kernel bugs have sometimes allowed memory information to leak. A recently proposed patch would eliminate a specific subset of those kinds of leaks by "sanitizing" pages as they are freed.

Larry Highsmith adapted code from the PaX project to add a flag to kernel pages marking them as "sensitive" pages. The pages would then be cleared as they were freed, so that any information leak from those pages would be useless. As part of the justification for the change, Highsmith noted Stanford University paper entitled "Shredding Your Garbage: Reducing Data Lifetime Through Secure Deallocation" as well as the "cold boot" attacks to recover memory from powered-down systems.

Highsmith's patch would eliminate cases where freed memory contents leak, either via a kernel bug or some other means, by clearing the page as it is freed, but only for memory marked as sensitive. The four additional patches in his original series then applied the sensitive flag to various kernel subsystems (crypto, audit, and key handling).

While the kernel hackers were generally agreeable to the idea of sanitizing memory, there were a number of objections to Highsmith's first attempt. A trivial one, which was fixed in later patches, was a Signed-off-by line that didn't give his full name (just "Larry H."). As the PaX project is developed by the pseudonymous "PaX Team"—thus not able to fulfill the requirements for a kernel sign off—several folks were quick to point out that a full name was required. More substantive objections were heard about using up a scarce resource in the form of a page flag. Alan Cox pointed out that a virtual memory area (VMA) flag would work as well, or that places in the patch that set the flag could just clear the memory instead:

[...] page flags are very precious, very few and if we run out will cost us a vast amount of extra kernel memory. If page flags were free the question would be trivial - but they are not. Thus it is worth asking whether its actually harder to remember to zap the buffer or set and clear the flag correctly.

There was a bit of a digression into the security issues surrounding suspend and hibernate, with Highsmith claiming that security conscious users just disabled that functionality altogether. Cox and Pavel Machek disagreed, noting the ability to encrypt the images that get written to disk with today's hibernate code. Cox was also concerned that marking things as sensitive makes an attacker's job easier:

If you've got a rogue module you already lost, except that by marking what is sensitive you made the bad guys job easier. Bit like the way people visually overlay maps and overhead shots from multiple sources and the 'scrubbed' secret locations stand out for you and are easier to find than if they were left.

In the end, any memory the kernel handles is potentially sensitive. Some applications—notably GPG—take great pains to try to ensure that their memory is not swapped and is cleared of keys and other sensitive data when they are no longer needed. As Ingo Molnar put it: "The whole kernel contains data that 'should not be leaked'." This led to a new approach: for users who want sanitized pages—based on the sanitize_mem boot time parameter—simply clear all pages when they are freed. A much smaller patch that implemented that scheme was then posted by Highsmith.

In addition, there are kernel allocations that are for objects smaller than a page which could contain sensitive data. Highsmith has also submitted changes to kfree() and kmem_cache_free() that would clear these objects as they are freed. In the end, with both of these patches applied in a kernel with sanitize_mem enabled, all free kernel memory will be cleared. But, of course, as several folks pointed out, in many cases the memory of interest will still be in use.

Certainly a kernel with sanitized memory is more resistant to leaking memory contents, but depending on the threat one is trying to defend against, it may not be enough. The physical attacks against memory contents (i.e. "cold boot") are still likely to be effective—though free memory won't be recoverable—and other kinds of bugs could still leak memory in use. Highsmith presented an analysis of kernel information leaks, which was partially based on this interesting list of CVEs and git commits that fixed them. In it, there were a half-dozen examples of information leaks that would have been prevented by his changes.

No further objections have been noted, and the patches are not terribly intrusive, so it would seem there is some chance they might make their way into 2.6.31.


(Log in to post comments)

Sanitizing kernel memory

Posted May 28, 2009 8:01 UTC (Thu) by gmaxwell (subscriber, #30048) [Link]

Why not make this a default behaviour?

When userspace sbrks to increase memory the kernel must hand it cleared memory or data leaks will result. From a performance perspective clearing at free is probably superior to clearing at the time the memory is allocated.

I expect that there would be a performance hit from the clearing of sub-page kernel allocations. But it may be negligible and if it isn't then just that part of this patch could be made an optional parameter.

With userspace memory protected and kernel cryptographic memory protected that would eliminate a significant part of the attack surface... am I missing a reason why this couldn't perform well?

Sanitizing kernel memory

Posted May 28, 2009 13:27 UTC (Thu) by lemmings (subscriber, #53618) [Link]

> Why not make this a default behaviour?

Memory that is free will upon allocation either be over-written (e.g. DMA from HDD/network) or zeroed (e.g. stack frame) prior to use.

Making this the default behaviour will add a performance cost in the over-written case. Part of the cost could be hidden though by having
a low priority memory clearing thread which used the appropriate instructions to avoid filling the data cache.

The extent of the cost can only be determined with proper benchmarks though. Then whether the cost is acceptable is another question...

Disclaimer: I'm not a kernel hacker so I could be full of it.

Sanitizing kernel memory

Posted May 28, 2009 18:25 UTC (Thu) by anton (guest, #25547) [Link]

Clearing on freeing will also add a performance cost in the zeroed case: Even if you avoid the cache and CPU cost of clearing in some way, there are still the memory bandwidth costs of writing the zeros out, and later loading them again; plus the CPU latency cost of having to load the zeros into the cache on use. In contrast, if you zero the page on-demand, the zeros are written to the cache, and in many cases are then available to the user program from the cache, requiring less CPU time and less memory bandwidth.

Sanitizing kernel memory

Posted May 28, 2009 18:38 UTC (Thu) by gmaxwell (subscriber, #30048) [Link]

My thought process was the that the writing at free is cheap (especially as you can avoid pulling in anything in from memory or evicting anything from the cache) and that any userspace app that reads memory handed to it by the kernel before writing it is broken. Don't these people have valgrind?

:)

Sanitizing kernel memory

Posted May 28, 2009 19:09 UTC (Thu) by anton (guest, #25547) [Link]

any userspace app that reads memory handed to it by the kernel before writing it is broken.
If the application writes a byte to a line that's not in cache, current CPUs load (the rest of) the cache line from main memory (write allocate). In a few special cases this read can be avoided, but for ordinary writes it happens.

Sanitizing kernel memory

Posted May 30, 2009 10:10 UTC (Sat) by willezurmacht (guest, #58372) [Link]

"any userspace app that reads memory handed to it by the kernel before writing it is broken. Don't these people have valgrind?"

Or maybe it's an exploit for an information/memory leak bug in the kernel that retrieves your root password and tty buffers contents from memory.

Sanitizing kernel memory

Posted May 28, 2009 22:06 UTC (Thu) by nix (subscriber, #2304) [Link]

Would a DMA engine help here? A lot of modern chipsets have them
(sometimes they even work).

Sanitizing kernel memory

Posted May 29, 2009 8:50 UTC (Fri) by anton (guest, #25547) [Link]

A DMA engine would "avoid the cache and CPU cost of clearing" (that's what I was thinking about when I wrote that), but the other costs would still be there.

Sanitizing kernel memory

Posted Jun 2, 2009 15:29 UTC (Tue) by etienne_lorrain@yahoo.fr (guest, #38022) [Link]

On the other side, if a function allocates more than it will use (allocating the maximum size is quite usual, and *alloc() may round up the size), then the zeroing at allocation will pollute all those very performance enhancing 32+ bytes cache 1 lines...
Moreover, DMA zeroing at free() time should set those cache lines in a state where they will be first reused (instead of more important lines).
Sorry, not statistics available.

Sanitizing kernel memory

Posted Jun 2, 2009 16:24 UTC (Tue) by anton (guest, #25547) [Link]

Yes, if the zeroed cache lines are not accessed or not accessed before being replaced in the cache, then the performance of on-demand zeroing for that cache line will be just as bad as eager zeroing in the cache, and a little worse than for eager methods that don't go through the cache. But I doubt that that's the case for the majority of cache lines. In particular, I don't think that there are replaced at all cache levels before being accessed. But yes, measurements would be a good idea.

Concerning eager zeroing of cache lines, that is certain to replace a page full of cache lines just as on-demand zeroing does, except that it is far less likely that the cache lines will be accessed before being replaced by other cache lines, so it is a bad idea. Tagging the line as least-recently-used helps only a little, if it is possible at all.

Sanitizing kernel memory

Posted Jun 4, 2009 18:30 UTC (Thu) by oak (guest, #2786) [Link]

> On the other side, if a function allocates more than it will use

I think all the user-space allocated pages point to the same zeroed/shared
physical page until they're written to?

Sanitizing kernel memory

Posted Jun 9, 2009 12:28 UTC (Tue) by etienne_lorrain@yahoo.fr (guest, #38022) [Link]

> user-space allocated pages
We were talking about kernel-space allocated memory...
Having a stock of cleared pages for user-space would probably also be an improvement (when user-space writes its first byte on a page), as long as clearing those pages does not wipe the memory cache (i.e. DMA to memory instead of processor writing).

Sanitizing kernel memory

Posted Jun 9, 2009 11:08 UTC (Tue) by robbe (subscriber, #16131) [Link]

Apart from the security benefits this probably helps save memory by
sharing it with other VMs inside a hypervisor that does page
consolidation (e.g. ksm in Linux).

A zero page is more shareable than garbage.

Copyright © 2009, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds