LFCS: Preparing Linux for nonvolatile memory devices [LWN.net]

LFCS: Preparing Linux for nonvolatile memory devices

Posted Apr 22, 2013 11:25 UTC (Mon) by etienne (guest, #25256)
Parent article: LFCS: Preparing Linux for nonvolatile memory devices

Now that the address space of a PC is 64 bits (well 40 or 48 physical), the long term goal should be having ready to use stuff in NVM:
- The Linux kernel should be loaded there, with its "persistent" data
- Most libraries should be loaded and pre-linked there (lazy linking), with their "persistent" data
- Maybe also have some/all servers and possibly some applications.

The problem is to define "persistent" data, i.e. data which is necessary but will be re-used; a first approximation would be data statically allocated (i.e. the data segment but nothing from malloc()) - but then where do we put the stack: "top" part in NVM and "bottom" part in standard DDR, or stack in NVM with COW (Copy to DDR on write) pages?

The other problem is upgrading either the kernel or some libraries, how to unload a library (and its dependencies) when upgrading, how to get where a library has been loaded (at previous boot) both its physical and virtual address - and which version/SHA1 was it?

Maybe we should use NVM with a parallel and explicit NVMmalloc()/NVMfree() (because there is no way to magically do the right thing), instead of using a filesystem?

to post comments

LFCS: Preparing Linux for nonvolatile memory devices

Posted Apr 25, 2013 19:23 UTC (Thu) by rahvin (guest, #16953) [Link] (5 responses)

Why are you assuming that the NVM will separate from the main memory subsystem like NAND is currently? Industry is already demoing several new kinds of main system memory that's non-volitile. I believe the expectation is that you won't even have DDR memory, that your entire main memory will be NVM.

LFCS: Preparing Linux for nonvolatile memory devices

Posted Apr 26, 2013 12:14 UTC (Fri) by etienne (guest, #25256) [Link] (4 responses)

> Why are you assuming that the NVM will separate from the main memory subsystem like NAND is currently?

Even if this NVM is on the same DDR socket, they seem to say that the number of writes is limited, and the time it takes to write NVM is longer.
The processor can really write a lot of times per second to the DDR, flushing the same cache line times and times again - we do not want that penalty nor do we want some wear levelling at that point.
Note that NVM obviously do not need refresh cycles, I wonder what effect it has on performance.

Moreover, I am not sure I always want to suspend Linux instead of powering off - sometimes I want a clean slate and come back to the login screen so that applications which have been "loosing" memory for the last 10 days restart from fresh - or after an upgrade of a library, be sure that no more application still use the old version (removing a library file from the filesystem do not automatically restart users of the old library version, which still have memory mapped the old and deleted file).
For the later point, maybe a "full boot" each time is the best solution.

LFCS: Preparing Linux for nonvolatile memory devices

Posted Apr 28, 2013 18:28 UTC (Sun) by Jonno (guest, #49613) [Link]

> Why are you assuming that the NVM will separate from the main memory subsystem like NAND is currently?

> Even if this NVM is on the same DDR socket, they seem to say that the number of writes is limited, and the time it takes to write NVM is longer.

There is several different types of NVM (Non-Volatile Memory), and while some have a limited number of write-cycles, others don't. Performance also vary, and several types are faster than DRAM (currently used as main memory), though to my knowledge none are quite as fast as SRAM (currently used as CPU cache).

That said, there is going to be a while before you can get anything with both good performance and unlimited write-cycles for anything resembling the price of regular DRAM, so while I expect some type of NVM will eventually be used in the main memory system, we are going to need some other system in order to use the current, imperfect, NVM types in the meantime...

LFCS: Preparing Linux for nonvolatile memory devices

Posted Oct 11, 2016 8:08 UTC (Tue) by ecloud (guest, #56624) [Link] (2 responses)

I think the ultimate point about getting userspace onboard is that we need next-generation languages that make memory leaks impossible, that maintain data structures compactly in memory (avoid linked lists and the like), and trade in the "filesystem" APIs for the appropriate object-storage APIs. (But yes, some databases are already appropriate places to start with this.) Instead of having APIs that make filesystem access completely different from memory manipulation, we need a way of marking data structures persistent. The language should then translate that into marking pages of memory persistent, and the OS should ensure that persistent pages are stored on the appropriate device. Applications should take care not to write to persistent structures more often than necessary; but otherwise either the language implementation or the OS should provide a way to cache frequently-updated persistent structures in volatile memory, and do checkpointing of changes. (Maybe marking the structure both volatile and persistent would mean that.) I guess the next issue is that sync-written structures could be temporarily out of sync with those which are cached; then either it means all writes need to be to cached first and then flushed to NVM at the next checkpoint, or else the system needs to be power-failure-proof (not a problem for battery-powered devices; line-powered machines can have at least a capacitor-based UPS sufficient that all writes can be completed before power fails).

So, rebooting, or even restarting applications, should become exceedingly rare. It places great demands on all software to be as reliable as the kernel itself: keep running for years with no leaks, no overflows, no bugs of the kind that require restarting the software as a workaround. You couldn't truly restart the application without losing all its stored data too. Using filesystems has made it harder to write software (so much persistence-related code that has to be written), but also allowed us to be too lazy for too long about reliability of the in-memory operations. If we invest as much effort into keeping memory beautifully organized as we have invested into file-based persistence, maybe we can get there?

I doubt that Linux will be the leader here, but there must be some current university research project by now? Anybody know of one? A long time ago there was KeyKOS which had checkpointing-based persistence; then there was Eros, but its focus shifted more strongly to capability-based security than on checkpointing. (And Linux still doesn't have such advanced capability-based security, either. This is why Sandstorm exists: the OS doesn't do it, so you have to rely on containers and management of them to isolate processes from each other.)

So now we have NVMe devices, like the M.2 flash drives. Can they be configured as memory-mapped, without using mmap()? Because using mmap() implies that all reads and writes will be cached in volatile RAM, right? If the hardware allows us to have RAM for one range of addresses and flash for another range, this work could begin.

LFCS: Preparing Linux for nonvolatile memory devices

Posted Oct 13, 2016 12:46 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Fundamentally it seems to me that we'll still want something like a filesystem: a collection of named, possibly hierarchically or otherwise structured blocks of data that can be used by multiple programs without regard for which program created them. Just arranging for programs to keep their data structures around forever doesn't do that: each program would have to implement some sort of organizational principle and if they're not all using the same one this is tantamount to every program having its own implementation of half a filesystem, non-interopable with any of the others or with external tooling. This seems drastically worse than what we have now.

Persistent memory is nice because it might mean that e.g. you could shut down a machine with a long-running computation on it and have it just restart again. Of course, with the CPU caches not persistent, it might have to go back a few seconds to a checkpoint. You can often do that *now* by just checkpointing to disk every few seconds, but with persistent storage you can presumably do that even if there are gigabytes of state (assuming that the persistent memory of discourse doesn't wear out on writes the way flash does).

But persistent memory will not allow us to do away with filesystems: neither the API nor the allocation layer. The fundamental role of the filesystem API -- naming things and letting users, and disparate programs, access them -- will still be needed, and cannot be replaced by object storage any more than you can replace a filesystem with an inode table and tell users to just access everything by inode number. Equally, the role of filesystems themselves -- the object allocation layer -- is still there: It's just a filesystem for persistent storage, with differently strange tradeoffs than every other filesystem's differently strange tradeoffs. Even having files with no name is not new: unlinked files have given us that for decades, and more recently open(..., O_TMPFILE) has too.

LFCS: Preparing Linux for nonvolatile memory devices

Posted Oct 13, 2016 14:32 UTC (Thu) by raven667 (guest, #5198) [Link]

> every program having its own implementation of half a filesystem, non-interopable with any of the others

Probably not every program, but every major language family that doesn't share low level compatibility of its data structures, like how today having a C API is a lowest common denominator for a languages compatibility with other languages. Or how JSON has become a medium of exchange for network software.

> naming things and letting users, and disparate programs, access them

With the popularity of application sandboxing, with Flatpak on the desktop and Docker on the server, there are far more defined and regimented ways for applications to share data, so I don't expect arbitrary disparate programs accessing data to be supported in this model.