LWN: Comments on "The volatile volatile ranges patch set"

Free the whole range rather than bits

epa — Wed, 16 Jul 2014 11:10:36 +0000

I was imagining that the image wouldn't be cached on disk. If it were, it would probably be more efficient just to store it on disk and use the kernel's excellent disk caching to keep the pages in memory if there is room - not try to reinvent the wheel by making your own 'disk cache' using nonvolatile memory. So the use case I was thinking of is where some JPEG image (or any other piece of data) is useful in its entirety, but not with chunks missing in the middle. In that case if one piece has to be discarded the rest of it might as well be thrown out at the same time. Surely there is a way to hint this to the kernel?

implementations, interfaces, mumble grumble

tomgj — Wed, 09 Jul 2014 13:22:00 +0000

Thanks for the useful article.

May I humbly suggest that the word “implementation” should be used more conservatively than this, especially as it relates to API design and interface specifiication.

The article says “Implementations of the volatile range concept have experienced more than the usual amount of change”. Though this is true, the way it is worded risks confusing issues around interface and implementation. This is because (i) implementations of the concept have involved different interface designs (as nicely expounded upon in the article), with (ii) implementations of those interfaces also having been through change. It would be nice, in a world where even advanced developers often don’t understand the value of well-specified interfaces, if we could, for clarity, reserve the word “implementation” for the latter kind of thing, marked (ii) above.

This confusion also arises in a couple of other places. We have “Previous volatile range implementations, though, had the property that marking a range nonvolatile could fail partway through. That meant that the interface had to be able to return two values”, which is using the term implementation to talk about the interface, and “This implementation will deliver a SIGBUS signal in that situation”, which is also better described in terms of being an interface thing.

Later in the article, we have “If the implementation and interface are not right…”, and here the term is clearly being used in the type (ii) sense as defined above. But, having set up use of the term “implementation” to mean either “implementation of a concept (possibly involving different interfaces)”, and “implementation of an interface”, the confusion sewn earlier is here reaped again.

All this comes back to a phenomenon way too common in Linux development: hack something together that “works” locally, and whatever happens to be exposed on the outside when it’s “working” then becomes the “interface”. This is the wrong way round, leads to low quality interface design, and wasted effort.

The article does describe thought going into the interface design. But how much better off would we be if interface design were more widely thought of as being the preceding process to implementation, rather than something to come later if at all? This doesn’t stop interface specification and implementation being co-iterative: the experience of actually attempting the implementation will often feed back proposed changes to the interface spec. But viewing interface specification as the fundamentally “earlier” process would have huge benefits. Most Linux-originated APIs are a wreck — here is part of the reason.

The volatile volatile ranges patch set

neilbrown — Thu, 26 Jun 2014 22:11:01 +0000

Life really does imitate fiction.

http://www.clivebanks.co.uk/THHGTTG/THHGTTGradio6.htm

CHAIRMAN:
Yes, and, and, and the wheel. What about this wheel thingy? Sounds a terribly interesting project to me.

MARKETING GIRL:
Er, yeah, well we’re having a little, er, difficulty here…

FORD:
Difficulty?! It’s the single simplest machine in the entire universe!

MARKETING GIRL:
Well alright mister wise guy, if you’re so clever you tell us what colour it should be!

[ and don't even ask about the nasally fitted fire ]

(sorry, this comment is totally unfair to John Stultz and all the others who have worked on this - it really isn't as trivial as that at all.
My personal opinion is that there are at least 2 and possibly 3 separate problems that people want to solve and no solution proposed so far solves all of them suitably. The problems need to be clearly separated and solved separately. I think that latest patchset was trying to head in this direction to some extent)

The volatile volatile ranges patch set

weue — Thu, 26 Jun 2014 21:09:00 +0000

How does this relatively simple feature take THREE $#@#$@ YEARS to not even be accepted yet?

Free the whole range rather than bits

alonz — Thu, 26 Jun 2014 10:52:27 +0000

Even for this use-case, I suspect repeating the entire decompression would still cost less (and have less bugs) than trying to reconstruct a single page from the middle of the image. And more modern formats (e.g. WebP) make the challenge even greater.

Free the whole range rather than bits

dlang — Thu, 26 Jun 2014 06:28:39 +0000

remember that this is less for caching data from the net and more for caching the uncompressed images for faster display.

so it's more likely to be 4K out of a uncompressed image that the app would just need to pull the appropriate range of data out of the .jpg that it has stored elsewhere.

Free the whole range rather than bits

alonz — Sat, 21 Jun 2014 13:52:03 +0000

I suspect the cost of refreshing just one page from the JPEG from the 'net will tend to be almost the same as refreshing the entire image… and refreshing two (disjoint) pages will almost certainly cost more than the entire image.

Of course, these are just my estimates; they may be wrong in the particular, but I've personally designed systems that did exhibit this behavior. An obvious example is when caching a compressed version of some resource — restarting compression from the middle is far harder than redoing it all from scratch.

Free the whole range rather than bits

niner — Sat, 21 Jun 2014 09:22:42 +0000

A a bit less simple web cache on the other hand should be able to handle a JPEG with a page missing in the middle. It either can load the page from the disk cache or request the missing range from the web server. No need to download the ranges it still has.

The volatile volatile ranges patch set

pbonzini — Fri, 20 Jun 2014 12:31:24 +0000

Kind of, because IIUC userfaultfd also applies to page faults in the kernel. In the case of KVM, this means that the asynchronous page fault machinery can kick in.

The volatile volatile ranges patch set

dgm — Fri, 20 Jun 2014 09:12:09 +0000

From a (this at least) user space developer perspective, madvise() sounds like the interface that fits best the proposed behavior.

But for the sake of clarity I would prefer it to be called MADV_FREEABLE (volatile is also a C keyword with very different semantics). In fact, I think the best flag names would be MADV_FREETRYREMEMBER and MADV_TRYRECALL.

The volatile volatile ranges patch set

josh — Thu, 19 Jun 2014 16:06:33 +0000

Wouldn't userfaultfd just work like a modified signalfd, usable for otherwise deadly signals like SIGBUS because it blocks the signaled thread until the signal gets read?

Free the whole range rather than bits

epa — Thu, 19 Jun 2014 13:44:17 +0000

For many use cases, if one page of the range is freed then the rest of it becomes useless. A simple web cache would not be able to serve a JPEG with four kilobytes missing in the middle. Is there a way to signal to the kernel 'if you have to free one page from this range, you might as well free the rest of it at the same time'? The alternative policy would be 'please keep as many pages as possible intact even if some are freed'.

The volatile volatile ranges patch set

pbonzini — Thu, 19 Jun 2014 08:06:38 +0000

Andrea Arcangeli has been working on a "userfaultfd" system call that would let you stop the faulting thread while another thread populates the page. He's using that together with anonymous memory, but perhaps this could be useful for volatile memory ranges too.

The volatile volatile ranges patch set

Cyberax — Thu, 19 Jun 2014 06:48:06 +0000

Why not both? Add another flag, something like MADV_FAIL_ON_MISSING to send a SIGBUS if you want this behavior.

Personally, I'd prefer a SIGBUS because it would allow to:
1) Distinguish between legitimate zero-filled pages and pages that have just been evicted.
2) Allow to find code that accesses the volatile regions without proper checking.

The volatile volatile ranges patch set

jstultz — Thu, 19 Jun 2014 05:22:35 +0000

Oops. miss-linked to the discussions there...

Again, my objections listed here:
http://article.gmane.org/gmane.linux.kernel.mm/116959

Minchan's agreement here:
http://article.gmane.org/gmane.linux.kernel.mm/116960

The volatile volatile ranges patch set

jstultz — Thu, 19 Jun 2014 05:19:37 +0000

Just as a minor clarification (and I can understand how this was confusing)...

Minchan's objection mentioned here was somewhat tangential and short-lived. Its stems from a use-case that the Google address-sanitizer folks wanted, where volatility would be sticky and data could be purged, then rewritten and then purged again without any explicit re-marking of volatility.

I *really* don't see how this use case is at all feasible. Especially as a generic implementation. My specific objections where listed here: http://thread.gmane.org/gmane.linux.kernel.mm/116952

And after that Minchan agreed and withdrew his objection:
http://thread.gmane.org/gmane.linux.kernel.mm/116952

Johannes' suggestion for zero-fill behavior is viable, and works more in-line with the existing VMM code by overloading the page clean/dirty state as a marker of volatility. My main objection is that this works well from the VMM perspective, but creates more surprising semantics for userspace.

Those semantics could become less surprising (but still not my ideal) to userspace with his additional suggestion of adding a SIGBUS option to MADV_FREE, so this is a potential route and I *really do* appreciate the feedback and suggestion (Johannes, Hugh and many other folks have been very kind and motivational in discussions at conferences). Its just that I've run a bit out of steam on this one and have other work I need to do.

I still think its a really great and needed feature, and it shames me to feel like I've failed in pushing it upstream. But its my hope someone else might be able to pick up the torch here.