LWN: Comments on "The return of network block device deadlock prevention"

Cluster nodes? Thin clients? blades?

ringerc — Thu, 24 Aug 2006 07:29:00 +0000

...there have not been streams of bug reports on this topic - and, in most cases, it can be avoided simply by swapping to a local disk.

This fails to consider diskless thin clients, blade systems, and cluster nodes where power use, heat, and noise as well as cost are major factors in the desire to avoid a local disk.

I suspect the bug reports haven't been coming because "everybody knows linux can't swap reliably over nfs or nbd" - or is told so as soon as they do any research on the matter. That rather reduces the test base, and reports will be further reduced by the expectation that problem reports will be dismissed with a comment along the lines of "why are you doing that, it's unreliable and can deadlock." .

Whether the statement about reliability in the previous paragraph is actually true, and whether the issues really arise very much, is a good question. I don't think there's any easy way to tell the difference between a feature everybody is told is unreliable and so avoids, and one that's used by a significant number of people without many issues despite theoretical problems.

I personally use LTSP thin clients that swap over NFS. The latest LTSP has moved to swap-over-nbd, so I'll be finding out first hand soon. Experience with LTSP suggests that swap over NFS, at least, is not something you want to rely on for more than the most casual use, as it DOES deadlock.

The set of systems whose owners can afford fancy network storage arrays, but where those same owners are unable to invest in a local disk for swapping, is thought to be small.

This might be true - but there are several types of systems that have this need, as noted above, and the reasons for avoiding local disks certainly aren't limited to cost. I'd argue that the number of people with >16 CPU machines is pretty small too, but the scheduler certainly gets a disproportionate amount of attention and added complexity for this small group.

Greetings from userland. Wish you were here.

pengo — Sun, 20 Aug 2006 02:49:12 +0000

How about some easy-to-use tools for allocating swap space and/or warning that memory is low? Or better yet, how about a module to dynamically create a temporary local swap file (rather than a partition) in the user's home directory when memory runs out? No, it's not the same problem as what this patch would address, and this patch isn't without its reasons, but there are more broad general-purpose solutions for memory allocation woes. So I'd like to give a gripe...

When I removed one of my hard drives (the drive was making grinding noises and about to die) I failed to notice it happened to be the drive with my only swap partition on it. However, a gig of ram is plenty to run Ubuntu and it ran just fine.. for the time being.

When I did run out of memory though, my system grinded to a halt very horribly--which is to be expected when memory fills up I guess (though I'm not sure why the OOM Killer didn't just kill the app (a badly written python script) that was actually hogging and allocating RAM like it was going out of fashion). However, the problem for me was that there was still no (user space) warning that I was out of memory, and no (kernel) attempt to allocate some sort of temporary swap file, and no warning to the user of what was going on or why (although it was obviously something swap related to me, it wouldn't have been obvious if I was remotely logged in and couldn't hear the machine trying to tip itself over). From a user perspective, a pretty horrible experience.

So instead of being Kernel-space-fighter-pilots, it would be wonderful if some kernel dev'ers would stick their heads out of the kernel-box and look at the bigger picture for a change and consider the actual scenarios where memory runs out and work out some real world SOLUTIONS from there, rather than constructing wonderfully intricate mechanisms out of matchsticks, like this one, which, while it certainly has merit, has a much narrower coverage than any solution which started by looking at actual problem scenarios, and didn't limit itself to kernel space.

I would like to see some better user space integration for dealing with this kind of thing, instead of pretending that world doesn't exist.

Pengo.

The return of network block device deadlock prevention

giraffedata — Fri, 18 Aug 2006 16:47:26 +0000

It's worth mentioning that network block devices aren't the only thing with this problem. Network filesystems have it too. (Specifically, any filesystem driver that has to talk to someone over a network in order to write out a dirty page of file data cache).

The article mentions how unimportant the problem is because of the unimportance of swapping to a network block device, but in contrast, buffering writes to a network filesystem is pretty important.

In fact, deadlocks are easy to come by with network filesystems, except that there are arbitrary limits placed on the amount of memory that can be used for file data cache -- in effect, a very large reserve is made for network memory requirements. If the resource inversion could actually be fixed, we could free filesystems to be more self-tuning and make more efficient use of available memory.

The return of network block device deadlock prevention

nicku — Fri, 18 Aug 2006 00:10:00 +0000

…without exhausting the reserves with useless data (streaming video from LinuxWorld keynotes, say).

I laughed uncontrollably, and then went to see who is delivering the keynotes. For the most part, the joke looks spot on.