Per-entity load tracking [LWN.net]

Per-entity load tracking

Posted Jan 12, 2013 14:53 UTC (Sat) by hummassa (subscriber, #307) [Link]

I tend to think it's a problem on "cp", since the same operation, when done via GUI, does not cause (for me) the same stalls. In my head, "cp file /media/usbdrive" tries to copy the files as fast as possible and trips on some strange load condition (ubuntu kernel, most recent, here) while "kde-cp", besides showing a fine progress bar and forecast-time-to-complete, takes approximately the same time to make the copy, without thrashing my whole system.

stalls caused by USB drive copy

Posted Jan 12, 2013 19:29 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

You can't blame 'cp'. It's the kernel's job to divide up the resources and 'cp' should selfishly concentrate on getting its own work done. If there is code in the alternative copying software which is specifically designed to leave CPU time slots or whatever for other processes it thinks are more important, I'm against that.

stalls caused by USB drive copy

Posted Jan 12, 2013 22:01 UTC (Sat) by man_ls (subscriber, #15091) [Link]

I have never seen this. Have you tried with CONFIG_PREEMPT? The kernel does a much better job of not letting random tasks hoard the CPU (or even CPUs these days), interactivity does not suffer even under arbitrary IO load.

stalls caused by USB drive copy

Posted Jan 12, 2013 23:24 UTC (Sat) by nix (subscriber, #2304) [Link]

It's got nothing to do with the CPU. This is a longstanding problem whereby with a sufficiently large copy, cp fills memory up with dirty pages awaiting writeout and then everything freezes solid because until the writeout to the slow block device has finished, there's hardly any memory left to do anything.

(It used to be much worse: it used to stall basically all I/O to *any* block device due to, IIRC from vague faded memory, everything sharing a single queue. That's not true anymore. So this has got better, though I can't tell whether I don't see it these days because the problem has improved or because machines just have huge amounts of memory these days...)

stalls caused by USB drive copy

Posted Jan 13, 2013 0:14 UTC (Sun) by man_ls (subscriber, #15091) [Link]

Ah, the old "a process paged all my memory" problem. Looks like bad design in cp then. But to trigger it nowadays you would have to copy a file bigger than free memory (which can be a few GB) to a USB key; I am not sure that this is what jospoortvliet and hummassa were reporting above.

The solution for the problem you describe should be, as usual, in several places: stop cp from dirtying more pages than it can move to their destination, and make the kernel avoid this behavior in any process. I really don't know how in the general case. To avoid this kind of problem I don't use swap any more, so that a misbehaving process cannot page everything else to disk. I still get severe slowdowns when memory runs out though; I am still trying to figure out how to avoid them.

stalls caused by USB drive copy

Posted Jan 13, 2013 0:22 UTC (Sun) by nybble41 (subscriber, #55106) [Link]

As I'm hardly an expert, take this with a grain of salt, but I put this line in /etc/sysctl.conf (or /etc/sysctl.d/*.conf):

vm.dirty_bytes = 201326592

This limits total dirty memory to 192MB before the kernel forces processes to work on writing data to disk rather than dirtying new pages. The default is something like 10% of total RAM. It seems to help.

stalls caused by USB drive copy

Posted Jan 13, 2013 1:12 UTC (Sun) by cmccabe (guest, #60281) [Link]

I don't think the issue here is cp's fault. It's the kernel's job to manage the page cache and writeback.

stalls caused by USB drive copy

Posted Jan 13, 2013 1:59 UTC (Sun) by hummassa (subscriber, #307) [Link]

if, as nybble41 stated above, vm.dirty_bytes really works, the blame is in the kernel that has a bad default (10% of all RAM) because RAM is cheaper nowadays, but not proportionally faster to access. It causes contention of the cleaning the dirty pages somehow. "cp" just happens to trigger this bug. For information, I can trigger this bug by copying anything (it does not seem to be necessary to be a single file, but a single file happens to trigger it more aggressively apparently) above 1/2 GiB via USB.
Trying to backup my VMs (20G each file) to a USB drive is a nightmare. Usually I do "nice -20 cp" or, more commonly, "kde-cp".

stalls caused by USB drive copy

Posted Apr 11, 2013 20:28 UTC (Thu) by serzan (subscriber, #8155) [Link]

> Usually I do "nice -20 cp" or, more commonly, "kde-cp".

have you tried ionice -c 3? though not sure it'd have much of an impact, if thrashing memory is the cause

Per-entity load tracking

Posted Jan 13, 2013 9:32 UTC (Sun) by BlueLightning (subscriber, #38978) [Link]

even then, copying data to a slow USB drive can lead to minute-long stalls of the entire system. Very embarrassing that this is still such an issue on linux. I am not sure if it is a hardware problem or a software one...

Have you seen this issue recently? I'm sure I read an LWN article a year or two ago where it was described how this issue was tracked down to a change in the I/O buffer window sizing or something similar (and fixed).

Per-entity load tracking

Posted Jan 13, 2013 12:56 UTC (Sun) by hummassa (subscriber, #307) [Link]

I had this problem fairly recently (two months ago), i suppose my kernel was already Quantal's (today it's 3.5.0-21).