| |||||||||||||
stalls caused by USB drive copystalls caused by USB drive copyPosted Jan 12, 2013 23:24 UTC (Sat) by nix (subscriber, #2304)In reply to: stalls caused by USB drive copy by man_ls Parent article: Per-entity load tracking
It's got nothing to do with the CPU. This is a longstanding problem whereby with a sufficiently large copy, cp fills memory up with dirty pages awaiting writeout and then everything freezes solid because until the writeout to the slow block device has finished, there's hardly any memory left to do anything.
(It used to be much worse: it used to stall basically all I/O to *any* block device due to, IIRC from vague faded memory, everything sharing a single queue. That's not true anymore. So this has got better, though I can't tell whether I don't see it these days because the problem has improved or because machines just have huge amounts of memory these days...)
(Log in to post comments)
stalls caused by USB drive copy Posted Jan 13, 2013 0:14 UTC (Sun) by man_ls (subscriber, #15091) [Link] Ah, the old "a process paged all my memory" problem. Looks like bad design in cp then. But to trigger it nowadays you would have to copy a file bigger than free memory (which can be a few GB) to a USB key; I am not sure that this is what jospoortvliet and hummassa were reporting above.The solution for the problem you describe should be, as usual, in several places: stop cp from dirtying more pages than it can move to their destination, and make the kernel avoid this behavior in any process. I really don't know how in the general case. To avoid this kind of problem I don't use swap any more, so that a misbehaving process cannot page everything else to disk. I still get severe slowdowns when memory runs out though; I am still trying to figure out how to avoid them.
stalls caused by USB drive copy Posted Jan 13, 2013 0:22 UTC (Sun) by nybble41 (subscriber, #55106) [Link]
As I'm hardly an expert, take this with a grain of salt, but I put this line in /etc/sysctl.conf (or /etc/sysctl.d/*.conf):
vm.dirty_bytes = 201326592
This limits total dirty memory to 192MB before the kernel forces processes to work on writing data to disk rather than dirtying new pages. The default is something like 10% of total RAM. It seems to help.
stalls caused by USB drive copy Posted Jan 13, 2013 1:12 UTC (Sun) by cmccabe (guest, #60281) [Link]
I don't think the issue here is cp's fault. It's the kernel's job to manage the page cache and writeback.
stalls caused by USB drive copy Posted Jan 13, 2013 1:59 UTC (Sun) by hummassa (subscriber, #307) [Link]
if, as nybble41 stated above, vm.dirty_bytes really works, the blame is in the kernel that has a bad default (10% of all RAM) because RAM is cheaper nowadays, but not proportionally faster to access. It causes contention of the cleaning the dirty pages somehow. "cp" just happens to trigger this bug. For information, I can trigger this bug by copying anything (it does not seem to be necessary to be a single file, but a single file happens to trigger it more aggressively apparently) above 1/2 GiB via USB.
Trying to backup my VMs (20G each file) to a USB drive is a nightmare. Usually I do "nice -20 cp" or, more commonly, "kde-cp".
stalls caused by USB drive copy Posted Apr 11, 2013 20:28 UTC (Thu) by serzan (subscriber, #8155) [Link]
> Usually I do "nice -20 cp" or, more commonly, "kde-cp".
have you tried ionice -c 3? though not sure it'd have much of an impact, if thrashing memory is the cause
|
|||||||||||||