| |||||||||||||
stalls caused by USB drive copystalls caused by USB drive copyPosted Jan 13, 2013 0:14 UTC (Sun) by man_ls (subscriber, #15091)In reply to: stalls caused by USB drive copy by nix Parent article: Per-entity load tracking Ah, the old "a process paged all my memory" problem. Looks like bad design in cp then. But to trigger it nowadays you would have to copy a file bigger than free memory (which can be a few GB) to a USB key; I am not sure that this is what jospoortvliet and hummassa were reporting above. The solution for the problem you describe should be, as usual, in several places: stop cp from dirtying more pages than it can move to their destination, and make the kernel avoid this behavior in any process. I really don't know how in the general case. To avoid this kind of problem I don't use swap any more, so that a misbehaving process cannot page everything else to disk. I still get severe slowdowns when memory runs out though; I am still trying to figure out how to avoid them.
(Log in to post comments)
stalls caused by USB drive copy Posted Jan 13, 2013 0:22 UTC (Sun) by nybble41 (subscriber, #55106) [Link]
As I'm hardly an expert, take this with a grain of salt, but I put this line in /etc/sysctl.conf (or /etc/sysctl.d/*.conf):
vm.dirty_bytes = 201326592
This limits total dirty memory to 192MB before the kernel forces processes to work on writing data to disk rather than dirtying new pages. The default is something like 10% of total RAM. It seems to help.
stalls caused by USB drive copy Posted Jan 13, 2013 1:12 UTC (Sun) by cmccabe (guest, #60281) [Link]
I don't think the issue here is cp's fault. It's the kernel's job to manage the page cache and writeback.
stalls caused by USB drive copy Posted Jan 13, 2013 1:59 UTC (Sun) by hummassa (subscriber, #307) [Link]
if, as nybble41 stated above, vm.dirty_bytes really works, the blame is in the kernel that has a bad default (10% of all RAM) because RAM is cheaper nowadays, but not proportionally faster to access. It causes contention of the cleaning the dirty pages somehow. "cp" just happens to trigger this bug. For information, I can trigger this bug by copying anything (it does not seem to be necessary to be a single file, but a single file happens to trigger it more aggressively apparently) above 1/2 GiB via USB.
Trying to backup my VMs (20G each file) to a USB drive is a nightmare. Usually I do "nice -20 cp" or, more commonly, "kde-cp".
stalls caused by USB drive copy Posted Apr 11, 2013 20:28 UTC (Thu) by serzan (subscriber, #8155) [Link]
> Usually I do "nice -20 cp" or, more commonly, "kde-cp".
have you tried ionice -c 3? though not sure it'd have much of an impact, if thrashing memory is the cause
|
|||||||||||||