LWN: Comments on "The Linux "copy problem""

The Linux "copy problem"

oliwer — Mon, 01 Feb 2021 13:18:12 +0000

Note that GNU cp(1) recently added support for copy_file_range(2).
https://git.savannah.gnu.org/cgit/coreutils.git/commit/sr...

The Linux "copy problem"

nix — Mon, 24 Jun 2019 14:28:47 +0000

Reflinking does, inevitably, reduce the fragmentation-resistance of the filesystem if you use it a lot and tend to keep the post-reflinked files for longer than the pre-reflinked ones. (That's the whole point, after all: the new blocks are necessarily going to be fragmented wrt the old ones, even if this is reduced by writing more than strictly needed in the CoW phase.)

Ideally, a background preen phase would rewrite such things after a while so they are non-fragmented again. I guess one could call xfs_fsr from cron to do that... on an SSD this matters not at all, of course, and not very much if there's a caching layer like LVM's or bcache in the way either. But it's a real concern on unintermediated spinning rust.

The Linux "copy problem"

jezuch — Thu, 20 Jun 2019 06:51:50 +0000

I think people were concerned with loss of redundancy. Some claimed that they make a copy because they want to *really* have an actual copy and making it COW by default is surprising to the user by basically changing the contract of the command.

That's how I remember it at least.

Anyway, when I copy something it's mostly because I screwed up and I am restoring things from a btrfs backup snapshot. Making a real copy would cause unnecessary duplication in the backup - and would be unnecessarily slow and drive-thrashing. But that's just my use case.

The copy problem is really the backup problem

nix — Wed, 19 Jun 2019 21:32:21 +0000

btrfs send and receive are not regular I/O, so they work fine. (Though I'm not sure what happens in conjunction with O_DIRECT, which is a bit... hard to grasp the semantics of on a CoW filesystem in any case.)

(You don't get -ETXTBSY if you read a file in any case, only if you try to modify it.)

The Linux "copy problem"

nevyn — Sun, 16 Jun 2019 17:40:19 +0000

Why is this not the default?

The Linux "copy problem"

smfrench — Mon, 10 Jun 2019 21:58:01 +0000

This is great news - looking forward to trying it. Am also very excited about the work Andreas at RedHat did, enabling GCM crypto for SMB3.1.1 mounts, which can more than double performance copying files to server when on encrypted mounts (in conjunction with two cifs.ko client patches that I recently merged into for-next that enable GCM on the client).

The copy problem is really the backup problem

mcr — Tue, 04 Jun 2019 14:06:36 +0000

What happens if a file is open (TXTBUSY)? or open O_DIRECT, or any of these other things that might be mutually exclusive with regular I/O?

The copy problem is really the backup problem

jezuch — Tue, 04 Jun 2019 09:18:20 +0000

At least on btrfs that's:

btrfs subvolume snapshot -r
btrfs send
btrfs receive

But it does not work on per-file basis, unfortunately. And yes, btrfs defines its own serialization format.

The Linux "copy problem"

desbma — Sun, 02 Jun 2019 12:45:37 +0000

You are right, I'm using ZSH and didn't realize that line was not portable across other shells.

curl -v --compressed 'https://lwn.net/' -o /dev/null 2>&1 | grep gzip

also works

The Linux "copy problem"

Jandar — Sun, 02 Jun 2019 12:20:52 +0000

> curl -v --compressed 'https://lwn.net/' > /dev/null 2>&1 | grep gzip

This command is obviously without any output.

$ curl -v --compressed 'https://lwn.net/' > /dev/null 2>&1 | wc
0 0 0

Perhaps you meant: curl -v --compressed 'https://lwn.net/' 2>&1 > /dev/null | grep gzip

The Linux "copy problem"

desbma — Sun, 02 Jun 2019 10:27:17 +0000

My browser (and curl) both disagree with you:
curl -v --compressed 'https://lwn.net/' > /dev/null 2>&1 | grep gzip
> Accept-Encoding: deflate, gzip
< Content-Encoding: gzip

The Linux "copy problem"

tarkasteve — Sun, 02 Jun 2019 05:12:03 +0000

So inspired by all this, I've updated xcp with the ability to do parallel copies (at the per-file level). The results are fairly good; I'm seeing 30%-60% speed-ups depending on caching.

The Linux "copy problem"

scientes — Sun, 02 Jun 2019 03:07:27 +0000

> (including when your browser decompresses this very page).

Compression is disabled for https sites due to various attacks on the file size information leak.

The Linux "copy problem"

scientes — Sun, 02 Jun 2019 03:02:57 +0000

I myself was using inotail until I reported the problem (tail -f didn't support inotify) to coreutils and it was actually fixed.

The Linux "copy problem"

desbma — Sat, 01 Jun 2019 21:13:17 +0000

This is most likely the infamous "dirty writeback" problem, best described in this LWN article: https://lwn.net/Articles/572911/

On my systems, I fix it permanently with:
echo "vm.dirty_background_bytes=$((48 * 1024 * 1024))
vm.dirty_ratio=10" | sudo tee /etc/sysctl.d/99-dirty-writeback.conf
sudo sysctl --system

That way the writeback kicks in with at most 48MB of dirty data, and also stalls the writer process before there is more than 10% of all memory consumed by dirty writeback pages.

The Linux "copy problem"

jmclnx — Sat, 01 Jun 2019 16:11:36 +0000

Speeding up things is always good, but the one thing I have noticed in the last few years. Once in a great while, if I cp a large file from one local fs to another local fs, the whole system would freeze for multiple 10s of seconds when a sync happens.

In the early days I never saw this and we use to make fun of Microsoft for having that freeze issue. I would rather have a slower cp and not have that sync freeze :)

The Linux "copy problem"

desbma — Sat, 01 Jun 2019 13:04:38 +0000

Thanks for the link.

It joins the list of great little tools that have taken inspiration from classic Unix command line tools, but rewritten them in Rust with many improvements along the way: grep -> ripgrep, find -> fd, hexdump -> hexyl, cat -> bat, du -> diskus, cloc -> tokei...

I'll be sure to look into xcp, and probably open a few issues along the way :)

The Linux "copy problem"

tarkasteve — Sat, 01 Jun 2019 01:40:21 +0000

I'd also humbly suggest `xcp`:

https://crates.io/crates/xcp

* Uses copy_file_range() where possible, falls back to userspace if not.
* Supports sparse files (with lseek; I wasn't aware of fiemap, is there any advantage to one over the other?)
* Partially parallel (recursive read is separate from copy operations; I have an todo for parallel copy as it seems to have advantages on nvme drives).
* Optional progress bar.
* Written in Rust
* Cross platform (well, Linux + other unix-like OSs; Windows may work, I've never managed to get Rust to work on it).

It doesn't support much in the way of permissions/ACLs ATM, it's still an intermittent WIP.

I did look at using O_DIRECT, but I get EINVAL. The open manpage lists a whole series of caveats and warnings about using it, including a disparaging quote from Linus.

Thanks for the discussion/article, it's given me some things to look into.

The Linux "copy problem"

rahulsundaram — Fri, 31 May 2019 15:46:00 +0000

Yes. Enabled by default in recent Fedora and RHEL 8 as well.

The Linux "copy problem"

jhoblitt — Fri, 31 May 2019 14:44:39 +0000

Doesn't XFS has reflink() support?

The Linux "copy problem"

LtWorf — Fri, 31 May 2019 09:33:48 +0000

Some of us don't use experimental filesystems.

The Linux "copy problem"

jmgao — Thu, 30 May 2019 18:56:29 +0000

> He does think that there should be a way to create files with all of their attributes atomically, however.

Isn't there already a solution to this? You can open a file on the filesystem you want to create the file on with O_TMPFILE to hide it until you've done all of your attribute twiddling, do your fsetfilecon, fsetxattr, etc. on the file descriptor, and then use linkat to put it into place.

The Linux "copy problem"

boutcher — Thu, 30 May 2019 16:20:05 +0000

I had to laugh that you brought up OS/2

The Linux "copy problem"

desbma — Thu, 30 May 2019 16:17:30 +0000

Unfortunately, there are many other example of cases like this.

For example zlib, one the most widely used software library in the world, has several forks (Intel, Cloudflare, zlib-ng...) with optimizations that improve compression/decompression speed.

Yet the changes have never been merged back in zlib, and everybody still uses the historic version, and happily wastes CPU cycles (including when your browser decompresses this very page).

The Linux "copy problem"

KAMiKAZOW — Thu, 30 May 2019 16:00:23 +0000

> It seems like it would be worth at least diving into `cp` and making sure it's as good as can be, since that gets used so much.

I find it insane that cp was fixed a decade ago by NASA and in all that time neither them nor anybody else thought about upstreaming the changes.

The copy problem is really the backup problem

mcr — Thu, 30 May 2019 15:01:45 +0000

In the old says of Unix, with a single file system, we used "dump" to get a good copy. It had all sorts ridiculous issues of having a userspace program trying to decipher file system contents from raw reads of the disk. On the other hand, when it worked, it got all the metadata, did it without destroying the buffer cache, and often was able to backup disks which were in the process of dying. When it failed, it failed, and the backups were sometimes garbage. And it didn't work for many things. So people mostly use tar for backup. And that's should be the most common copy problem, which is not just about data centers or cluster environments. And tar fails for any file system that does something innovative.
My claim is that our VFS layer is incomplete: it should include an atomic backup and an atomic restore operation, at least on a file level, but optionally on a directory basis. If we had that, then cp would always usefully be backup file | restore file2. This means that file systems have to serialize file contents and meta data, and have to deserialize it too. We Linux a microkernel architecture, then probably much of this deserialization could be done in some system-provided, non-ring0 context. Should we pick tar for serialization, or something more modern like CBOR, that's a bike shed for a design team.
I would just be happy if we could agree that we need this functionality.

The Linux "copy problem"

desbma — Thu, 30 May 2019 14:00:36 +0000

> I haven't actually tried this, but in theory it should enable the kernel to do a zero-copy copy where the underlying files support that.

This is exactly what sendfile does, with a single system call, instead of 3 for your example.

The Linux "copy problem"

ecree — Thu, 30 May 2019 13:54:00 +0000

> > one of the file descriptors must refer to a pipe
Yes, so you make two splice calls:

int p[2];
pipe(p);
splice(fd_in, NULL, p[1], NULL, len, flags);
splice(p[0], NULL, fd_out, NULL, len, flags);

I haven't actually tried this, but in theory it should enable the kernel to do a zero-copy copy where the underlying files support that. The pipe is really no more than a way to associate a userspace handle with a kernel buffer; see https://yarchive.net/comp/linux/splice.html for details.

The Linux "copy problem"

desbma — Thu, 30 May 2019 13:38:06 +0000

According to `man splice`:
> splice() moves data between two file descriptors [...] where one of the file descriptors must refer to a pipe

sendfile does not have this restriction (it can also work if the destination fd is a socket), so it is the ideal candidate for copying files.

Apparently, someone proposed to update coreutil's cp to use sendfile in 2012, but it was rejected: https://lists.gnu.org/archive/html/coreutils/2012-10/msg0...

A while ago I did some benchmarks in Python to compare "read/write chunk" vs "sendfile" based copy, and it let to a 30-50% speedup : https://github.com/desbma/pyfastcopy#performance

The Linux "copy problem"

ecree — Thu, 30 May 2019 13:22:33 +0000

I was wondering about splice(). Wouldn't that be a more efficient way to do copies?

The Linux "copy problem"

jezuch — Thu, 30 May 2019 10:35:46 +0000

I don't cp often but when I do I cp --reflink

The Linux "copy problem"

k3ninho — Thu, 30 May 2019 07:29:03 +0000

I had a range of responses to this 'build the interfaces' thing.

How people use those interfaces is something that needs guidance, always: you form a symbiotic loop in reliance on each other.

Conway's Law hasn't gone away and is implications about interacting with the users of your interfaces still stand. I think that Chris Mason's comment here is misguided and that we need to help people work together and communicate better. I think we need to balance this view of APIs with criticism of xattrs being difficult to copy and racy (with security implications).

K3n.

The Linux "copy problem"

roc — Wed, 29 May 2019 23:05:10 +0000

That makes sense, but you also want to the default to be as good as can be.

The Linux "copy problem"

ewen — Wed, 29 May 2019 21:38:32 +0000

The sendfile system call was my thought too. If a copy tool used sendfile, eg, for within a file system copying then the kernel has lots more information on the high level goal with which to optimise the approach taken. And even between file systems it could potentially make use of file system internal knowledge (block layout, etc) and buffer access.

If copy tools just do read/write in a loop, the kernel is left guessing the high level intent (including read ahead and whether to cache it in the kernel buffers),

Ewen

The Linux "copy problem"

smfrench — Wed, 29 May 2019 21:05:45 +0000

In the presentation I listed seven options that could be added (e.g. to cp and rsync). Other copy tools (like robocopy for Windows) have these (as well as others that may be less important for us on Linux) and may be useful examples.

For example some options which other tools like robocopy let the user select:
- parallel i/o (especially for the uncached copy case)
- allow setting file size first (to reduce the number of metadata updates during the copy operation)
- allow calling the copy system call (copy_file_range API) for file systems which support it
- allow copying additional metadata (e..g xattr and ACLs)
- allow choosing larger i/o (overriding the block size). For some filesystems i/o > 1MB can be much faster than small I/O (some tools will default to 4K or smaller which can be more than 10 times slower)

And then following up on other discussions at the sumimt:
- allow options like encryption or compression (which could be supported over SMB3 for example and probably other filesystems).

The Linux "copy problem"

roc — Wed, 29 May 2019 20:20:52 +0000

> Mason pointed out that kernel developers are not ambassadors to go fix applications across the open-source world, however; "our job is to build the interfaces", so that is where the focus of the discussion should be.

It seems like it would be worth at least diving into `cp` and making sure it's as good as can be, since that gets used so much. Then you can point developers of those other applications at `cp` as an example of how to do things right.

The Linux "copy problem"

desbma — Wed, 29 May 2019 19:57:27 +0000

No mention of the sendfile system call?
It does not work for every case, but when it does, it is much more efficient than doing chunk based copy, and it solves the "what is the optimal I/O size" dilemma.