Deprecating scp [LWN.net]

Deprecating scp

Posted Nov 6, 2020 6:10 UTC (Fri) by nilsmeyer (guest, #122604)
In reply to: Deprecating scp by carORcdr
Parent article: Deprecating scp

scp manpage has 148 lines on my system, rsync man page has 2444. "drop in" replacement would mean it would just work like scp does and one could simply alias scp="rsync". I don't think that's the case. I would say the allure of scp is that it is relatively simple to use, behaves like another well known tool (cp) and you can accomplish a simple task without having to read through lines and lines of help text.

to post comments

Deprecating scp

Posted Nov 6, 2020 6:49 UTC (Fri) by marcH (subscriber, #57642) [Link] (16 responses)

It is true that rsync can do everything but the kitchen sink but in practice there are only two rsync things you need to remember:

- Use -a ("archive") always.
- The infamous source trailing slash: on the source side, foo/ is equivalent to foo/*, including dot files

And that's it, I never remember anything else. Plus who reads man pages from top to bottom? I have no idea how long most man pages are because I only search them.

I dropped not just scp for rsync but I also dropped cp -R because if you run "cp -R dir/ brandnewdir/" multiple times then it is not idempotent, it does something different the first time. Crazy!

Except on macOS (and BSDs?) where "cp -R dir/ brandnewdir/" is idempotent.

But wait, on macOS "cp -R dir brandnewdir/" (no trailing slash on the source dir) is again not idempotent!

Imagine you're copying something very big and the first time gets interrupted for some reason. Now you are _really_ screwed if you run it again and the second time does something different...

tl;dr: cp -R is insane, use rsync -a. Using it more than once is not just safer it's also much faster of course: it can be used as an actual backup tool.

rsync -a is not the same as cp -R

Posted Nov 6, 2020 7:55 UTC (Fri) by mchehab (subscriber, #41156) [Link] (4 responses)

> It is true that rsync can do everything but the kitchen sink but in practice there are only two rsync things you need to remember:
>
> - Use -a ("archive") always.
> - The infamous source trailing slash: on the source side, foo/ is equivalent to foo/*, including dot files
>
> And that's it, I never remember anything else. Plus who reads man pages from top to bottom? I have no idea how long most man pages are because I only search them.

You should revisit your concept and read the manuals ;-)

> tl;dr: cp -R is insane, use rsync -a. Using it more than once is not just safer it's also much faster of course: it can be used as an actual backup tool.

If you're copying big sparse file(s) (for instance, VM images) and don't use -S on rsync, you may end running out of disk space at the destination, as rsync will create a non-sparsed files at the destination. See:

    $ dd if=/dev/zero of=test bs=1k seek=2048k count=1
    $ rsync -a test test2
    $ du -h test test2
       4,0K	test
       2,1G	test2

Also, depending on the type of file permissions, "-a" is not enough. You may need other flags (-A, -H, -X).

rsync -a is not the same as cp -R

Posted Nov 6, 2020 8:06 UTC (Fri) by Wol (subscriber, #4433) [Link] (3 responses)

My photographs directory (large files, 24MP raw) is heavily sym-linked. Does -a retain hard symlinks - I believe not. Copying that directory could well cause a "disk full" on the destination, too ...

Cheers,
Wol

rsync -a is not the same as cp -R

Posted Nov 6, 2020 10:01 UTC (Fri) by taneli (subscriber, #95265) [Link]

Indeed, you will need -H to preserve hard links.

rsync -a is not the same as cp -R

Posted Nov 7, 2020 23:22 UTC (Sat) by Sesse (subscriber, #53779) [Link] (1 responses)

FWIW, links in UNIX are either hard (hardlinks) or symbolic (symlinks). There's no such thing as “hard symlinks”.

rsync -a is not the same as cp -R

Posted Nov 8, 2020 0:03 UTC (Sun) by Wol (subscriber, #4433) [Link]

Whoops :-)

I meant, of course, hard links.

Oh - and if you don't need to preserve hard links (which I do), it's not a good idea to try. I think my rsync or cp crawled, I had so many it needed to keep track of ...

Cheers,
Wol

Deprecating scp

Posted Nov 6, 2020 8:05 UTC (Fri) by pabs (subscriber, #43278) [Link]

In some cases there are some more things to preserve (sparse files, hard links, ACLs and extended attributes) and you definitely want to transfer deletions as well. I tend to use this:

rsync -aSHAX --delete

Deprecating scp

Posted Nov 6, 2020 10:32 UTC (Fri) by gray_-_wolf (subscriber, #131074) [Link] (5 responses)

> tl;dr: cp -R is insane, use rsync -a. Using it more than once is not just safer it's also much faster of course: it can be used as an actual backup tool.

Does rsync -a also use CoW the way cp does as long as the filesystem supports it?

Deprecating scp

Posted Nov 6, 2020 12:48 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

Dunno what option it is (and I don't think it's -a), but rsync does have an option that says "only write stuff that's changed". Great for making backups on a journalled file system as each snapshot is a full backup, but apart from the first only takes up the space of an incremental... (and yes for large files eg databases it only writes that part of the file)

Cheers,
Wol

Deprecating scp

Posted Nov 19, 2020 17:46 UTC (Thu) by nye (guest, #51576) [Link]

> rsync does have an option that says "only write stuff that's changed"

--inplace --no-whole-file

By default rsync will only use --whole-file if both source and destination are local, so the second option is redundant in that case.

Deprecating scp

Posted Nov 6, 2020 12:50 UTC (Fri) by nix (subscriber, #2304) [Link] (2 responses)

No, and apparently it never will use reflinks because it is out of scope. It also can't copy the filesystem-specific attributes (like the immutable flag) set via chattr/lsattr. Patches to do this have been proposed and rejected :(

Deprecating scp

Posted Nov 19, 2020 22:27 UTC (Thu) by rodgerd (guest, #58896) [Link] (1 responses)

That's very disappointing. What was the rationale?

Deprecating scp

Posted Nov 20, 2020 22:46 UTC (Fri) by nix (subscriber, #2304) [Link]

That it was "out of scope", that rsync's job was to copy files, and nonportable stuff doesn't count. (I guess they define what is "nonportable".)

Deprecating scp

Posted Nov 6, 2020 10:52 UTC (Fri) by grawity (subscriber, #80596) [Link]

My trick is to use "cp -a source/. dest/" to always get just the contents.

rsync options exist for a reason

Posted Nov 6, 2020 17:02 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] (1 responses)

I disagree, the many other rsync options exist for a reason. I have a situation where I use three lines of options for rsync (!).

The world needs a "simple replacement for cp that copies remotely" and a "sophisticated tool for remote synchronization". They don't need to be the same tool, and historically have been different. I'd like to see a more secure version of the first category. In the first category most people don't really need the scp *protocol*, they just need a simple replacement for "cp" that works remotely. That is, it's the CLI interface, not the protocol, that matters.

rsync options exist for a reason

Posted Nov 6, 2020 21:02 UTC (Fri) by marcH (subscriber, #57642) [Link]

> I disagree, the many other rsync options exist for a reason. I have a situation where I use three lines of options for rsync (!).

Of course other rsync options exist for a reason and I do look them up and use them sometimes. Above I was only referring to basic, [s]cp-like usage.

Answering other comments too: if you must preserve hardlinks, sparse files, ACLs or some other unusual stuff then you better check the man page of _any_ tool you use.

> The world needs a "simple replacement for cp that copies remotely" and a "sophisticated tool for remote synchronization". They don't need to be the same tool, and historically have been different.

They don't need to be different tools either. I've been using a unique tool for "simple replacement for recursive, _local_ cp" all the way to "sophisticated tool for remote synchronization" and it has made simple things easy and complex things possible. So if you want a solution that already exists now then give it a try. If scp muscle memory is just too strong then a sincere "best of luck!"

Deprecating scp

Posted Nov 7, 2020 6:45 UTC (Sat) by ncm (guest, #165) [Link]

The UNIX tradition is to read all the man pages from front to back and memorize it all.

That is harder nowadays, because there are a lot more with more pages each. But a reasonable selection of them deserve the old-school treatment. Rsync it probably one that does. Anyway, a curated selection of its options. (Eyes rolling? That's our world now.)

Instead of sitting and memorizing the whole collection, of an evening, why not bring home rsync(1), and memorize just that? What else would you do, look at your phone?