CLI Magic: Simple backup is Mirdir (Linux.com) [LWN.net]

CLI Magic: Simple backup is Mirdir (Linux.com)

Posted Nov 24, 2005 10:28 UTC (Thu) by mp (subscriber, #5615)
In reply to: CLI Magic: Simple backup is Mirdir (Linux.com) by tjw.org
Parent article: CLI Magic: Simple backup is Mirdir (Linux.com)

For example if I want to mirror a directory containing millions of files totalling over a TiB in disk space; I wouldn't want use the following command for the intitial copying:
rsync -azP somehost:/some/dir/ /some/dir/
This is because this command could take days to complete and could make the machine swap to death as it deals with the mind bogglingly huge checksumming.

True, though the --whole-file option of rsync would help with the checksumming problem.

(Log in to post comments)

CLI Magic: Simple backup is Mirdir (Linux.com)

Posted Nov 27, 2005 3:03 UTC (Sun) by zblaxell (subscriber, #26385) [Link]

The problem isn't checksumming. By default rsync does no checksumming until it encounters a file that exists on both source and destination with different size or timestamp, unless you add '-c'.

The problem is that rsync *requires* a full list of all files be generated and sent to the client, with both client and server having a copy of this list stored in RAM, before *any* data is transferred. A 32-bit processor has enough address bits for a few million files. After that, you need better software, or more address bits.