LWN: Comments on "The casync filesystem image distribution tool"

The casync filesystem image distribution tool

abo — Mon, 26 Jun 2017 10:02:47 +0000

As far as I can tell, SHA-1 is not used.

The casync filesystem image distribution tool

nix — Fri, 23 Jun 2017 13:51:27 +0000

Are relatively unlikely to collide, then. :) (and git already checks for that collision mechanism. One wonders how a future SHA-256 migration would work with existing bup repos, though... something to look at.)

The casync filesystem image distribution tool

nix — Fri, 23 Jun 2017 13:49:24 +0000

I don't think it wrong that backups are encrypted -- mine are -- I think that the *backup program* is usually the wrong place for encryption, because frankly encryption is a horribly difficult problem best worked on by people who specialize in it, not people who are doing it on the side of writing a backup program.

So I'd go with an unencrypted backup atop an encrypted filesystem or encrypted block device layer. This has the advantage that you can use whatever ridiculously contrived means you like to acquire the passphrase, which is relatively rarely possible with backup-level encryption (e.g. mine currently comes from a shell script that does a challenge-response on a yubikey -- and getting that requires an ssh to the machine with the key plugged in...)

The casync filesystem image distribution tool

mgedmin — Fri, 23 Jun 2017 11:37:30 +0000

> represented as SHA-1 hashes so can't collide

*cough* https://shattered.io

The casync filesystem image distribution tool

Sesse — Fri, 23 Jun 2017 10:20:59 +0000

Why is backup the wrong place for encryption? I'm very happy that my backup host can't compromise my servers—all it can do is get encrypted backups out of it. Less trust is good.

(Likewise, my servers don't have SSH access to my backup host.)

The casync filesystem image distribution tool

drag — Wed, 21 Jun 2017 17:26:26 +0000

> I'm not entirely sure what the point of turning it into a backup tool is

Simplify the user experience with these sorts of tools, I expect.

If the operations involving the creating, pushing, distributing, tracking, sharing, pulling, backing up, and restoration of file systems images have a lot of overlap then it makes sense to have the same basic tools and protocols for all those things. That way it's easier to build high-level solutions on top this tool.

So right now if I want to have, in my environment, the ability to use containers, virtual machines, sync my workstation home directory and backup my workstation and state-full applications/databases then these all require lots of different tools and services to configure. I have to, maybe, setup cinder for virtual machine images, docker registry for containers images, rsync for backing up, etc etc.

Wouldn't it be nice to have a single thing that could provide all that?

Provided, of course, the actual operations for these different needs have a lot of overlap and similarities. Don't want to end up with some massive mess of conflicting and mismatched functionality in a single service...

The casync filesystem image distribution tool

rkeene — Wed, 21 Jun 2017 14:33:36 +0000

For a slightly different, but similar, take on a similar problem take a look at AppFS ( http://appfs.rkeene.org/ )

The casync filesystem image distribution tool

nix — Wed, 21 Jun 2017 11:29:24 +0000

So does bup if you ignore encryption (which you should IMHO: the backup is the wrong place for it, we have perfectly good encryption at lower levels, be that cryptfs or LUKS). (It's gaining remote restore, too, as we speak, though this is fairly redundant because you can use *any* other method to ship the backup contents over for a restore run anyway: I just use NFS. So this is only a convenience, really.)

The casync filesystem image distribution tool

Sesse — Wed, 21 Jun 2017 07:42:35 +0000

duplicity is painfully slow, though.

The casync filesystem image distribution tool

walex — Wed, 21 Jun 2017 06:43:53 +0000

Not really sure what's the point of this, considering the existence of Jigdo or of the content-based Arvados Keep distributed filesystem, which match two different application areas quite well.

The casync filesystem image distribution tool

aggelos — Tue, 20 Jun 2017 22:10:02 +0000

Seems the main contribution is the definition of the chunk store. Then you'd just need a tracker for the set of mirrors that could have any related data set (oh wait :-)).

What I didn't find in the blog post is a strategy for pruning chunks - specifically one that would work well for CDNs and mirror operators. Reasonable strategies could be devised without any extra effort of course (e.g. ctime/atime based), but perhaps the format could accommodate more elaborate algorithms?

The casync filesystem image distribution tool

jhoblitt — Tue, 20 Jun 2017 20:23:03 +0000

Doesn't duplicity already nicely handle chunked backups, with encryption, and support for remote push?

The casync filesystem image distribution tool

compenguy — Tue, 20 Jun 2017 20:12:07 +0000

> I'm not entirely sure what the point of turning it into a backup tool is [...]

Perhaps the point isn't about backup, but rather about restore. It seems like it might enable fast, network-efficient snapshot-type restore operations.

Not really sure, though... I'm still trying to wrap my head around it.

The casync filesystem image distribution tool

nix — Tue, 20 Jun 2017 19:36:37 +0000

Very nice. It appears to be a network-synchronization version of bup, almost (at least with regard to the method used to deduplicate, though its use of a better version of tar feels nicer than the bup approach of representing metadata by stuffing it into a "file" in each directory's representation on the backup, though this hack does work well enough given that all real on-disk file chunks are represented as SHA-1 hashes so can't collide, and has the major advantage that bup can just use git for its underlying storage, which is redundant for casync since it wants its "blobs" to be independent operating-system files).

I'm not entirely sure what the point of turning it into a backup tool is, since as a backup tool it feels like everything it does, bup would do better... but as a distribution tool, particularly over CDNs, it seems likely to be without equal. Ah well, the more options the merrier, for backup!