LWN: Comments on "DVCS-autosync"

DVCS-autosync

Jan_Zerebecki — Tue, 31 May 2011 20:42:59 +0000

One can just keep the whole home directory in git to avoid needing symlinks or something similar. By ignoring * and then negating specific ignores like !.bashrc one gets the same selectiveness. Negated ignores are documented in man gitignore.

symlinks, immutability, wrappers, hardlinks, mount-bind

Duncan — Fri, 27 May 2011 04:56:43 +0000

At least four independent techniques exist for keeping files in two different trees in sync, three of which have already been mentioned in other replies, but it's worth collecting them into one comment, and I didn't see the fourth mentioned at all, yet.

Symlinks: When I switched from MS dark side, symlinks quickly became my favorite *ix filesystem feature. =:^) Unfortunately, by themselves, they have one significant already mentioned limitation: apps that write a temp file and then mv it over the old permanent file, possibly mv-ing the old version to backup at the same time. (This is A good technique on its own as it better ensures integrity of either the file or at last resort its backup, in case of a crash at the wrong moment, but as implemented by many apps, it's not compatible with symlinking.)

In some cases, it might even be possible to reverse symlink, putting the symlinks in the sync-tree, if it's possible to tell whatever's syncing them (git by default, here, I don't know git well enough to know if it can handle that or not) to dereference symlinks instead of syncing the symlink itself.

Ext2/3/4 users at least might also be able to work around the rewriting problem by setting the immutable bit on the file, particularly in the mentioned scenario where an app is overwriting the file even when nothing changed. For filesystems that don't implement immutable, filesystem permissions can be used, setting the file, or its directory if setting only the file doesn't work, read-only, and if necessary, changing the ownership (assuming one has the privs to do so, of course). I didn't see these mentioned as options, yet.

Wrapper scripts: Someone mentioned application wrapper scripts as another option. Create a script by the same name and set it earlier in the path, have it copy the file in question out of the synced tree, then launch the desired app via absolute path name. The wrapper then waits around until the desired app exits, after which it copies the changed file back to the synced tree. (If the user doesn't want to save the changes, the app launch can use exec or similar to avoid having the wrapper stick around for the main app to finish.) Note that GUI launcher users may need to customize their GUI launcher to point at the wrapper, as it may already be using the absolute path or may not be using the same path ordering. FWIW I suspect many LWN users may already be using the wrapper technique for other purposes but may not have thought of using it as a solution for this problem.

Hardlinks: Another suggested solution is hardlinks. That can work for some things, particularly in combination with the immutable/read-only technique above, but hard links have their own negatives. A major one discouraging their use here is that hardlinks are same-filesystem-only, thus limiting one's filesystem management flexibility. A second negative is that files with multiple hard links simply aren't particularly intuitive to those who started on the MS side of things, another weakness of at least the FAT filesystems (I'm unsure about NTFS), where such linking tended to be the result of filesystem corruption and wasn't supported as a feature.

Mount-bind: Not yet mentioned, however, is the very useful mount --bind option, allowing one part of a filesystem, even an individual file, to be mounted elsewhere in the tree as well. =:^) This is a relatively new addition to the toolbox, and thus relatively unknown as it hasn't yet really entered the bank of common wisdom out there yet, as the Linux kernel and the mount command from the util-linux package haven't supported this for all that long. But at least for users with suitable mount privileges available, it is certainly one of the most powerful options, since it not only allows a user to make a directory or file existing in one location in the tree appear elsewhere either in addition or instead of the original location, but ALSO comes with all thue usual mount permissions restrictions ability, including making the mount read-only or read/write, nodev/noexec/nosuid, etc. Mount-binding a file as read-only in effect makes it immutable as addressed from that location, but without the usual filesystem-must-support-immutable limitations, as it's enforced by the kernel mount infrastruction instead of the filesystem. And while it makes little sense in the context of an individual file, directory/subtree level noexec/nodev/nosuid certainly has its uses, as well. Mount-bind functions in this way like an admin-level symlink, and I'm appreciating its benefits more and more as I use it, much as I did symlinks early on, when I first switched from the dark side.

Certainly there's no shortage of choices for keeping a sync-tree file in sync with a file elsewhere in the tree. Given all the choices available, /one/ of them should be workable.

Duncan

DVCS-autosync

nix — Thu, 26 May 2011 19:43:38 +0000

Yes indeed. 'man gitattributes' and search for 'filter' for something that may help here.

fsniper

cdmiller — Thu, 19 May 2011 22:13:28 +0000

Fsniper works nicely for DIY dropbox scripts. It can trigger version control systems, rsync, etc..

DVCS-autosync

dlang — Thu, 19 May 2011 05:20:22 +0000

actually, since git does the compression itself, the answer is to have git uncompress the documents and version the uncompressed data. then when you check it out git assembles the version you want, then compresses it as you check it out.

you can even insert XML aware diff engines if you want.

DVCS-autosync

smurf — Thu, 19 May 2011 05:07:57 +0000

There are a couple of document formats which are singularly not suited for VCSes, though.

OOo documents, for instance, are compressed XML files. There's no sane way to store multiple versions of these in a git archive. Store them uncompressed (dunno how to teach OOo that) and you get a signficant decrease in storage requirements, long-term.

Still needs a domain specific conflict rresolver, of course. You could probably script your way into LibreOffice to do it, though it's nontrivial.

DVCS-autosync

zooko — Tue, 17 May 2011 21:15:04 +0000

(Tahoe-LAFS hacker here)

There are hooks to integrate git-annex with Tahoe-LAFS, and that might make it easier to do the sort of pruning you are talking about here.

http://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects

Also, send me email at zooko@zooko.com and I'll hook you up with a Tahoe-LAFS storage service that you can use for testing.

DVCS-autosync

nye — Tue, 17 May 2011 14:45:28 +0000

You appear to be describing bup: https://github.com/apenwarr/bup

DVCS-autosync

dlang — Mon, 16 May 2011 22:16:40 +0000

I agree that git does not do anything that useful for large binary files, and if that is all you are wanting to sync, then you are better looking at modifying git rather than using it directly.

but this depends in large part on what these binary files are. git supports configurable diff/merge engines, so if there is any sane way to merge your 'binary' files, git will allow you to use it.

please don't get me wrong, I'm not saying that git is perfect, just that it does a better job than anything else for the general case and brings a lot to the party. This makes basing a tool on git (or one of the other DVCS systems if you dislike git for some reason) a very reasonable thing to do rather than just developing your app from scratch.

DVCS-autosync

njs — Mon, 16 May 2011 22:11:02 +0000

> yes, for this use case you may be better of with CVS, but only until you have to reconcile differences between different locations. DVCS tools give you the framework (and many of the mechanisms) for doing this as part of their heritage

Yes, of course. (Though in practice I'm not sure git's current merge mechanisms are well-optimized for the collection-of-large-binary-files case either.)

But that doesn't change the point, which is that git is not a perfect match for this problem, and a better tool that was similar to git in some ways but not in others could potentially do substantially better.

DVCS-autosync

dlang — Mon, 16 May 2011 21:53:16 +0000

I don't know that it always uses mmap, but it's the use of mmap that imposes the file size limit on individual files.

pack files are limited in that they use a 32 bit offset into them, but that's a matter of optimisation for files that can be diffed and compressed.

yes, for this use case you may be better of with CVS, but only until you have to reconcile differences between different locations. DVCS tools give you the framework (and many of the mechanisms) for doing this as part of their heritage

DVCS-autosync

njs — Mon, 16 May 2011 21:34:22 +0000

Oh, does it use mmap exclusively for all file access now? Including repacking and everything? Neat.

You need two local copies if you want to do local version control, and also to let people edit files normally on disk (as opposed to, say, interposing a FUSE filesystem to observe edits as they happen). But the systems we're talking about are not trying to do local version control. They're trying to do remote backup and syncing!

*For this use-case*, you might almost be better off with CVS than with git. Its handling of binary files is dumb, but at least it wouldn't double your local storage requirements. Even better, of course, would be a system that stored the second copy on the remote server only, and then used something clever like rsync to upload the deltas.

Or maybe one could do something clever with libgit2 and librsync to let you directly and efficiently commit a local set of files to a remote bare repository...

DVCS-autosync

dlang — Mon, 16 May 2011 21:12:43 +0000

you have a valid point in terms of being able to mmap things (which is not quite the same as being able to fit them in memory). this is a limitation on 32 bit systems

however, how can you do version control if you don't keep a copy of the file somewhere else? if someone changes it, how can you get back what was there before without another copy?

DVCS-autosync

njs — Mon, 16 May 2011 20:58:10 +0000

So what's the difference? It's an "information tracker or content-addressable filesystem" that is heavily optimized on the assumptions that the information/content its holding is highly compressible (by deltas and by gzip), which makes it okay to have two copies of everything (one in the repo and one in the filesystem), and also (last I checked) that it can hold any individual file in RAM. Those things are totally reasonable assumptions for source code, and terrible assumptions for a directory full of Blu-Ray rips. (And those are just the two issues that occur off the top of my head.)

Linus' point is that git is designed with good decoupled interfaces between its internal components, not that it's always going to be good at solving problems that it wasn't designed for.

DVCS-autosync

Cato — Mon, 16 May 2011 16:10:23 +0000

I found SpiderOak didn't work at all well, though that was two years back. I was using it mostly for online backups, and found it would never complete syncing, and on one machine with only a few GB to back up, it didn't back up most of the files. It also generated a corrupt ZIP file when trying to recover the data. I found the UI very complex given what it's trying to achieve is quite simple.

It may have improved, but I'd encourage very careful testing at least.

DVCS-autosync

mathstuf — Mon, 16 May 2011 15:37:13 +0000

Hmm...somehow missed that option. The rationale behind symlinks is that "uninstallation" of an application's configuration is easier (just clean up broken links in $HOME/{.*,bin}). Using hardlinks would solve the "keeping things in sync" problem, but still be less-than-ideal (false positives would be high when detecting "uninstalled" applications).

DVCS-autosync

drag — Mon, 16 May 2011 11:00:10 +0000

Well that's confusing then.

So Git is the backend for Git DVCS...

DVCS-autosync

dgm — Mon, 16 May 2011 10:53:24 +0000

"Git is very carefully optimized to provide a high performance revision control system for text files."

You were almost right up until "revision control system for text files". Linus described Git more in the line of an information tracker, or content addressable filesystem, used to implement a DVCS that accidentally bears the same name.

DVCS-autosync

yohahn — Mon, 16 May 2011 05:44:32 +0000

This looks a bit out of date (last version 2009), but I've wondered about boxbackup http://www.boxbackup.org/ as a DIY dropbox

DVCS-autosync

madscientist — Mon, 16 May 2011 00:18:39 +0000

I've been using SpiderOak since last year and I think it's great. Very easy to use, and as mentioned it works on my Linux systems, my wife's Mac, and our kids' Windows systems. I like the encryption model where my content is encrypted locally using my local passphrase and only the already-encrypted content is stored remotely. The backup utility is quite flexible and doesn't require that you collect all your files to be backed up in one place.

And it's 2G free or $100/yr for 100G of storage which is quite reasonable, IMO.

ObDisclaimer: I don't work for or have any financial interest in SpiderOak.

DVCS-autosync

drag — Sun, 15 May 2011 23:47:04 +0000

Yeah I love git for a variety of reasons. I just worry that there is a impedance mismatch between it and the goal of 'Dropbox replacement'.

Git is very carefully optimized to provide a high performance revision control system for text files.

However typical Dropbox usage only deals with a tiny amount of text data and revision control is borderline irrelevant for most people's uses.

I just remember using Git for a variety of purposes and realizing 'Hey putting that ISO image for the cdrom I made into my repository was a very very stupid thing to do'... yet people are going to want to store ISOs, mp3s, zip files, and huge raw-formatted camera images and other things of that nature in anything calling itself a 'A open source Dropbox replacement'

DVCS-autosync

bradfitz — Sun, 15 May 2011 23:42:35 +0000

Yes, Camlistore is rapidly becoming a reasonable Dropbox replacement. Not quite yet, but I'm working on it every free second I have.

DVCS-autosync

nicooo — Sun, 15 May 2011 23:22:59 +0000

How about using hard links for those files?

DVCS-autosync

dan_a — Sun, 15 May 2011 21:38:05 +0000

Git in particular has one killer feature here - its built in checksumming which ensures that any copies of files are uncorrupted.

DVCS-autosync

sailorxyz — Sun, 15 May 2011 19:49:24 +0000

A good commercial alternative to Dropbox is SpiderOak https://spideroak.com/ It's not quite as simple as Dropbox but uses proper encryption and only you have the keys. SpiderOak cannot access your files. It's also cheaper than Dropbox. They also have a free 2GB account. Works across Linux Mac and MS platforms.

Unison!

Velmont — Sun, 15 May 2011 00:11:56 +0000

It is easy to build something similar around this. Using inotify, or incron, you can wait for close-signal, and then run unison on just that path-tree automatically.

I do something like that, only a bit less advanced with my unison script. I really love Unison, and it blows the butt of Dropbox and has for my kind of use. :-)

I have many machines, all of them using unison-323, it's not hard to standardize on a version, although yes, this could be easier.

DVCS-autosync

davidsarah — Sat, 14 May 2011 23:58:27 +0000

The developers of Tahoe-LAFS are actively considering adding inotify-based synchronization, either via something like DVCS-Autosync or more directly using Pyinotify; see http://tahoe-lafs.org/pipermail/tahoe-dev/2011-May/006356...

DVCS-autosync

drag — Sat, 14 May 2011 21:45:16 +0000

So when I want to share my 800MB movie with a friend or have it available on another machine it's just not going to happen.

From dropbox's website:
https://www.dropbox.com/help/5
> Files uploaded to Dropbox via the desktop application have no file size limit.

> Files uploaded through the website (by pressing the upload button) have a 300 MB cap. In other words, each file you upload through the website must be 300 MB or less.

> All files uploaded to your Dropbox must be smaller than the size of your Dropbox account's storage quota. For example, if you have a free 2 GB account, you can upload one 2 GB file or many files that all add up to 2 GB. If you are over your storage quota, Dropbox will stop syncing until you are below your limit.

Dropbox's revision control system is optional and only will save revisions for 30 days. Many people want sync software to sync a significant amount of data.... I am guessing that for most people's purposes revision control is much less important then just automatic syncing.

The ability to carelessly drop a file into your drop box and have it automatically available on any machine you happen to want to use is the 'killer feature' for Dropbox. The idea of trying to do something like manage a 4GB mp3 collection using something like Git commit sounds like a nightmare to me.

DVCS-autosync

dlang — Sat, 14 May 2011 20:53:51 +0000

when trimming the history, I can see a couple of desired approaches

1. I want to keep all revisions up to X days/revisions ago, but I don't care much about anything before that.

2. I know that this is a good version, I want to make sure I can always see this version, but I don't care about all the other versions I haven't tagged.

3. I want to save one version per day/week/?? but not _every_ version (i.e. get me back into the approximate area, but don't spend the resources to save every version)

I can also see combinations of these

for example I want a snapshot each day, but every revision for the last week

given how efficient the git compression can be, I would suggest having a dry-run mode that will tell you how much space you are saving by throwing away the history. for binary files it may be quite a bit, but for other files you may be surprised at how little it saves.

DVCS-autosync

dlang — Sat, 14 May 2011 20:47:00 +0000

people are attracted to DVCS systems because they already address a lot of the problems that can come up when you have more than two locations that you are trying to sync.

why reinvent the wheel when you can reuse the work that someone else has done?

DVCS-autosync

dlang — Sat, 14 May 2011 20:45:39 +0000

it's not that git is especially inefficient at handling binary files, it's that the compression and other efficiencies that git can get with text files don't work on binary files.

but git is not any worse in dealing with binary files than any other solution where you want to be able to retrieve any version that ever existed.

that said, git does have some limitations in terms of max sizes of things

DVCS-autosync

elanthis — Sat, 14 May 2011 19:40:20 +0000

You mean... people who _write code_ think like people who _write code_?

How surprising.

DVCS-autosync

nix — Sat, 14 May 2011 13:08:50 +0000

Actually, you can plug in domain-specific diff/merge tools into git (OpenOffice documents were a named use case for this). Obviously the merge tools for binary stuff need their own way to do conflict resolution as well.

Git is exactly as efficient at handling binary files as handling text files: you have to go back to CVS to find something that isn't good at binary files.

DVCS-autosync

loevborg — Sat, 14 May 2011 12:50:41 +0000

I agree completely. DVCS are useful for plain text, which for almost anyone except programmers is only a tiny portion of your data. Using git doesn't gain you anything, because we can't use diff for ODF etc. What we need is pretty much a drop-in replacement for dropbox.

With all due respect, this seems the expression of a sort of narrow-mindedness typical of programmers. People who work with source code all day assume that code is the model of computer work. However, code is actually very particular. What we really need is a place to store our letters, PDF forms, spreadsheets and so on, which are quite different from code in many respects.

The situation is similar in the field of text editors. Here every week or so someone starts a new editor which targets ... programmers. But there still is hardly a single text editor which is any good for writing simple English language prose! Here there is little use for syntax coloring, line numbers, nonproportional fonts. Instead you need configurable line spacing, easy-to-read proportional fonts, paragraph-based (not line-based) navigation, word-wrapping, etc. Still instead of solving this very real problem, people come up with new solutions to already solved problems.

DVCS-autosync

juliank — Sat, 14 May 2011 12:12:07 +0000

> esmtp is similar in that it refuses to work
> if $HOME/.esmtprc is a symlink which is at
> least understandable, but still less than ideal.

At least it works for /etc/esmtprc, I use that on my system to keep the esmtprc with password in it in an encrypted mount point.

jak@jak-thinkpad:~$ ls -l /etc/esmtprc
lrwxrwxrwx 1 root root 25 Aug 2 2010 /etc/esmtprc -> /home/jak/Private/esmtprc

DVCS-autosync

nteon — Sat, 14 May 2011 09:54:55 +0000

I stumbled across this earlier, which seems similar: http://camlistore.org/

DVCS-autosync

rmayr — Sat, 14 May 2011 09:44:31 +0000

My personal use case is a tree of most of my work so far, including source code and documents. For these, I explicitly want the history (thinking of papers, theses, project proposals, reports, etc.) and have relied upon tracking the changes in quite a few instances. However, I try to keep large binary blobs outside this tree (that might change with decent git-annex integration, although I don't know (yet) how best to do that).

DVCS-autosync

rmayr — Sat, 14 May 2011 09:21:06 +0000

> I don't know how DVCS-autosync uses git, but there is nothing magical about "master" in git, in fact I rarely even care about it. All branches are equal with git, and all repos are equal. There is no need to even clone repos, simply init a new repo and fetch into it the branch you care about. git is a DVCS, it is not subversion.

dvcs-autosync doesn't care about it at all. Whatever the checked out branch is, it will work on. Dieter and I have already been discussing that explicit support for branches may be good to have in future versions for accessing (and possibly modifying) older revisions. But right now, it's completely transparent. You can even have two clones of the repository (in different directories) pointing to different branches and have two instances of dvcs-autosync working on those branches concurrently.

DVCS-autosync

rmayr — Sat, 14 May 2011 09:17:37 +0000

[dvcs-autosync author here...]

That is what I will be trying to do for those files not managed by git-annex (not that git-annex support is done yet...). However, I don't know yet what the best way to rewrite the history will be and am happy to take any suggestions at this point. filter-branch is something I have used before for splitting repositories, but there may be better ways to reduce history. From a user point of view, the use cases that should be supported by the trimmed history are IMHO: a) "Give me the state of this directory at a point in time up to X days/weeks/months ago" and b) "I want this file as it was Y revisions ago". Everything that does not fit in either a) or b) with user-configured values for X and Y can be pruned.

DVCS-autosync

ptman — Sat, 14 May 2011 06:24:29 +0000

If you only sync in one direction (mirror), rsync is usually a better choice than unison as it uses a (somewhat) standard protocol instead of requiring the exact same version of unison in both ends.