LWN.net Logo

DVCS-autosync

DVCS-autosync

Posted May 13, 2011 23:16 UTC (Fri) by mathstuf (subscriber, #69389)
Parent article: DVCS-autosync

I'm still looking for a way to handle data file synchronization. It appears that this might by the tool I use in the future, but at the moment, it doesn't quite do what I need. The problem is that it's just a few files spread around my home directory. My configuration files are handled via git (this tool wouldn't work because I have, e.g., branches for work-related changes at work). What I need is a tool to sync my newsbeuter database, bookmarks, and the like that don't logically belong in a configuration repository.

For configurations, one major problem that I've come across is with applications that don't behave correctly with a configuration file that is a symlink into the repository so I can see changes and commit them appropriately. My current list of apps that break this are: tin (overwrites the configuration file with a mv when exiting; even if no changes are made; also writes things like "last search string" to the file which don't belong), finch (probably similar; haven't investigated it much), and weechat (I'd use it over irssi if it didn't do this). esmtp is similar in that it refuses to work if $HOME/.esmtprc is a symlink which is at least understandable, but still less than ideal.

For data files, the tool has to support a list of files to sync (symlinking these will probably work even less well than configuration files) rather than just syncing a directory. It appears as though DVCS-autosync doesn't support this (yet) and nothing I've found does this to my satisfaction either.

The automatic committing is also something that I'm personally wary of (newsbeuter would create commits on the 8MB sqlite db all the time) and git-annex would be the way to go for it, preferably waiting until newsbeuter is closed to make the commit.


(Log in to post comments)

DVCS-autosync

Posted May 14, 2011 0:29 UTC (Sat) by shredwheat (guest, #4188) [Link]

I saw Wuala recently and want to check it out. If anyone has experimented with it, I'd like to hear some analysis. It comes with several of the drawbacks of Dropbox, but seems to provide 'better' everything?
http://www.wuala.com

DVCS-autosync

Posted May 14, 2011 0:48 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

Just looking through the feature list, the encryption is nice as is the "trading" idea, but the drawbacks of Dropbox (namely "...add a file to your Wuala Sync folder...") prevents it from meeting my requirements. A way to do profiles would be nice as well (the MLs I subscribe to at work in tin are not the same as those I subscribe to at home, so syncing that depending on the machine's use would be ideal, but not a hard requirement).

DVCS-autosync

Posted May 14, 2011 5:53 UTC (Sat) by ewen (subscriber, #4772) [Link]

FWIW, with some tools I always run them via a wrapper (often named the same, but earlier in the search path, and explicitly calling the real one via a full pathname), so that it can do various tidying up before/after. Eg, my cash tracking application is run from a wrapper that auto-commits changes into git, if there are any, and I used to run trn via a similar wrapper.

You could probably do the same thing with tin: copy the state file out of a repository somewhere to the name it expects, run tin, let it overwrite the file, diff the resulting file against the repository, and auto-commit/push if it has changed. Aside from the last-write-wins-whole-file issue (or whatever your auto-resolve-conflicts policy becomes), it seems like it should automagically do what you want.

Ewen

PS: You're still left with problems of state stuck inside always-running applications that only write state on exit, but that's baked in to the applications in question.

DVCS-autosync

Posted May 14, 2011 12:12 UTC (Sat) by juliank (subscriber, #45896) [Link]

> esmtp is similar in that it refuses to work
> if $HOME/.esmtprc is a symlink which is at
> least understandable, but still less than ideal.

At least it works for /etc/esmtprc, I use that on my system to keep the esmtprc with password in it in an encrypted mount point.

jak@jak-thinkpad:~$ ls -l /etc/esmtprc
lrwxrwxrwx 1 root root 25 Aug 2 2010 /etc/esmtprc -> /home/jak/Private/esmtprc

DVCS-autosync

Posted May 15, 2011 23:22 UTC (Sun) by nicooo (guest, #69134) [Link]

How about using hard links for those files?

DVCS-autosync

Posted May 16, 2011 15:37 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

Hmm...somehow missed that option. The rationale behind symlinks is that "uninstallation" of an application's configuration is easier (just clean up broken links in $HOME/{.*,bin}). Using hardlinks would solve the "keeping things in sync" problem, but still be less-than-ideal (false positives would be high when detecting "uninstalled" applications).

symlinks, immutability, wrappers, hardlinks, mount-bind

Posted May 27, 2011 4:56 UTC (Fri) by Duncan (guest, #6647) [Link]

At least four independent techniques exist for keeping files in two different trees in sync, three of which have already been mentioned in other replies, but it's worth collecting them into one comment, and I didn't see the fourth mentioned at all, yet.

Symlinks: When I switched from MS dark side, symlinks quickly became my favorite *ix filesystem feature. =:^) Unfortunately, by themselves, they have one significant already mentioned limitation: apps that write a temp file and then mv it over the old permanent file, possibly mv-ing the old version to backup at the same time. (This is A good technique on its own as it better ensures integrity of either the file or at last resort its backup, in case of a crash at the wrong moment, but as implemented by many apps, it's not compatible with symlinking.)

In some cases, it might even be possible to reverse symlink, putting the symlinks in the sync-tree, if it's possible to tell whatever's syncing them (git by default, here, I don't know git well enough to know if it can handle that or not) to dereference symlinks instead of syncing the symlink itself.

Ext2/3/4 users at least might also be able to work around the rewriting problem by setting the immutable bit on the file, particularly in the mentioned scenario where an app is overwriting the file even when nothing changed. For filesystems that don't implement immutable, filesystem permissions can be used, setting the file, or its directory if setting only the file doesn't work, read-only, and if necessary, changing the ownership (assuming one has the privs to do so, of course). I didn't see these mentioned as options, yet.

Wrapper scripts: Someone mentioned application wrapper scripts as another option. Create a script by the same name and set it earlier in the path, have it copy the file in question out of the synced tree, then launch the desired app via absolute path name. The wrapper then waits around until the desired app exits, after which it copies the changed file back to the synced tree. (If the user doesn't want to save the changes, the app launch can use exec or similar to avoid having the wrapper stick around for the main app to finish.) Note that GUI launcher users may need to customize their GUI launcher to point at the wrapper, as it may already be using the absolute path or may not be using the same path ordering. FWIW I suspect many LWN users may already be using the wrapper technique for other purposes but may not have thought of using it as a solution for this problem.

Hardlinks: Another suggested solution is hardlinks. That can work for some things, particularly in combination with the immutable/read-only technique above, but hard links have their own negatives. A major one discouraging their use here is that hardlinks are same-filesystem-only, thus limiting one's filesystem management flexibility. A second negative is that files with multiple hard links simply aren't particularly intuitive to those who started on the MS side of things, another weakness of at least the FAT filesystems (I'm unsure about NTFS), where such linking tended to be the result of filesystem corruption and wasn't supported as a feature.

Mount-bind: Not yet mentioned, however, is the very useful mount --bind option, allowing one part of a filesystem, even an individual file, to be mounted elsewhere in the tree as well. =:^) This is a relatively new addition to the toolbox, and thus relatively unknown as it hasn't yet really entered the bank of common wisdom out there yet, as the Linux kernel and the mount command from the util-linux package haven't supported this for all that long. But at least for users with suitable mount privileges available, it is certainly one of the most powerful options, since it not only allows a user to make a directory or file existing in one location in the tree appear elsewhere either in addition or instead of the original location, but ALSO comes with all thue usual mount permissions restrictions ability, including making the mount read-only or read/write, nodev/noexec/nosuid, etc. Mount-binding a file as read-only in effect makes it immutable as addressed from that location, but without the usual filesystem-must-support-immutable limitations, as it's enforced by the kernel mount infrastruction instead of the filesystem. And while it makes little sense in the context of an individual file, directory/subtree level noexec/nodev/nosuid certainly has its uses, as well. Mount-bind functions in this way like an admin-level symlink, and I'm appreciating its benefits more and more as I use it, much as I did symlinks early on, when I first switched from the dark side.

Certainly there's no shortage of choices for keeping a sync-tree file in sync with a file elsewhere in the tree. Given all the choices available, /one/ of them should be workable.

Duncan

DVCS-autosync

Posted May 31, 2011 20:42 UTC (Tue) by Jan_Zerebecki (guest, #70319) [Link]

One can just keep the whole home directory in git to avoid needing symlinks or something similar. By ignoring * and then negating specific ignores like !.bashrc one gets the same selectiveness. Negated ignores are documented in man gitignore.

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds