Weekly Edition
Return to the Development page
| |||||||||||||
DVCS-autosyncDropbox is a fantastic concept, but it has some major flaws — namely that the software is proprietary, and that users must entrust their files to a third party. Unfortunately, Linux users have few alternatives to Dropbox — and even fewer that are free software and mature enough for daily use. While not quite mature, DVCS-Autosync is looking like an interesting alternative for file synchronization between Linux systems. One other option is SparkleShare, an attempt to provide a multi-platform, git-based replacement for Dropbox. However, SparkleShare is still in early development, and is somewhat ambitious — it attempts to replicate not only Dropbox's main feature (synchronizing files seamlessly) but also to provide collaboration tools and an easy to use GUI frontend for viewing file revisions. SparkleShare also puts off some potential users because of the project's technology choices — namely that SparkleShare is written in Mono and uses git rather than another Distributed Version Control System (DVCS). Many users simply want a tool that will synchronize files, and have no need for collaboration features, multiplatform support, or a GUI to manage all of it. For those users who don't find SparkleShare appealing, DVCS-Autosync is a Python-based alternative that is not as full-featured — but it does get the job done. René Mayrhofer announced DVCS-Autosync on March 10th of this year, though the tool has actually been around for quite a while before that. Mayrhofer says that the script started life before SparkleShare was announced, as part of managing his home directory using SVN and then git. Mayrhofer says he lacks time to try to contribute to SparkleShare, but was "pleased to see others go into the same direction" and that he wanted to publish what he had so far. All about DVCS-AutosyncSo what is DVCS-Autosync? It's a Python script to keep DVCS repositories in sync when files are added, changed, or removed by automatically committing and pushing or pulling the files to the server and clients. It does this by watching one directory for changes, using inotify and communicating between clients using XMPP (Jabber). Thus, when changes are made clients receive a message over XMPP to notify them of the change and to initiate a pull from the git repository. DVCS-Autosync has been based around (and tested with) git, but it's been written to allow use with other distributed version control systems — so if a user has a strong loyalty to Mercurial, they would be able to configure DVCS-Autosync to use Mercurial or another DVCS instead by editing the commands used for various DVCS-Autosync operations. For example, users could replace the default pull and push commands ("git pull" and "git push," respectively) with the appropriate pull and push commands for another DVCS.
Setting up and Using DVCS-AutosyncRight now, DVCS-Autosync is not yet packaged for most distributions, excepting Arch. Packages for Debian/Ubuntu are on the way but for now if you want to use DVCS-Autosync it requires grabbing the source off of Gitorious and installing the necessary dependencies. Users will need Python 2.6 or later (2.7 seems to work just fine), Pyinotify, and xmpppy (packaged as python-jabberbot on Ubuntu). Of course git or your favorite DVCS is also required. Because the utility uses Jabber/XMPP to communicate between clients, you'll also need a Jabber account that you can use for its notifications. The installation instructions on the DVCS-Autosync page seem to be a bit
outdated. There's no "autosync.py" included with the source, the
dvcs-autosync script is now what you're looking for. Also you can install
DVCS-Autosync using " The next step is to initialize a git repository on a central server, and to then clone that repository on each client that will be using DVCS-Autosync. Each client also requires a config file (included as .autosync-example in the source, under the main directory) that needs to be customized to add Jabber account information. After that, run dvcs-autosync and, if all is configured properly, it should just go. As long as dvcs-autosync is running, it will automatically commit files after they're added to the directory or changed, and synchronize files to the other clients after they've been checked in. Users do not need (or have the ability to) manually make commits with dvcs-autosync. Note that it will not operate quietly, however. The application is set up to display desktop notifications for each change. This is similar to Dropbox's default behavior — but seems utterly unnecessary past the first few hours when one might wish to ensure that the script is actually working. Mayrhofer says that the devel branch now allows users to configure notifications to go to the desktop, XMPP, both, or turn them off entirely. Users can recover older versions of files, or view commit logs, using the standard git utilities — but dvcs-autosync doesn't have any special tools for that, either. In short, it is currently a single-purpose utility that syncs files between machines. Looking ahead for DVCS-AutosyncDVCS-Autosync is an interesting alternative to Dropbox and SparkleShare, but it still has some room for improvement and a few missing features that may be problematic for some users. First and foremost, DVCS-Autosync is missing encryption. While Dropbox's encryption is significantly compromised by the fact that the company can decrypt its users files without the users' permission or knowledge, it does at least offer the feature. DVCS-Autosync, on the other hand, makes no provision for encryption at all. Mayrhofer says that he "definitely" plans to implement encryption at some point — but for now it's assumed that the repository is trusted. Mayrhofer is undecided how to implement encryption but may look at using git-annex, which has gained support for encrypted backends recently. Using git-annex might help with another weakness in the current implementation: if you're using DVCS-Autosync for syncing large files that change often, you may be looking at a fairly large repository after a while because it will store the entire file for each change. However, this is a known problem and DVCS-Autosync contributor Dieter Plaetinck has suggested using git-annex to handle large files as a way around the current problem. Moving directories gracefully, and providing a single commit entry for multiple file events (such as moving a directory, or uncompressing a tarball within the synced directory) are also on the list of bugs to handle soon. Longer term, Mayrhofer has indicated that he'd like to add context to commit messages (such as which applications are open at the time), and a way to specify commit messages via a traybar icon or popup rather than the general commit messages that are generated by DVCS-Autosync currently. In its current state, DVCS-Autosync has a few kinks to be worked out before it can reliably replace Dropbox for most users. However, there seems to be some strong interest in the application already, and it could quickly become reliable enough to become a staple for Linux users who just want a simple file synchronization tool that requires little attention. (Log in to post comments)
DVCS-autosync Posted May 13, 2011 18:50 UTC (Fri) by bfields (subscriber, #19510) [Link] Wouldn't it make more sense to base this on top of unison instead of a version control system?
DVCS-autosync Posted May 13, 2011 20:01 UTC (Fri) by iranzo (subscriber, #43448) [Link]
I was looking some alternatives to dropbox but on the same time allow multicomputer syncing in a reliable and automated way.
I'm still using unison on three computers to sync backups from station to server 1 or server 2.
I would really love some program to autosync between defined computers without the need of a central repository and with the ability to work offline, then sync between them, etc...
But that's gonna be a bit tricky :), would require some nat-transversal program for connecting computers between them like hamachi in order to sync and a good sync engine to deal with all changes that could happen on different computers and reconcile them...
DVCS-autosync Posted May 14, 2011 6:24 UTC (Sat) by ptman (subscriber, #57271) [Link]
If you only sync in one direction (mirror), rsync is usually a better choice than unison as it uses a (somewhat) standard protocol instead of requiring the exact same version of unison in both ends.
ShareBox Posted May 13, 2011 19:59 UTC (Fri) by dato (subscriber, #30206) [Link] Not sure what the status of this is / haven't tried it / etc.; but, quoting from Joey Hess' New git-annex use cases:
Unison! Posted May 13, 2011 20:38 UTC (Fri) by martin.langhoff (subscriber, #61417) [Link]
I have done this, for many years, across various OSs, with unison. Many file/folder sync tools came and went; MS desktops had various flawed incarnations of "my briefcase"; Novell had one that they pimped for very nice profit; but none never as good as Unison.
Some pretty UIs for unison exist, at least on Mac and Linux -- last I tried ~10 years ago it also worked on Windows but you had to get cygwin's ssh going.
Unison is the measure of file sync tools. Is this DVCS-autosync gadget as good as unison across the board? Better in some clear, significant way? Enlighten us!
Unison! Posted May 13, 2011 20:54 UTC (Fri) by hisdad (subscriber, #5375) [Link]
I tried to get unison to work for me.
According to the docs it would only work if all copies on all platforms were the same version.
Because I'm running different distros, -- including windows --, this never seemed possible.
If you have had success, I suppose this may be a documentation error.
--John
Unison! Posted May 13, 2011 21:06 UTC (Fri) by JohnMorris (subscriber, #73531) [Link]
I have the same problem with Unison, to the extent that I now keep the installation files for one particular old version around to use every time I get a new machine.
Unison works well for synchronizing between two machines, and that's how I use it, to sync work files between home and work desktops. With three or more computers, though, you have to be careful what order you synchronize between them, or else things get into a mess. This is a result of its batch oriented way of working - syncs are manually initiated - and the lack of a central master repository which everything else syncs to. To be fair to the Unison authors, they do suggest that to keep multiple computers synced up one of them should be designated as the master and all others should sync only to/from that one.
Dropbox - which I also use, to keep my personal files co-ordinated between four or five computers - on the other hand is much friendlier in this regard, as it records deltas immediately.
Unison! Posted May 15, 2011 0:11 UTC (Sun) by Velmont (guest, #46433) [Link]
It is easy to build something similar around this. Using inotify, or incron, you can wait for close-signal, and then run unison on just that path-tree automatically.
I do something like that, only a bit less advanced with my unison script. I really love Unison, and it blows the butt of Dropbox and has for my kind of use. :-)
I have many machines, all of them using unison-323, it's not hard to standardize on a version, although yes, this could be easier.
DVCS-autosync Posted May 13, 2011 21:45 UTC (Fri) by atai (subscriber, #10977) [Link]
Tahoe-LAFS? http://tahoe-lafs.org/trac/tahoe-lafs
DVCS-autosync Posted May 14, 2011 23:58 UTC (Sat) by davidsarah (guest, #69804) [Link]
The developers of Tahoe-LAFS are actively considering adding inotify-based synchronization, either via something like DVCS-Autosync or more directly using Pyinotify; see http://tahoe-lafs.org/pipermail/tahoe-dev/2011-May/006356...
DVCS-autosync Posted May 13, 2011 22:35 UTC (Fri) by martinfick (subscriber, #4455) [Link]
To avoid repositories growing too much when large files change a lot, why not simply rebase every now and then to a simpler/smaller history? That way you can at least select which points in history you no longer care about instead of having no history at all.
DVCS-autosync Posted May 13, 2011 22:46 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
rebasing doesn't solve the problem, you are wanting to actually trim the history (a 'shallow' clone in git terms, but then making that the real repository and throwing away it's history)
I think there is a command along the lines of git filter-history that can walk one repository and create a new one with less history than the old one had. This could be used to trim the old history, but when you do so, you will have to re-clone the (now smaller) repository to all your machines
DVCS-autosync Posted May 13, 2011 22:56 UTC (Fri) by martinfick (subscriber, #4455) [Link]
I think that you are assuming that you always keep alternate/old branches around. Why couldn't you simply rebase onto a new smaller branch and fetch that branch on all your copies, then drop the old master branch?
DVCS-autosync Posted May 13, 2011 22:59 UTC (Fri) by martinfick (subscriber, #4455) [Link]
Really, there are many ways with git to rewrite your history, and that is the main point that I was trying to make. And, surely rewriting your history every now and then to keep it reasonably sized is surely better than just having no history, isn't it?
DVCS-autosync Posted May 13, 2011 23:10 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
You also have to prune the remote repository. If the old blobs are in the packs, they will still get downloaded when cloned even if they're unreferenced. Same with pushing: blobs reachable from commits in the reflog is considered "referenced" and blobs will linger from that as well. To remove these blobs, a combination of "git repack", "git fsck --unreachable HEAD", and "git prune" would be necessary. I don't know of one tool/command that does a pruning of *all* blobs not reachable from a set of commits.
DVCS-autosync Posted May 14, 2011 9:17 UTC (Sat) by rmayr (subscriber, #16880) [Link]
[dvcs-autosync author here...]
That is what I will be trying to do for those files not managed by git-annex (not that git-annex support is done yet...). However, I don't know yet what the best way to rewrite the history will be and am happy to take any suggestions at this point. filter-branch is something I have used before for splitting repositories, but there may be better ways to reduce history. From a user point of view, the use cases that should be supported by the trimmed history are IMHO: a) "Give me the state of this directory at a point in time up to X days/weeks/months ago" and b) "I want this file as it was Y revisions ago". Everything that does not fit in either a) or b) with user-configured values for X and Y can be pruned.
DVCS-autosync Posted May 14, 2011 20:53 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
when trimming the history, I can see a couple of desired approaches
1. I want to keep all revisions up to X days/revisions ago, but I don't care much about anything before that.
2. I know that this is a good version, I want to make sure I can always see this version, but I don't care about all the other versions I haven't tagged.
3. I want to save one version per day/week/?? but not _every_ version (i.e. get me back into the approximate area, but don't spend the resources to save every version)
I can also see combinations of these
for example I want a snapshot each day, but every revision for the last week
given how efficient the git compression can be, I would suggest having a dry-run mode that will tell you how much space you are saving by throwing away the history. for binary files it may be quite a bit, but for other files you may be surprised at how little it saves.
DVCS-autosync Posted May 17, 2011 21:15 UTC (Tue) by zooko (subscriber, #2589) [Link]
(Tahoe-LAFS hacker here)
There are hooks to integrate git-annex with Tahoe-LAFS, and that might make it easier to do the sort of pruning you are talking about here.
http://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects
Also, send me email at zooko@zooko.com and I'll hook you up with a Tahoe-LAFS storage service that you can use for testing.
DVCS-autosync Posted May 13, 2011 23:02 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
switching your syncing from one branch to another would seem to be at least as problematic as re-generating your master branch.
in addition, the history you want to keep around is usually the more recent stuff, rebasing the latest version onto old history seems to be loosing the most valuable revisions.
DVCS-autosync Posted May 13, 2011 23:21 UTC (Fri) by martinfick (subscriber, #4455) [Link]
> switching your syncing from one branch to another would seem to be at least as problematic as re-generating your master branch.
I don't know how DVCS-autosync uses git, but there is nothing magical about "master" in git, in fact I rarely even care about it. All branches are equal with git, and all repos are equal. There is no need to even clone repos, simply init a new repo and fetch into it the branch you care about. git is a DVCS, it is not subversion.
> in addition, the history you want to keep around is usually the more recent stuff, rebasing the latest version onto old history seems to be loosing the most valuable revisions.
I fail to see how that is an argument against keeping history when the alternative, no history, surely is no better than the wrong history, is it?
But, who said anything about specifically cleaning up old history, clean up any history you want. Go ahead, clean it up to only one checkin, only the latest if you want. You will still be no worse off than having only one copy and you will have the ability to add history whenever you want.
DVCS-autosync Posted May 14, 2011 9:21 UTC (Sat) by rmayr (subscriber, #16880) [Link]
> I don't know how DVCS-autosync uses git, but there is nothing magical about "master" in git, in fact I rarely even care about it. All branches are equal with git, and all repos are equal. There is no need to even clone repos, simply init a new repo and fetch into it the branch you care about. git is a DVCS, it is not subversion.
dvcs-autosync doesn't care about it at all. Whatever the checked out branch is, it will work on. Dieter and I have already been discussing that explicit support for branches may be good to have in future versions for accessing (and possibly modifying) older revisions. But right now, it's completely transparent. You can even have two clones of the repository (in different directories) pointing to different branches and have two instances of dvcs-autosync working on those branches concurrently.
DVCS-autosync Posted May 13, 2011 23:16 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
I'm still looking for a way to handle data file synchronization. It appears that this might by the tool I use in the future, but at the moment, it doesn't quite do what I need. The problem is that it's just a few files spread around my home directory. My configuration files are handled via git (this tool wouldn't work because I have, e.g., branches for work-related changes at work). What I need is a tool to sync my newsbeuter database, bookmarks, and the like that don't logically belong in a configuration repository.
For configurations, one major problem that I've come across is with applications that don't behave correctly with a configuration file that is a symlink into the repository so I can see changes and commit them appropriately. My current list of apps that break this are: tin (overwrites the configuration file with a mv when exiting; even if no changes are made; also writes things like "last search string" to the file which don't belong), finch (probably similar; haven't investigated it much), and weechat (I'd use it over irssi if it didn't do this). esmtp is similar in that it refuses to work if $HOME/.esmtprc is a symlink which is at least understandable, but still less than ideal.
For data files, the tool has to support a list of files to sync (symlinking these will probably work even less well than configuration files) rather than just syncing a directory. It appears as though DVCS-autosync doesn't support this (yet) and nothing I've found does this to my satisfaction either.
The automatic committing is also something that I'm personally wary of (newsbeuter would create commits on the 8MB sqlite db all the time) and git-annex would be the way to go for it, preferably waiting until newsbeuter is closed to make the commit.
DVCS-autosync Posted May 14, 2011 0:29 UTC (Sat) by shredwheat (guest, #4188) [Link]
I saw Wuala recently and want to check it out. If anyone has experimented with it, I'd like to hear some analysis. It comes with several of the drawbacks of Dropbox, but seems to provide 'better' everything?
http://www.wuala.com
DVCS-autosync Posted May 14, 2011 0:48 UTC (Sat) by mathstuf (subscriber, #69389) [Link]
Just looking through the feature list, the encryption is nice as is the "trading" idea, but the drawbacks of Dropbox (namely "...add a file to your Wuala Sync folder...") prevents it from meeting my requirements. A way to do profiles would be nice as well (the MLs I subscribe to at work in tin are not the same as those I subscribe to at home, so syncing that depending on the machine's use would be ideal, but not a hard requirement).
DVCS-autosync Posted May 14, 2011 5:53 UTC (Sat) by ewen (subscriber, #4772) [Link] FWIW, with some tools I always run them via a wrapper (often named the same, but earlier in the search path, and explicitly calling the real one via a full pathname), so that it can do various tidying up before/after. Eg, my cash tracking application is run from a wrapper that auto-commits changes into git, if there are any, and I used to run trn via a similar wrapper. You could probably do the same thing with tin: copy the state file out of a repository somewhere to the name it expects, run tin, let it overwrite the file, diff the resulting file against the repository, and auto-commit/push if it has changed. Aside from the last-write-wins-whole-file issue (or whatever your auto-resolve-conflicts policy becomes), it seems like it should automagically do what you want. Ewen PS: You're still left with problems of state stuck inside always-running applications that only write state on exit, but that's baked in to the applications in question.
DVCS-autosync Posted May 14, 2011 12:12 UTC (Sat) by juliank (subscriber, #45896) [Link]
> esmtp is similar in that it refuses to work
> if $HOME/.esmtprc is a symlink which is at > least understandable, but still less than ideal.
At least it works for /etc/esmtprc, I use that on my system to keep the esmtprc with password in it in an encrypted mount point.
jak@jak-thinkpad:~$ ls -l /etc/esmtprc
DVCS-autosync Posted May 15, 2011 23:22 UTC (Sun) by nicooo (guest, #69134) [Link] How about using hard links for those files?
DVCS-autosync Posted May 16, 2011 15:37 UTC (Mon) by mathstuf (subscriber, #69389) [Link]
Hmm...somehow missed that option. The rationale behind symlinks is that "uninstallation" of an application's configuration is easier (just clean up broken links in $HOME/{.*,bin}). Using hardlinks would solve the "keeping things in sync" problem, but still be less-than-ideal (false positives would be high when detecting "uninstalled" applications).
symlinks, immutability, wrappers, hardlinks, mount-bind Posted May 27, 2011 4:56 UTC (Fri) by Duncan (guest, #6647) [Link]
At least four independent techniques exist for keeping files in two different trees in sync, three of which have already been mentioned in other replies, but it's worth collecting them into one comment, and I didn't see the fourth mentioned at all, yet.
Symlinks: When I switched from MS dark side, symlinks quickly became my favorite *ix filesystem feature. =:^) Unfortunately, by themselves, they have one significant already mentioned limitation: apps that write a temp file and then mv it over the old permanent file, possibly mv-ing the old version to backup at the same time. (This is A good technique on its own as it better ensures integrity of either the file or at last resort its backup, in case of a crash at the wrong moment, but as implemented by many apps, it's not compatible with symlinking.)
In some cases, it might even be possible to reverse symlink, putting the symlinks in the sync-tree, if it's possible to tell whatever's syncing them (git by default, here, I don't know git well enough to know if it can handle that or not) to dereference symlinks instead of syncing the symlink itself.
Ext2/3/4 users at least might also be able to work around the rewriting problem by setting the immutable bit on the file, particularly in the mentioned scenario where an app is overwriting the file even when nothing changed. For filesystems that don't implement immutable, filesystem permissions can be used, setting the file, or its directory if setting only the file doesn't work, read-only, and if necessary, changing the ownership (assuming one has the privs to do so, of course). I didn't see these mentioned as options, yet.
Wrapper scripts: Someone mentioned application wrapper scripts as another option. Create a script by the same name and set it earlier in the path, have it copy the file in question out of the synced tree, then launch the desired app via absolute path name. The wrapper then waits around until the desired app exits, after which it copies the changed file back to the synced tree. (If the user doesn't want to save the changes, the app launch can use exec or similar to avoid having the wrapper stick around for the main app to finish.) Note that GUI launcher users may need to customize their GUI launcher to point at the wrapper, as it may already be using the absolute path or may not be using the same path ordering. FWIW I suspect many LWN users may already be using the wrapper technique for other purposes but may not have thought of using it as a solution for this problem.
Hardlinks: Another suggested solution is hardlinks. That can work for some things, particularly in combination with the immutable/read-only technique above, but hard links have their own negatives. A major one discouraging their use here is that hardlinks are same-filesystem-only, thus limiting one's filesystem management flexibility. A second negative is that files with multiple hard links simply aren't particularly intuitive to those who started on the MS side of things, another weakness of at least the FAT filesystems (I'm unsure about NTFS), where such linking tended to be the result of filesystem corruption and wasn't supported as a feature.
Mount-bind: Not yet mentioned, however, is the very useful mount --bind option, allowing one part of a filesystem, even an individual file, to be mounted elsewhere in the tree as well. =:^) This is a relatively new addition to the toolbox, and thus relatively unknown as it hasn't yet really entered the bank of common wisdom out there yet, as the Linux kernel and the mount command from the util-linux package haven't supported this for all that long. But at least for users with suitable mount privileges available, it is certainly one of the most powerful options, since it not only allows a user to make a directory or file existing in one location in the tree appear elsewhere either in addition or instead of the original location, but ALSO comes with all thue usual mount permissions restrictions ability, including making the mount read-only or read/write, nodev/noexec/nosuid, etc. Mount-binding a file as read-only in effect makes it immutable as addressed from that location, but without the usual filesystem-must-support-immutable limitations, as it's enforced by the kernel mount infrastruction instead of the filesystem. And while it makes little sense in the context of an individual file, directory/subtree level noexec/nodev/nosuid certainly has its uses, as well. Mount-bind functions in this way like an admin-level symlink, and I'm appreciating its benefits more and more as I use it, much as I did symlinks early on, when I first switched from the dark side.
Certainly there's no shortage of choices for keeping a sync-tree file in sync with a file elsewhere in the tree. Given all the choices available, /one/ of them should be workable.
Duncan
DVCS-autosync Posted May 31, 2011 20:42 UTC (Tue) by Jan_Zerebecki (guest, #70319) [Link]
One can just keep the whole home directory in git to avoid needing symlinks or something similar. By ignoring * and then negating specific ignores like !.bashrc one gets the same selectiveness. Negated ignores are documented in man gitignore.
DVCS-autosync Posted May 14, 2011 6:07 UTC (Sat) by drag (subscriber, #31333) [Link]
I don't understand why people find it attractive to use source code management systems to build syncing file systems...
The vast majority of data, in terms of volume, is going to be binary data only. And a significant number of popular formats use compression for various reasons.
Are not things like Git really inefficient at handling binary data?
It seems to me that it would be far more efficient to handle things using a block-like storage mechanism..
DVCS-autosync Posted May 14, 2011 9:44 UTC (Sat) by rmayr (subscriber, #16880) [Link]
My personal use case is a tree of most of my work so far, including source code and documents. For these, I explicitly want the history (thinking of papers, theses, project proposals, reports, etc.) and have relied upon tracking the changes in quite a few instances. However, I try to keep large binary blobs outside this tree (that might change with decent git-annex integration, although I don't know (yet) how best to do that).
DVCS-autosync Posted May 14, 2011 12:50 UTC (Sat) by loevborg (guest, #51779) [Link]
I agree completely. DVCS are useful for plain text, which for almost anyone except programmers is only a tiny portion of your data. Using git doesn't gain you anything, because we can't use diff for ODF etc. What we need is pretty much a drop-in replacement for dropbox.
With all due respect, this seems the expression of a sort of narrow-mindedness typical of programmers. People who work with source code all day assume that code is the model of computer work. However, code is actually very particular. What we really need is a place to store our letters, PDF forms, spreadsheets and so on, which are quite different from code in many respects.
The situation is similar in the field of text editors. Here every week or so someone starts a new editor which targets ... programmers. But there still is hardly a single text editor which is any good for writing simple English language prose! Here there is little use for syntax coloring, line numbers, nonproportional fonts. Instead you need configurable line spacing, easy-to-read proportional fonts, paragraph-based (not line-based) navigation, word-wrapping, etc. Still instead of solving this very real problem, people come up with new solutions to already solved problems.
DVCS-autosync Posted May 14, 2011 13:08 UTC (Sat) by nix (subscriber, #2304) [Link]
Actually, you can plug in domain-specific diff/merge tools into git (OpenOffice documents were a named use case for this). Obviously the merge tools for binary stuff need their own way to do conflict resolution as well.
Git is exactly as efficient at handling binary files as handling text files: you have to go back to CVS to find something that isn't good at binary files.
DVCS-autosync Posted May 19, 2011 5:07 UTC (Thu) by smurf (subscriber, #17840) [Link]
There are a couple of document formats which are singularly not suited for VCSes, though.
OOo documents, for instance, are compressed XML files. There's no sane way to store multiple versions of these in a git archive. Store them uncompressed (dunno how to teach OOo that) and you get a signficant decrease in storage requirements, long-term.
Still needs a domain specific conflict rresolver, of course. You could probably script your way into LibreOffice to do it, though it's nontrivial.
DVCS-autosync Posted May 19, 2011 5:20 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
actually, since git does the compression itself, the answer is to have git uncompress the documents and version the uncompressed data. then when you check it out git assembles the version you want, then compresses it as you check it out.
you can even insert XML aware diff engines if you want.
DVCS-autosync Posted May 26, 2011 19:43 UTC (Thu) by nix (subscriber, #2304) [Link]
Yes indeed. 'man gitattributes' and search for 'filter' for something that may help here.
DVCS-autosync Posted May 14, 2011 19:40 UTC (Sat) by elanthis (guest, #6227) [Link]
You mean... people who _write code_ think like people who _write code_?
How surprising.
DVCS-autosync Posted May 14, 2011 20:45 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
it's not that git is especially inefficient at handling binary files, it's that the compression and other efficiencies that git can get with text files don't work on binary files.
but git is not any worse in dealing with binary files than any other solution where you want to be able to retrieve any version that ever existed.
that said, git does have some limitations in terms of max sizes of things
DVCS-autosync Posted May 14, 2011 21:45 UTC (Sat) by drag (subscriber, #31333) [Link]
So when I want to share my 800MB movie with a friend or have it available on another machine it's just not going to happen.
From dropbox's website:
> Files uploaded through the website (by pressing the upload button) have a 300 MB cap. In other words, each file you upload through the website must be 300 MB or less.
> All files uploaded to your Dropbox must be smaller than the size of your Dropbox account's storage quota. For example, if you have a free 2 GB account, you can upload one 2 GB file or many files that all add up to 2 GB. If you are over your storage quota, Dropbox will stop syncing until you are below your limit.
Dropbox's revision control system is optional and only will save revisions for 30 days. Many people want sync software to sync a significant amount of data.... I am guessing that for most people's purposes revision control is much less important then just automatic syncing.
The ability to carelessly drop a file into your drop box and have it automatically available on any machine you happen to want to use is the 'killer feature' for Dropbox. The idea of trying to do something like manage a 4GB mp3 collection using something like Git commit sounds like a nightmare to me.
DVCS-autosync Posted May 14, 2011 20:47 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
people are attracted to DVCS systems because they already address a lot of the problems that can come up when you have more than two locations that you are trying to sync.
why reinvent the wheel when you can reuse the work that someone else has done?
DVCS-autosync Posted May 15, 2011 21:38 UTC (Sun) by dan_a (subscriber, #5325) [Link]
Git in particular has one killer feature here - its built in checksumming which ensures that any copies of files are uncorrupted.
DVCS-autosync Posted May 15, 2011 23:47 UTC (Sun) by drag (subscriber, #31333) [Link]
Yeah I love git for a variety of reasons. I just worry that there is a impedance mismatch between it and the goal of 'Dropbox replacement'.
Git is very carefully optimized to provide a high performance revision control system for text files.
However typical Dropbox usage only deals with a tiny amount of text data and revision control is borderline irrelevant for most people's uses.
I just remember using Git for a variety of purposes and realizing 'Hey putting that ISO image for the cdrom I made into my repository was a very very stupid thing to do'... yet people are going to want to store ISOs, mp3s, zip files, and huge raw-formatted camera images and other things of that nature in anything calling itself a 'A open source Dropbox replacement'
DVCS-autosync Posted May 16, 2011 10:53 UTC (Mon) by dgm (subscriber, #49227) [Link]
"Git is very carefully optimized to provide a high performance revision control system for text files."
You were almost right up until "revision control system for text files". Linus described Git more in the line of an information tracker, or content addressable filesystem, used to implement a DVCS that accidentally bears the same name.
DVCS-autosync Posted May 16, 2011 11:00 UTC (Mon) by drag (subscriber, #31333) [Link]
Well that's confusing then.
So Git is the backend for Git DVCS...
DVCS-autosync Posted May 16, 2011 20:58 UTC (Mon) by njs (guest, #40338) [Link]
So what's the difference? It's an "information tracker or content-addressable filesystem" that is heavily optimized on the assumptions that the information/content its holding is highly compressible (by deltas and by gzip), which makes it okay to have two copies of everything (one in the repo and one in the filesystem), and also (last I checked) that it can hold any individual file in RAM. Those things are totally reasonable assumptions for source code, and terrible assumptions for a directory full of Blu-Ray rips. (And those are just the two issues that occur off the top of my head.)
Linus' point is that git is designed with good decoupled interfaces between its internal components, not that it's always going to be good at solving problems that it wasn't designed for.
DVCS-autosync Posted May 16, 2011 21:12 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
you have a valid point in terms of being able to mmap things (which is not quite the same as being able to fit them in memory). this is a limitation on 32 bit systems
however, how can you do version control if you don't keep a copy of the file somewhere else? if someone changes it, how can you get back what was there before without another copy?
DVCS-autosync Posted May 16, 2011 21:34 UTC (Mon) by njs (guest, #40338) [Link]
Oh, does it use mmap exclusively for all file access now? Including repacking and everything? Neat.
You need two local copies if you want to do local version control, and also to let people edit files normally on disk (as opposed to, say, interposing a FUSE filesystem to observe edits as they happen). But the systems we're talking about are not trying to do local version control. They're trying to do remote backup and syncing!
*For this use-case*, you might almost be better off with CVS than with git. Its handling of binary files is dumb, but at least it wouldn't double your local storage requirements. Even better, of course, would be a system that stored the second copy on the remote server only, and then used something clever like rsync to upload the deltas.
Or maybe one could do something clever with libgit2 and librsync to let you directly and efficiently commit a local set of files to a remote bare repository...
DVCS-autosync Posted May 16, 2011 21:53 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
I don't know that it always uses mmap, but it's the use of mmap that imposes the file size limit on individual files.
pack files are limited in that they use a 32 bit offset into them, but that's a matter of optimisation for files that can be diffed and compressed.
yes, for this use case you may be better of with CVS, but only until you have to reconcile differences between different locations. DVCS tools give you the framework (and many of the mechanisms) for doing this as part of their heritage
DVCS-autosync Posted May 16, 2011 22:11 UTC (Mon) by njs (guest, #40338) [Link]
> yes, for this use case you may be better of with CVS, but only until you have to reconcile differences between different locations. DVCS tools give you the framework (and many of the mechanisms) for doing this as part of their heritage
Yes, of course. (Though in practice I'm not sure git's current merge mechanisms are well-optimized for the collection-of-large-binary-files case either.)
But that doesn't change the point, which is that git is not a perfect match for this problem, and a better tool that was similar to git in some ways but not in others could potentially do substantially better.
DVCS-autosync Posted May 16, 2011 22:16 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
I agree that git does not do anything that useful for large binary files, and if that is all you are wanting to sync, then you are better looking at modifying git rather than using it directly.
but this depends in large part on what these binary files are. git supports configurable diff/merge engines, so if there is any sane way to merge your 'binary' files, git will allow you to use it.
please don't get me wrong, I'm not saying that git is perfect, just that it does a better job than anything else for the general case and brings a lot to the party. This makes basing a tool on git (or one of the other DVCS systems if you dislike git for some reason) a very reasonable thing to do rather than just developing your app from scratch.
DVCS-autosync Posted May 17, 2011 14:45 UTC (Tue) by nye (guest, #51576) [Link]
You appear to be describing bup: https://github.com/apenwarr/bup
DVCS-autosync Posted May 14, 2011 9:54 UTC (Sat) by nteon (subscriber, #53899) [Link]
I stumbled across this earlier, which seems similar: http://camlistore.org/
DVCS-autosync Posted May 15, 2011 19:49 UTC (Sun) by sailorxyz (subscriber, #52650) [Link]
A good commercial alternative to Dropbox is SpiderOak https://spideroak.com/ It's not quite as simple as Dropbox but uses proper encryption and only you have the keys. SpiderOak cannot access your files. It's also cheaper than Dropbox. They also have a free 2GB account. Works across Linux Mac and MS platforms.
DVCS-autosync Posted May 16, 2011 0:18 UTC (Mon) by madscientist (subscriber, #16861) [Link]
I've been using SpiderOak since last year and I think it's great. Very easy to use, and as mentioned it works on my Linux systems, my wife's Mac, and our kids' Windows systems. I like the encryption model where my content is encrypted locally using my local passphrase and only the already-encrypted content is stored remotely. The backup utility is quite flexible and doesn't require that you collect all your files to be backed up in one place.
And it's 2G free or $100/yr for 100G of storage which is quite reasonable, IMO.
ObDisclaimer: I don't work for or have any financial interest in SpiderOak.
DVCS-autosync Posted May 16, 2011 16:10 UTC (Mon) by Cato (subscriber, #7643) [Link]
I found SpiderOak didn't work at all well, though that was two years back. I was using it mostly for online backups, and found it would never complete syncing, and on one machine with only a few GB to back up, it didn't back up most of the files. It also generated a corrupt ZIP file when trying to recover the data. I found the UI very complex given what it's trying to achieve is quite simple.
It may have improved, but I'd encourage very careful testing at least.
DVCS-autosync Posted May 15, 2011 23:42 UTC (Sun) by bradfitz (subscriber, #4378) [Link]
Yes, Camlistore is rapidly becoming a reasonable Dropbox replacement. Not quite yet, but I'm working on it every free second I have.
DVCS-autosync Posted May 16, 2011 5:44 UTC (Mon) by yohahn (subscriber, #4107) [Link]
This looks a bit out of date (last version 2009), but I've wondered about boxbackup http://www.boxbackup.org/ as a DIY dropbox
fsniper Posted May 19, 2011 22:13 UTC (Thu) by cdmiller (subscriber, #2813) [Link]
Fsniper works nicely for DIY dropbox scripts. It can trigger version control systems, rsync, etc..
|
|||||||||||||