Large File Storage

Posted Apr 9, 2015 7:06 UTC (Thu) by Felix.Braun (guest, #3032)
Parent article: What's coming in Git 2.4.0

It would have been interesting how git LFS differs from the venerable git-annex.

Large File Storage

Posted Apr 9, 2015 7:58 UTC (Thu) by fishface60 (subscriber, #88700) [Link] (1 responses)

Or indeed git-fat, or any of the other attempts to support large data files in a git repository.

Large File Storage

Posted Apr 9, 2015 8:04 UTC (Thu) by fishface60 (subscriber, #88700) [Link]

Having had a look at how it works, it appears to use exactly the same mechanism as git-fat, where it uses a smudge filter to translate files as they go between the index to the committed tree object.

I'm just concerned that the result will be yet another weird proprietary extension. git-fat at least used the wide-spread rsync daemon as the backend for file storage.

Large File Storage

Posted Apr 9, 2015 13:05 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (5 responses)

Well, git-annex uses symlinks into a repo and broken links indicate files are missing. This obviously doesn't work on Windows or other broken file systems, so instead, files are empty if they are not local in those cases.

One nice thing about git-annex is that I am not required to have the data locally (I use this to rotate out photos from my mobile devices onto my main server while having a few months' worth of photos locally). It also has a semblance of where the data is and can fetch it from any of those places (if it is still available at that time). This is also what allows it to do the "keep X MB local" logic since it can safely delete data it knows is on the server. It would appear to me that git-lfs requires the data to always be available since it uses a smudge filter.

Large File Storage

Posted Apr 27, 2015 15:53 UTC (Mon) by nye (subscriber, #51576) [Link] (4 responses)

>Well, git-annex uses symlinks into a repo and broken links indicate files are missing. This obviously doesn't work on Windows

Why not?

Large File Storage

Posted Apr 27, 2015 16:00 UTC (Mon) by nye (subscriber, #51576) [Link] (3 responses)

>>Well, git-annex uses symlinks into a repo and broken links indicate files are missing. This obviously doesn't work on Windows
>Why not?

Oh wait, I think I've worked it out: It appears that, by default, only administrators can create symlinks on Windows for some crazy reason. This would therefore require either running as an administrator or changing the system security policy, neither of which make for an enjoyable user experience.

Large File Storage

Posted Apr 27, 2015 20:27 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

Also only supported on NTFS (and exFAT?). My Android phone has the "brokenfs" option set and also has empty files sitting there, so maybe exFAT also doesn't support them.

Basically it just means that the Gallery app just shows lots of empty boxes after I sync.

Large File Storage

Posted Apr 28, 2015 9:59 UTC (Tue) by nye (subscriber, #51576) [Link]

>Also only supported on NTFS

Well I wouldn't say that's really a problem - I think NTFS is the only fs Windows can be installed on.

But anyway, surely the use of symlinks is really just an implementation detail that could be replaced with fake symlinks (Cygwin style, or whatever) easily enough if there was a will? I guess there must be bigger (design/conceptual) differences between git-annex and LFS.

The LFS web site certainly looks swanky; it must be good software.

Large File Storage

Posted Apr 27, 2015 20:28 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

As for the administrator, I think it's their way to avoid symlink attacks since ~no applications are prepared for such things.