Large files with Git: LFS and git-annex

Posted Dec 11, 2018 20:28 UTC (Tue) by anarcat (subscriber, #66354)
Parent article: Large files with Git: LFS and git-annex

as usual, the bug reports and feature requests I opened while writing this article:

git-annex: LFS API support
git-annex: "why are all those files modified" found while testing v7 mode
git-annex: clarify that v7 applies to all clones
git-annex: v7 fails to fetch files on FAT filesystem

For the sake of transparency, I should also mention that I am a long time git-annex user and even contributor, as my name sits in the thanks page under the heading "code and other bits" section, which means I probably contributed some code to the project. I can't remember now what code exactly I contributed, but I certainly contributed to the documentation. That, in turn, may bias my point of view in favor of git-annex even though I tried to be as neutral as possible in my review of both projects, both of which I use on a regular basis, as I hinted in the article.

Large files with Git: LFS and git-annex

Posted Dec 11, 2018 20:45 UTC (Tue) by warrax (subscriber, #103205) [Link] (3 responses)

I really *wanted* to like git-annex and use, but the lack of tutorial material (at the time, possibly different now) about how to would around NATs and things of that ilk really hampered me.

That and... some software just doesn't want to work sensibly with symlinks, unforunately :(.

In the end I just chose unison for a star-topology sync (which it looks like git-annex effectively requires if you're being a NAT). Works equally well with large and small files, but obviously not really *versioned* per se.

Large files with Git: LFS and git-annex

Posted Dec 11, 2018 20:49 UTC (Tue) by warrax (subscriber, #103205) [Link]

Sorry for the absolute mess I made of the spelling in that.

*and use it

*how to would => how to work

I can only apologize.

problems symlinks and p2p: might be worth looking into git-annex again

Posted Dec 11, 2018 21:15 UTC (Tue) by anarcat (subscriber, #66354) [Link] (1 responses)

I've been thoroughly impressed by the new v6/v7 "unlocked files" mode. I only brushed over it in the article, but it's a radical change in the way git-annex manages files. It makes things *much* easier with regards to interoperability with other software: they can just modify files and then the operator commits the files normally with git. While there are still a few rough edges in the implementation, the idea is there and makes the entire thing actually workable on USB keys and so on. So you may want to reconsider from that aspect.

I find the p2p implementation to be a little too complex to my taste, but it's there: it uses magic-wormhole and Tor to connect peers across NAT. And from there you can create whatever topology you want. I would rather seen a wormhole-only implementation, honestly, but maybe would have been less of a match for g-a...

Anyways, long story short: if you ever looked at git-annex in the past and found it weird, well, it might be soon time to take a look again. It's still weird in some places (it's haskell after all :p) and it's a complex piece of software, but I generally find that I can do everything I need with it. I am hoping to write a followup article about more in-depth git-annex use cases, specifically about archival and file synchronisation soon (but probably after the new year)... I just had to get this specific article out first so that I don't get a "but what about LFS" blanket response to that other article.

problems symlinks and p2p: might be worth looking into git-annex again

Posted Dec 11, 2018 22:37 UTC (Tue) by warrax (subscriber, #103205) [Link]

I think I might try it again. Thanks for the "update", so to speak.