Re: resumable git clone [LWN.net]

Re: resumable git clone

Posted Jan 28, 2016 10:38 UTC (Thu) by jnareb (subscriber, #46500)
In reply to: A change of maintainership for Mercurial by viro
Parent article: A change of maintainership for Mercurial

> Out of curiosity - what's the problem with salvaging already downloaded on an interrupted fetch? [...]

The problem is the following (from what I understand; I didn't try to implement such protocol myself).

First, with "smart" protocols client and server negotiate what server has and client wants, and what revisions client and server have in common, to send minimal number of objects (well, as minimal as possible with constraints they work under, like time spent on finding minimal set). This means that between starting an interrupted fetch and re-starting the fetch, both client and server might been updated with new revisions (or some revisions might be "deleted").

Second, server creates individualized packfile on the fly, with the data (revisions) client needs. Packing is multi-threaded for performance reasons, and is driven by heuristics... and that means that even if the set of objects is the same, packfile might be not byte for byte identical. That's I think the major stumbling point - IIRC any attempts of making packing predictable slowed the operation down; also, ultimately failed.

HTH

BTW. there is something called GitTorrent there...

to post comments

Re: resumable git clone

Posted Jan 28, 2016 11:57 UTC (Thu) by johill (subscriber, #25196) [Link] (1 responses)

Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

I don't immediately see where that breaks down, even looking at the packfile format.

Re: resumable git clone

Posted Jan 28, 2016 12:45 UTC (Thu) by jnareb (subscriber, #46500) [Link]

> Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

The problem might be in that the client-server negotiation very much depends on the fact that the client has all reachable objects (this assumptions tremendously reduces numbers of "have" client has to send), but in packfiles it is usually more recent first (because this way it packs better). Also, objects are sent as deltas; if you have delta chain without base object, it is useless.

So I guess that it would be hard work to extract a bit of history from partial packfile, for little benefit - the amount of history that one can extract from interrupted pack is, I guess, rather small.

Re: resumable git clone

Posted Jan 28, 2016 13:19 UTC (Thu) by viro (subscriber, #7872) [Link] (2 responses)

So don't try to get it to send the rest of packfile; you want a smaller packfile anyway, with the rest of the *objects* in it. The tricky part is to convince the damn thing that you do have such-and-such objects already; AFAICS, normally it goes by _references_, not by actual availability of objects. Which is a bloody annoyance in its own right - the first push after the merge window goes like this:

Linus' repository has everything up to v4.5-rc1
kernel.org vfs.git has it in alternates and its refs cover everything up to v4.4
local mirror of Linus' repository has everything up to v4.5-rc1
local vfs.git has that in its alternates and remotes and kernel.org vfs.git in remotes

git checkout for-linus
git merge origin
<edit, test, commit, etc.>
git push vfs for-linus

... and settle down for quite a bit of waiting, since it will push *everything* in range from v4.4 to v4.5-rc1 into kernel.org vfs.git, even though the damn thing _has_ all those objects available via alternates. Back when kernel.org trees came with shell access, ssh over there and git remote update origin from vfs.git prior to the push used to reduce the traffic by several orders of magnitude; no such thing these days...

Re: resumable git clone

Posted Jan 28, 2016 16:51 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

FWIW, it might make sense to teach git-{fetch,upload,send,receive}-pack to optionally ask the sending side send the list of SHA1 it's going to send, so that recepient could say "don't send <this subset>, I already have those" or "send only <this subset>, I already have everything else".

In situations where recepient has alternates or scattered objects from an aborted fetch it has a potential to cut down on the amount of traffic quite a bit. And it shouldn't affect the tricky parts of calculating the set - it would be done _after_ the set is found.

So in the situation of git push into a repository that has most of that push in alternates, but doesn't have them covered by any references (which is very common for kernel.org trees) it would still calculate the same huge pile, send 20*(number of objects from the last merge window) bytes, get back "no, I'm only interested in these 50 objects" and proceed to send just those. It's easily a couple of orders of magnitude reduction of traffic.

And it case of e.g. aborted git fetch we would see a bunch of scattered objects (the loss from broken delta chains will be a small fraction), have the remote send the same (20 * number of objects) bytes (~90Mb for pulling Linus' tree into an empty one), then respond to it with up to half of that, in form of either "send me those" or "send me all but those" and proceed to fetch the rest. And that - on a fairly large repository...

Re: resumable git clone

Posted Jan 31, 2016 13:17 UTC (Sun) by kleptog (subscriber, #1183) [Link]

ISTM an easier and possibly more backward compatible way to do this is to have receive-pack not just send a list of the refs it has, but also a list of hashes that are older that the sender can use for matching. For example for each ref something like ref~10, ref~20, ref~40, etc.. Or parents of recent merges, recent common ancestors, entries from the reflog, etc... I'm sure someone can think of something. Unless that would bloat the data for the common case. Perhaps a "receive-pack --try-harder".

I think the problem with partial pack files is due to git storing the current revisions as full files and storing old revisions as diffs w.r.t. them. Which makes it hard to make sense of a partial pack file because you actually need the current revisions to make sense of the old ones.

You could make it more reliable by, if send-pack would notice it is sending 1000 revisions, to split its job internally into chucks of 100 revisions, making it more likely a partial pack will be somewhat useful. In the end you're sending the same data, just in a different order, though there is some inefficiency.