Re: resumable git clone [LWN.net]

Re: resumable git clone

Posted Jan 28, 2016 16:51 UTC (Thu) by viro (subscriber, #7872)
In reply to: Re: resumable git clone by viro
Parent article: A change of maintainership for Mercurial

FWIW, it might make sense to teach git-{fetch,upload,send,receive}-pack to optionally ask the sending side send the list of SHA1 it's going to send, so that recepient could say "don't send <this subset>, I already have those" or "send only <this subset>, I already have everything else".

In situations where recepient has alternates or scattered objects from an aborted fetch it has a potential to cut down on the amount of traffic quite a bit. And it shouldn't affect the tricky parts of calculating the set - it would be done _after_ the set is found.

So in the situation of git push into a repository that has most of that push in alternates, but doesn't have them covered by any references (which is very common for kernel.org trees) it would still calculate the same huge pile, send 20*(number of objects from the last merge window) bytes, get back "no, I'm only interested in these 50 objects" and proceed to send just those. It's easily a couple of orders of magnitude reduction of traffic.

And it case of e.g. aborted git fetch we would see a bunch of scattered objects (the loss from broken delta chains will be a small fraction), have the remote send the same (20 * number of objects) bytes (~90Mb for pulling Linus' tree into an empty one), then respond to it with up to half of that, in form of either "send me those" or "send me all but those" and proceed to fetch the rest. And that - on a fairly large repository...

to post comments

Re: resumable git clone

Posted Jan 31, 2016 13:17 UTC (Sun) by kleptog (subscriber, #1183) [Link]

ISTM an easier and possibly more backward compatible way to do this is to have receive-pack not just send a list of the refs it has, but also a list of hashes that are older that the sender can use for matching. For example for each ref something like ref~10, ref~20, ref~40, etc.. Or parents of recent merges, recent common ancestors, entries from the reflog, etc... I'm sure someone can think of something. Unless that would bloat the data for the common case. Perhaps a "receive-pack --try-harder".

I think the problem with partial pack files is due to git storing the current revisions as full files and storing old revisions as diffs w.r.t. them. Which makes it hard to make sense of a partial pack file because you actually need the current revisions to make sense of the old ones.

You could make it more reliable by, if send-pack would notice it is sending 1000 revisions, to split its job internally into chucks of 100 revisions, making it more likely a partial pack will be somewhat useful. In the end you're sending the same data, just in a different order, though there is some inefficiency.