A change of maintainership for Mercurial [LWN.net]

A change of maintainership for Mercurial

Posted Jan 26, 2016 19:02 UTC (Tue) by error27 (guest, #8346)
In reply to: A change of maintainership for Mercurial by pinkpony
Parent article: A change of maintainership for Mercurial

The facebook folks wrote a compelling article why they use Mercurial instead of git.

https://code.facebook.com/posts/218678814984400/scaling-m...

For me, git clone is really annoying because I have crap internet. It seems like it should be fixable to make git clone work over unreliable internet but it hasn't happened yet...

to post comments

A change of maintainership for Mercurial

Posted Jan 26, 2016 21:37 UTC (Tue) by zuki (subscriber, #41808) [Link] (7 responses)

I listened to Sarah Sharp's great SCaLE 14x talk today, and she touches the same subject. 'git clone' of the 500MB kernel repo costs quite a lot over the unreliable and expensive network in India, and inability of git clone to resume the download is a problem.

A change of maintainership for Mercurial

Posted Jan 27, 2016 2:21 UTC (Wed) by viro (subscriber, #7872) [Link] (6 responses)

; rsync -av rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git .
; git clone linux.git linux
; sed -ie 's!url = .*!url = git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git!' linux/.git/config
; cd linux
; git gc
; cd ..
; rm -rf linux.git

at which point you have a local tree matching the mainline. git pull on it will keep it up to date, and all subsequent pulls will use git protocol, not rsync one, so they'll be reasonably small. Moreover, adding that tree as reference when cloning *other* remote trees based on Linus' one will avoid most of the traffic for those - see --reference in git clone(1) for details.

Of course, initial rsync can be interrupted and restarted as many times as needed.

A change of maintainership for Mercurial

Posted Jan 27, 2016 3:25 UTC (Wed) by zuki (subscriber, #41808) [Link] (5 responses)

Of course it can be done. Why isn't something as reliable the default?

A change of maintainership for Mercurial

Posted Jan 27, 2016 3:39 UTC (Wed) by neilbrown (subscriber, #359) [Link]

> Of course it can be done. Why isn't something as reliable the default?

Presumably because the people whose itch this is haven't stepped forward to scratch it.

A change of maintainership for Mercurial

Posted Jan 27, 2016 4:29 UTC (Wed) by viro (subscriber, #7872) [Link]

Ask git developers; I'm not one.

A change of maintainership for Mercurial

Posted Jan 27, 2016 12:40 UTC (Wed) by pboddie (guest, #50784) [Link]

Because with Git, this kind of thing is always a "feature", whereas in any other tool it is a frequently-paraded "deal-breaker" that instantly sent the person opining on such matters back to Git, regardless of any accompanying loss of convenience. (But with the apparent benefit of silencing the gathered crowd who were pestering the individual concerned about not using Git - and having all their repositories on GitHub - in the first place.)

Resumable data transfer

Posted Jan 28, 2016 11:43 UTC (Thu) by tialaramex (subscriber, #21167) [Link] (1 responses)

Why doesn't this work if you just have working TCP? Merely losing the ability to move IP packets for a few seconds shouldn't cause TCP itself any trouble, they'll be queued or retried automatically. It's crazy for us to insist on doing all this work, again, in every high level application protocol.

Or is this problem that in fact people are stuck with "Internet" service providers who are actually more like "Internet Explorer service providers", offering a half-hearted commitment that web browsers will sort-of work, mostly and nothing resembling actual Internet service except as a coincidence?

Resumable data transfer

Posted Jan 28, 2016 13:17 UTC (Thu) by zuki (subscriber, #41808) [Link]

> Why doesn't this work if you just have working TCP?

This is not about short interruptions, but about the TCP stream going down. Maybe if we had STCP...

A change of maintainership for Mercurial

Posted Jan 27, 2016 16:08 UTC (Wed) by jnareb (subscriber, #46500) [Link] (8 responses)

> For me, `git clone` is really annoying because I have crap internet. It seems like it should be fixable to make `git clone` work over unreliable internet but it hasn't happened yet...

Creating *"resumable clone"* with Git "smart" protocols is not easy. It was attempted many times, and never finished.

What was done is speeding up cloning thanks to 'bitmap indexes' (on server-side), and unfortunately not formalized solution of providing "seed" bundles (which are ordinary files, and can be downloaded in a resumable way: resumable HTTP, resumable FTP, rsync, BitTorrent, etc.) under similar URL as repo (e.g. repo is at https://git.example.com/project, bundle is at https://git.example.com/project.bundle). Unfortunately http://bundler.caurea.org/ is no longer.

A change of maintainership for Mercurial

Posted Jan 27, 2016 16:56 UTC (Wed) by viro (subscriber, #7872) [Link] (7 responses)

Out of curiosity - what's the problem with salvaging already downloaded on an interrupted fetch? Basically, create an empty repository, add a remote, then start fetching it; if interrupted, at least leave the fully fetched objects in local repository, so that the next attempt at fetch wouldn't bother with those.

Sure, transfer aborted while passing the pack index is probably too hard to bother with, but most of the time the shit will hit the fan during the transfer of pack contents...

Disclaimer: I hadn't looked at the send-pack in details, so I might be easily missing something.

A change of maintainership for Mercurial

Posted Jan 27, 2016 21:09 UTC (Wed) by viro (subscriber, #7872) [Link]

As the matter of fact, looks like the pack index is rebuilt rather than being sent and git-unpack-objects can handle a truncated pack just fine, so its combination with listing everything in .git/objects/[0-9a-f][0-9a-f]/* and feeding that to git-pack-objects should do just fine; clearly wasteful, but it shouldn't be hard to brew something like git-repair-truncated-pack out of builtin/unpack-objects.c with less disk space eaten in process...

What am I missing here, other than "nobody bothered to do that so far"? Looks like handling restartable fetches over smart protocols should be doable without too much PITA - just treat aborted transfer as "repair .git/objects/pack/tmp_pack* into normal
*.{idx,pack} pair and tell the caller that <recovered objects> out of <total> had been fetched and that fetch needs to be retried to get the rest"...

Re: resumable git clone

Posted Jan 28, 2016 10:38 UTC (Thu) by jnareb (subscriber, #46500) [Link] (5 responses)

> Out of curiosity - what's the problem with salvaging already downloaded on an interrupted fetch? [...]

The problem is the following (from what I understand; I didn't try to implement such protocol myself).

First, with "smart" protocols client and server negotiate what server has and client wants, and what revisions client and server have in common, to send minimal number of objects (well, as minimal as possible with constraints they work under, like time spent on finding minimal set). This means that between starting an interrupted fetch and re-starting the fetch, both client and server might been updated with new revisions (or some revisions might be "deleted").

Second, server creates individualized packfile on the fly, with the data (revisions) client needs. Packing is multi-threaded for performance reasons, and is driven by heuristics... and that means that even if the set of objects is the same, packfile might be not byte for byte identical. That's I think the major stumbling point - IIRC any attempts of making packing predictable slowed the operation down; also, ultimately failed.

HTH

BTW. there is something called GitTorrent there...

Re: resumable git clone

Posted Jan 28, 2016 11:57 UTC (Thu) by johill (subscriber, #25196) [Link] (1 responses)

Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

I don't immediately see where that breaks down, even looking at the packfile format.

Re: resumable git clone

Posted Jan 28, 2016 12:45 UTC (Thu) by jnareb (subscriber, #46500) [Link]

> Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

The problem might be in that the client-server negotiation very much depends on the fact that the client has all reachable objects (this assumptions tremendously reduces numbers of "have" client has to send), but in packfiles it is usually more recent first (because this way it packs better). Also, objects are sent as deltas; if you have delta chain without base object, it is useless.

So I guess that it would be hard work to extract a bit of history from partial packfile, for little benefit - the amount of history that one can extract from interrupted pack is, I guess, rather small.

Re: resumable git clone

Posted Jan 28, 2016 13:19 UTC (Thu) by viro (subscriber, #7872) [Link] (2 responses)

So don't try to get it to send the rest of packfile; you want a smaller packfile anyway, with the rest of the *objects* in it. The tricky part is to convince the damn thing that you do have such-and-such objects already; AFAICS, normally it goes by _references_, not by actual availability of objects. Which is a bloody annoyance in its own right - the first push after the merge window goes like this:

Linus' repository has everything up to v4.5-rc1
kernel.org vfs.git has it in alternates and its refs cover everything up to v4.4
local mirror of Linus' repository has everything up to v4.5-rc1
local vfs.git has that in its alternates and remotes and kernel.org vfs.git in remotes

git checkout for-linus
git merge origin
<edit, test, commit, etc.>
git push vfs for-linus

... and settle down for quite a bit of waiting, since it will push *everything* in range from v4.4 to v4.5-rc1 into kernel.org vfs.git, even though the damn thing _has_ all those objects available via alternates. Back when kernel.org trees came with shell access, ssh over there and git remote update origin from vfs.git prior to the push used to reduce the traffic by several orders of magnitude; no such thing these days...

Re: resumable git clone

Posted Jan 28, 2016 16:51 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

FWIW, it might make sense to teach git-{fetch,upload,send,receive}-pack to optionally ask the sending side send the list of SHA1 it's going to send, so that recepient could say "don't send <this subset>, I already have those" or "send only <this subset>, I already have everything else".

In situations where recepient has alternates or scattered objects from an aborted fetch it has a potential to cut down on the amount of traffic quite a bit. And it shouldn't affect the tricky parts of calculating the set - it would be done _after_ the set is found.

So in the situation of git push into a repository that has most of that push in alternates, but doesn't have them covered by any references (which is very common for kernel.org trees) it would still calculate the same huge pile, send 20*(number of objects from the last merge window) bytes, get back "no, I'm only interested in these 50 objects" and proceed to send just those. It's easily a couple of orders of magnitude reduction of traffic.

And it case of e.g. aborted git fetch we would see a bunch of scattered objects (the loss from broken delta chains will be a small fraction), have the remote send the same (20 * number of objects) bytes (~90Mb for pulling Linus' tree into an empty one), then respond to it with up to half of that, in form of either "send me those" or "send me all but those" and proceed to fetch the rest. And that - on a fairly large repository...

Re: resumable git clone

Posted Jan 31, 2016 13:17 UTC (Sun) by kleptog (subscriber, #1183) [Link]

ISTM an easier and possibly more backward compatible way to do this is to have receive-pack not just send a list of the refs it has, but also a list of hashes that are older that the sender can use for matching. For example for each ref something like ref~10, ref~20, ref~40, etc.. Or parents of recent merges, recent common ancestors, entries from the reflog, etc... I'm sure someone can think of something. Unless that would bloat the data for the common case. Perhaps a "receive-pack --try-harder".

I think the problem with partial pack files is due to git storing the current revisions as full files and storing old revisions as diffs w.r.t. them. Which makes it hard to make sense of a partial pack file because you actually need the current revisions to make sense of the old ones.

You could make it more reliable by, if send-pack would notice it is sending 1000 revisions, to split its job internally into chucks of 100 revisions, making it more likely a partial pack will be somewhat useful. In the end you're sending the same data, just in a different order, though there is some inefficiency.