A change of maintainership for Mercurial [LWN.net]

Matt Mackall, the creator of the Mercurial source-code management system, has announced that he is ready to move on to a new project. "So over the course of this year, I'm going to gradually remove myself from daily involvement in the project. As lots of people and companies have a lot invested in Mercurial, I'm doing this over a long period of time to make sure it goes smoothly."

to post comments

A change of maintainership for Mercurial

Posted Jan 26, 2016 2:03 UTC (Tue) by jeff@uclinux.org (guest, #8024) [Link] (36 responses)

Perhaps a minority view:

I am nervous about the decline of tools other than git.

While of course I don't begrudge Matt a change of project, I think it important and hope that hg remains viable. Both for those of us who rely on it, and as an alternative to monoculture.

A change of maintainership for Mercurial

Posted Jan 26, 2016 2:29 UTC (Tue) by faheem (guest, #47574) [Link]

You think "it" important, as in Mercurial, presumably?

I agree. I think it's important to have alternatives, even to free software projects.
And Mercurial is an excellent piece of software and has lots of good stuff going on.
And it has a better UI than git.

But I don't think that Matt Mackall is planning to stop working on Mercurial completely. He will still be around, I expect. Just not as much as before. But if that is wrong, I expect someone will correct me.

And there are quite a lot of other capable developers that are invested in the project. I think Mercurial will survive even if Matt Mackall stops working on the project completely.

A change of maintainership for Mercurial

Posted Jan 26, 2016 7:42 UTC (Tue) by oldtomas (guest, #72579) [Link] (2 responses)

> I am nervous about the decline of tools other than git.

So am I. Although I might qualify as a kind of Git fanboi. After years of CVS (which I kinda enjoyed), suffering under SVN's bureaucracy and trying to wrap my head around TLA (I didn't like Bazaar that much) -- git hit home for me. It somehow resonated with my brain structure. I'm happy with it.

But this tendency (which seems to be accelerating dramatically) of "winner takes all" makes me worry. Not only in the realm of version control systems.

A change of maintainership for Mercurial

Posted Jan 26, 2016 16:02 UTC (Tue) by drag (guest, #31333) [Link] (1 responses)

Eventually software, and this is especially relevant for open source, gets to the point were it's just a 'public good'. Just stuff that is used to build other stuff. While previously it was considered of critical importance and was cutting edge.. eventually it becomes mundane and taken for granted as 'just works'.

If you want to avoid the potential problems of relying on software technology that may end up being sub-par then it becomes important for layering and formal APIs. You need to have interfaces that carry out a action, but whose behavior is not tied to a particular implementation. So if your API is command line arguments and environmental variables.. this is good because another tool can come along and provide a interface for compatibility with existing code. However if your application that depends on git goes and does 'fancy' things, like descending into the .git directories and fiddling with bits in there... then it's not going to be robust enough to survive major git revisions, much less replacing git with git++ (or whatever it may be).

A change of maintainership for Mercurial

Posted Jan 27, 2016 12:25 UTC (Wed) by oldtomas (guest, #72579) [Link]

> Eventually software, and this is especially relevant for open source, gets to the point were it's just a 'public good'.

I like to squint at it from a slightly different angle. That's why I arrive at slightly different conclusions :)

Ultimately I view the problem of making "the best software" as an optimization problem, faced with a monstrous, ill-defined, constantly changing "goodness function".

Free software attacks it (in the ideal) by some kind of genetic annealing, where solutions in the wild are combined and then start descending some valleys of that function.

If the population is too small, then let's hope we have one solution trickling down the right valley. And that this "ricght valley" doesn't become the "wrong valley" tomorrow.

Actually it's much more complicated than that, because "software" encompasses the mind share and the people around it, capable of care and feeding it.

> If you want to avoid the potential problems of relying on software technology that may end up being sub-par then it becomes important for layering and formal APIs.

APIs (or API philosophies) are just software. Contrast UNIX's traditional spartan-typed ("all is files and streams of bytes") philosophy to old mainframes (structured records, akin to punched cards) or to Microsoft's power shell, passing OO goodness around. That's a lot of constraints which will reflect on the system "evolving" under those respective shadows.

And... the more complex the APIs, the more difficult to implement an "alternative service". That's especially dangerous if "both sides" of the APIs are codeveloped in one project, even in one repo.

A change of maintainership for Mercurial

Posted Jan 26, 2016 9:41 UTC (Tue) by farnz (subscriber, #17727) [Link]

At least for now, Mercurial is in a safe position - Mozilla, Google and Facebook all have very active contributors, and Unity3D and Fujitsu both provide regular patches. Looking at the link, Matt's plan is to slowly hand over to the existing contributors over the next 6 months; I'm sure Facebook is supporting him in this move; assuming no seriously bad luck, Mercurial will thrive without Matt.

A change of maintainership for Mercurial

Posted Jan 26, 2016 10:05 UTC (Tue) by pinkpony (guest, #92373) [Link] (21 responses)

Is it a decline? I'm often surprised how many people use hg (our corp has it everywhere and we have to use it) and even like it. It certainly doesn't look like there is a decline going on.

(Personally, I'd probably like to see hg decline, I have to use it a lot and never managed to see anything else than a bad git implementation in it. I mean, it works and it's a nice piece of software that is fine for the purpose and I'd probably pick it if git didn't exist, but it offers nothing extra compared to git, except for relative slowness, issues, python dependencies and more restricted workflow...)

A change of maintainership for Mercurial

Posted Jan 26, 2016 11:15 UTC (Tue) by Wol (subscriber, #4433) [Link] (2 responses)

> (Personally, I'd probably like to see hg decline, I have to use it a lot and never managed to see anything else than a bad git implementation in it. I mean, it works and it's a nice piece of software that is fine for the purpose and I'd probably pick it if git didn't exist, but it offers nothing extra compared to git, except for relative slowness, issues, python dependencies and more restricted workflow...)

The problem is different people have their brains wired differently. It's the vi/emacs wars all over again. If the git workflow suits the way your brain is wired, then fine. But it appears that some people just cannot understand git because their brains are wired like Mercurial :-)

I have the same problem with word processors - I *H*A*T*E* Word-like processors, Word itself and all the clones (including LO Writer, oddly enough, seeing as I'm a minor contributor to LO :-) But WordPerfect *just* *works* - although as Corel have steadily pushed it towards being a Word clone, it's started jarring more and more.

At the end of the day, you want alternatives, just to allow for those people who "don't get on" with the majority tool. (And which is why the fact that I believe you can use the git UI on mercurial, or the mercurial UI on git, is so important. Which is why I want a "reveal codes" window on Writer ... :-)

Cheers,
Wol

A change of maintainership for Mercurial

Posted Jan 26, 2016 12:11 UTC (Tue) by anselm (subscriber, #2796) [Link]

For me personally, the nifty things you can purportedly do in git but not in Mercurial aren't worth the excess cognitive load that comes with git even for the things that Mercurial does too. So far, anyway. Oh, and there's TortoiseHG :^)

A change of maintainership for Mercurial

Posted Jan 26, 2016 12:55 UTC (Tue) by pinkpony (guest, #92373) [Link]

Not really. I don't think I care about the UI that much. It's about having issues like hg running out of memory for a slightly larger commit because it's running on 32-bit python and you are on a pretty obscure platform where you don't have 64-bit python binaries on hand... Or being repeatedly told to restrict usage of branches, because you cannot delete them, once they are published/pushed (only 'close') and they may eventually clog the entire SCM, or so it is told. It's about it being generally slow, when you have to often clone repositories that aren't even very big. Stuff like this...

A change of maintainership for Mercurial

Posted Jan 26, 2016 19:02 UTC (Tue) by error27 (guest, #8346) [Link] (17 responses)

The facebook folks wrote a compelling article why they use Mercurial instead of git.

https://code.facebook.com/posts/218678814984400/scaling-m...

For me, git clone is really annoying because I have crap internet. It seems like it should be fixable to make git clone work over unreliable internet but it hasn't happened yet...

A change of maintainership for Mercurial

Posted Jan 26, 2016 21:37 UTC (Tue) by zuki (subscriber, #41808) [Link] (7 responses)

I listened to Sarah Sharp's great SCaLE 14x talk today, and she touches the same subject. 'git clone' of the 500MB kernel repo costs quite a lot over the unreliable and expensive network in India, and inability of git clone to resume the download is a problem.

A change of maintainership for Mercurial

Posted Jan 27, 2016 2:21 UTC (Wed) by viro (subscriber, #7872) [Link] (6 responses)

; rsync -av rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git .
; git clone linux.git linux
; sed -ie 's!url = .*!url = git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git!' linux/.git/config
; cd linux
; git gc
; cd ..
; rm -rf linux.git

at which point you have a local tree matching the mainline. git pull on it will keep it up to date, and all subsequent pulls will use git protocol, not rsync one, so they'll be reasonably small. Moreover, adding that tree as reference when cloning *other* remote trees based on Linus' one will avoid most of the traffic for those - see --reference in git clone(1) for details.

Of course, initial rsync can be interrupted and restarted as many times as needed.

A change of maintainership for Mercurial

Posted Jan 27, 2016 3:25 UTC (Wed) by zuki (subscriber, #41808) [Link] (5 responses)

Of course it can be done. Why isn't something as reliable the default?

A change of maintainership for Mercurial

Posted Jan 27, 2016 3:39 UTC (Wed) by neilbrown (subscriber, #359) [Link]

> Of course it can be done. Why isn't something as reliable the default?

Presumably because the people whose itch this is haven't stepped forward to scratch it.

A change of maintainership for Mercurial

Posted Jan 27, 2016 4:29 UTC (Wed) by viro (subscriber, #7872) [Link]

Ask git developers; I'm not one.

A change of maintainership for Mercurial

Posted Jan 27, 2016 12:40 UTC (Wed) by pboddie (guest, #50784) [Link]

Because with Git, this kind of thing is always a "feature", whereas in any other tool it is a frequently-paraded "deal-breaker" that instantly sent the person opining on such matters back to Git, regardless of any accompanying loss of convenience. (But with the apparent benefit of silencing the gathered crowd who were pestering the individual concerned about not using Git - and having all their repositories on GitHub - in the first place.)

Resumable data transfer

Posted Jan 28, 2016 11:43 UTC (Thu) by tialaramex (subscriber, #21167) [Link] (1 responses)

Why doesn't this work if you just have working TCP? Merely losing the ability to move IP packets for a few seconds shouldn't cause TCP itself any trouble, they'll be queued or retried automatically. It's crazy for us to insist on doing all this work, again, in every high level application protocol.

Or is this problem that in fact people are stuck with "Internet" service providers who are actually more like "Internet Explorer service providers", offering a half-hearted commitment that web browsers will sort-of work, mostly and nothing resembling actual Internet service except as a coincidence?

Resumable data transfer

Posted Jan 28, 2016 13:17 UTC (Thu) by zuki (subscriber, #41808) [Link]

> Why doesn't this work if you just have working TCP?

This is not about short interruptions, but about the TCP stream going down. Maybe if we had STCP...

A change of maintainership for Mercurial

Posted Jan 27, 2016 16:08 UTC (Wed) by jnareb (subscriber, #46500) [Link] (8 responses)

> For me, `git clone` is really annoying because I have crap internet. It seems like it should be fixable to make `git clone` work over unreliable internet but it hasn't happened yet...

Creating *"resumable clone"* with Git "smart" protocols is not easy. It was attempted many times, and never finished.

What was done is speeding up cloning thanks to 'bitmap indexes' (on server-side), and unfortunately not formalized solution of providing "seed" bundles (which are ordinary files, and can be downloaded in a resumable way: resumable HTTP, resumable FTP, rsync, BitTorrent, etc.) under similar URL as repo (e.g. repo is at https://git.example.com/project, bundle is at https://git.example.com/project.bundle). Unfortunately http://bundler.caurea.org/ is no longer.

A change of maintainership for Mercurial

Posted Jan 27, 2016 16:56 UTC (Wed) by viro (subscriber, #7872) [Link] (7 responses)

Out of curiosity - what's the problem with salvaging already downloaded on an interrupted fetch? Basically, create an empty repository, add a remote, then start fetching it; if interrupted, at least leave the fully fetched objects in local repository, so that the next attempt at fetch wouldn't bother with those.

Sure, transfer aborted while passing the pack index is probably too hard to bother with, but most of the time the shit will hit the fan during the transfer of pack contents...

Disclaimer: I hadn't looked at the send-pack in details, so I might be easily missing something.

A change of maintainership for Mercurial

Posted Jan 27, 2016 21:09 UTC (Wed) by viro (subscriber, #7872) [Link]

As the matter of fact, looks like the pack index is rebuilt rather than being sent and git-unpack-objects can handle a truncated pack just fine, so its combination with listing everything in .git/objects/[0-9a-f][0-9a-f]/* and feeding that to git-pack-objects should do just fine; clearly wasteful, but it shouldn't be hard to brew something like git-repair-truncated-pack out of builtin/unpack-objects.c with less disk space eaten in process...

What am I missing here, other than "nobody bothered to do that so far"? Looks like handling restartable fetches over smart protocols should be doable without too much PITA - just treat aborted transfer as "repair .git/objects/pack/tmp_pack* into normal
*.{idx,pack} pair and tell the caller that <recovered objects> out of <total> had been fetched and that fetch needs to be retried to get the rest"...

Re: resumable git clone

Posted Jan 28, 2016 10:38 UTC (Thu) by jnareb (subscriber, #46500) [Link] (5 responses)

> Out of curiosity - what's the problem with salvaging already downloaded on an interrupted fetch? [...]

The problem is the following (from what I understand; I didn't try to implement such protocol myself).

First, with "smart" protocols client and server negotiate what server has and client wants, and what revisions client and server have in common, to send minimal number of objects (well, as minimal as possible with constraints they work under, like time spent on finding minimal set). This means that between starting an interrupted fetch and re-starting the fetch, both client and server might been updated with new revisions (or some revisions might be "deleted").

Second, server creates individualized packfile on the fly, with the data (revisions) client needs. Packing is multi-threaded for performance reasons, and is driven by heuristics... and that means that even if the set of objects is the same, packfile might be not byte for byte identical. That's I think the major stumbling point - IIRC any attempts of making packing predictable slowed the operation down; also, ultimately failed.

HTH

BTW. there is something called GitTorrent there...

Re: resumable git clone

Posted Jan 28, 2016 11:57 UTC (Thu) by johill (subscriber, #25196) [Link] (1 responses)

Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

I don't immediately see where that breaks down, even looking at the packfile format.

Re: resumable git clone

Posted Jan 28, 2016 12:45 UTC (Thu) by jnareb (subscriber, #46500) [Link]

> Yeah but viro's point was that if you have the partial packfile, and the download gets interrupted, you can index it, see which objects you actually have in there (some/many might be corrupt, but the check is trivial), and then do another fetch - now negotiating a download where you already have more objects.

The problem might be in that the client-server negotiation very much depends on the fact that the client has all reachable objects (this assumptions tremendously reduces numbers of "have" client has to send), but in packfiles it is usually more recent first (because this way it packs better). Also, objects are sent as deltas; if you have delta chain without base object, it is useless.

So I guess that it would be hard work to extract a bit of history from partial packfile, for little benefit - the amount of history that one can extract from interrupted pack is, I guess, rather small.

Re: resumable git clone

Posted Jan 28, 2016 13:19 UTC (Thu) by viro (subscriber, #7872) [Link] (2 responses)

So don't try to get it to send the rest of packfile; you want a smaller packfile anyway, with the rest of the *objects* in it. The tricky part is to convince the damn thing that you do have such-and-such objects already; AFAICS, normally it goes by _references_, not by actual availability of objects. Which is a bloody annoyance in its own right - the first push after the merge window goes like this:

Linus' repository has everything up to v4.5-rc1
kernel.org vfs.git has it in alternates and its refs cover everything up to v4.4
local mirror of Linus' repository has everything up to v4.5-rc1
local vfs.git has that in its alternates and remotes and kernel.org vfs.git in remotes

git checkout for-linus
git merge origin
<edit, test, commit, etc.>
git push vfs for-linus

... and settle down for quite a bit of waiting, since it will push *everything* in range from v4.4 to v4.5-rc1 into kernel.org vfs.git, even though the damn thing _has_ all those objects available via alternates. Back when kernel.org trees came with shell access, ssh over there and git remote update origin from vfs.git prior to the push used to reduce the traffic by several orders of magnitude; no such thing these days...

Re: resumable git clone

Posted Jan 28, 2016 16:51 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

FWIW, it might make sense to teach git-{fetch,upload,send,receive}-pack to optionally ask the sending side send the list of SHA1 it's going to send, so that recepient could say "don't send <this subset>, I already have those" or "send only <this subset>, I already have everything else".

In situations where recepient has alternates or scattered objects from an aborted fetch it has a potential to cut down on the amount of traffic quite a bit. And it shouldn't affect the tricky parts of calculating the set - it would be done _after_ the set is found.

So in the situation of git push into a repository that has most of that push in alternates, but doesn't have them covered by any references (which is very common for kernel.org trees) it would still calculate the same huge pile, send 20*(number of objects from the last merge window) bytes, get back "no, I'm only interested in these 50 objects" and proceed to send just those. It's easily a couple of orders of magnitude reduction of traffic.

And it case of e.g. aborted git fetch we would see a bunch of scattered objects (the loss from broken delta chains will be a small fraction), have the remote send the same (20 * number of objects) bytes (~90Mb for pulling Linus' tree into an empty one), then respond to it with up to half of that, in form of either "send me those" or "send me all but those" and proceed to fetch the rest. And that - on a fairly large repository...

Re: resumable git clone

Posted Jan 31, 2016 13:17 UTC (Sun) by kleptog (subscriber, #1183) [Link]

ISTM an easier and possibly more backward compatible way to do this is to have receive-pack not just send a list of the refs it has, but also a list of hashes that are older that the sender can use for matching. For example for each ref something like ref~10, ref~20, ref~40, etc.. Or parents of recent merges, recent common ancestors, entries from the reflog, etc... I'm sure someone can think of something. Unless that would bloat the data for the common case. Perhaps a "receive-pack --try-harder".

I think the problem with partial pack files is due to git storing the current revisions as full files and storing old revisions as diffs w.r.t. them. Which makes it hard to make sense of a partial pack file because you actually need the current revisions to make sense of the old ones.

You could make it more reliable by, if send-pack would notice it is sending 1000 revisions, to split its job internally into chucks of 100 revisions, making it more likely a partial pack will be somewhat useful. In the end you're sending the same data, just in a different order, though there is some inefficiency.

A change of maintainership for Mercurial

Posted Jan 26, 2016 18:26 UTC (Tue) by Nelson (subscriber, #21712) [Link] (6 responses)

Things come and go and change. If there are better ideas or better solutions to some of the problems, something will come along and be better than git. It'll have to be different and have to be better though.

The "winner takes all" aspect doesn't bother me so much as the passion about a revision control tool. I completely understand emacs vs vi, it's so much more intimate and ultimately each member of a team could use a different editor and it wouldn't make a lick of difference to the other members. At it's core, revision control is about playing along with others and "community standards" of sorts, getting overly passionate about that stuff has lead to bad things in my experience. In most software shops just the revision control is a tiny tiny piece of that too, it's even a tiny piece of the overall software tracking software. You've got defect and feature tracking, change review software, etc.. all on top of it, maybe some sort of agile/kanban board for features. Those things seem a lot easier to get upset about because damn near all of them are crap in some way or other. I guess the passion doesn't "bother" me, so much as I just find it odd, it's just a tool after all, right? It seems like getting really passionate about flat head wood screws vs Phillips head wood screws. The supposed different mental models is somewhat concerning though, I think I might stay away from developers that make those claims...

A change of maintainership for Mercurial

Posted Jan 26, 2016 22:53 UTC (Tue) by Wol (subscriber, #4433) [Link] (5 responses)

> The supposed different mental models is somewhat concerning though, I think I might stay away from developers that make those claims...

Just read Dick Feynmann, about counting time in your head, or about his experiences with Calculus.

Dick got a bit of a reputation for being a whizz with calculus, because when he learnt he was taught an unusual technique. So problems that his fellow students found hard, he found easy. What they didn't realise was that problems they found easy, Dick found hard (because he had a different mental set-up).

Or counting time - some (most?) people have their time-keeping disrupted if they have to hold a conversation at the same time - they count by speaking silently to themselves. But some people hold a conversation no problem, but if you ask them to watch TV while counting, they can't. They keep time by mentally watching a ticker-tape.

Different brains work differently. Some tools just don't work for some people because of that. And it could very easily be muscle memory - what you are exposed to first programs you to expect things to work that way, and relearning it is HARD. WordPerfect was designed to work like a typewriter ...

Cheers,
Wol

A change of maintainership for Mercurial

Posted Jan 27, 2016 9:03 UTC (Wed) by karkhaz (subscriber, #99844) [Link]

> read

or watch: https://www.youtube.com/watch?v=Cj4y0EUlU-Y

starting at 01:36

A change of maintainership for Mercurial

Posted Jan 27, 2016 16:02 UTC (Wed) by Nelson (subscriber, #21712) [Link] (3 responses)

Not to disagree with you, I do understand that people have different mental models for things. We're not talking about calculus though, we're talking about mechanisms for tracking changes to some files, usually source code and usually sharing them with a team or others. And it's not like you have to know the intricacies of the git file format or the .hg tree to make use of them.

If the mental model is so rigid that someone can't pretty easily adapt to a new tool, I'm of the belief that they'll have a hell of a time solving the problems our customers need solved. Dislike? I get that. You've typed 'hg ...' so long that you tend to automatically type it instead of 'git ...' I kind of understand that and I'd accept that for a while. But claiming to not understand it because of your mental model? I'd not hire you.

A change of maintainership for Mercurial

Posted Jan 30, 2016 4:06 UTC (Sat) by shmget (guest, #58347) [Link] (2 responses)

" And it's not like you have to know the intricacies of the git file format or the .hg tree to make use of them."

Again different brains....
I like git _because_ I got a mental picture of the 'intricacies' of git's data model -- which is quite simple with essentially 3 base components types: commit, tree, blob --
And thanks to that, git command are not magic incantations, I can 'see' what I want to do to the repo and I can match that with commands.

A change of maintainership for Mercurial

Posted Jan 30, 2016 15:52 UTC (Sat) by Wol (subscriber, #4433) [Link]

Exactly.

Do you use "any tool" because you've been taught "monkey see monkey do", or do you use it because you actually *understand* what's going on under the hood.

My mental model of a word processor matches WordPerfect, so given a problem I can guess the solution. I then try it and it usually works first time. But if I have to do it in Word, I'm lost, and it's a case of monkey see monkey do trial and error.

And it takes a LOT of retraining to get past that hurdle. If it's even possible.

Cheers,
Wol

A change of maintainership for Mercurial

Posted Feb 1, 2016 2:20 UTC (Mon) by Nelson (subscriber, #21712) [Link]

That's great but ultimately the repo structure isn't what matters, it's what is stored in it.

Maybe I should have said it different, I'd avoid any developer that had trouble using any revision control system.

Re: I am nervous about the decline of tools other than git.

Posted Jan 27, 2016 0:57 UTC (Wed) by ldo (guest, #40946) [Link] (1 responses)

I don’t see why it’s a problem. Like any Free Software project, it lives and dies by its community. If Mercurial continues to have an active community (as opposed to just a bunch of passive users), it will survive, otherwise it won’t.

Re: I am nervous about the decline of tools other than git.

Posted Jan 27, 2016 12:56 UTC (Wed) by pboddie (guest, #50784) [Link]

In general, I agree with you, although your example isn't a good one since we're only talking about the principal maintainer for a project stepping aside, presumably to be able to do other things with his time, rather than advocating the abandonment of the project.

Meanwhile, people are strongly discouraged from doing anything with Python 2. Firstly, there are indications that anyone developing it independently would have to call it something else or face action over use of the trademark, hence the potential naming of Stackless Python as just "Stackless" if any kind of "2.8" release were to occur. Secondly, there is a certain amount of peer pressure from the core development community against any further development of Python 2, although I note that GvR seems to want some of the type annotations stuff backported to 2.7 so that Dropbox can take advantage of it. Thirdly, a lot of the "messaging" that has taken place in the last decade has involved telling people that Python 3 is what they should be using, even before it was a viable choice, and so there has been a deliberate policy of undermining any community around Python 2 by making everyone regard it as legacy software (and thus undermining Python in general, but there you go).