10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com) [LWN.net]

Linux.com talks with Linus Torvalds about the development of Git. "Just to pick an example: the concept of 'merging' was generally considered to be something really quite painful and hard in most SCM's. You'd plan your merges, because they were big deals. That's not acceptable to me, since I commonly do tens of merges a day when in the merge window, and even then, the biggest overhead shouldn't be the merge itself, it should be testing the result. The 'git' part of the merge is just a couple of seconds, it should take me much longer just to write the merge explanation message."

to post comments

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 6, 2015 18:02 UTC (Mon) by pr1268 (guest, #24648) [Link] (19 responses)

Fascinating—self-hosting SCMs FTW!

I find it especially interesting how Linus had git working well so quickly because of good design ("The actual amount of that early code is actually fairly small, it all depended on getting the basic ideas right.")

I consider myself a git beginner, but I do find its style and use quite easy (compared to other SCMs). Thanks for linking this article!

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 6, 2015 18:24 UTC (Mon) by proski (guest, #104) [Link] (6 responses)

Eric Raymond was right - "Smart data structures and dumb code works a lot better than the other way around"

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 6, 2015 21:45 UTC (Mon) by job (guest, #670) [Link] (1 responses)

If esr said that he stole it from Fred Brooks, of the Mythical Man-Month fame.

It's not theft when it's attributed

Posted Apr 8, 2015 13:06 UTC (Wed) by michaelkjohnson (subscriber, #41438) [Link]

He specifically attributed it as a paraphrase from MMM

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 13:59 UTC (Tue) by epa (subscriber, #39769) [Link] (3 responses)

Isn't the pretext of git that the data *model* is as dumb as possible? Now, under the covers, git does lots of clever things to reduce duplication, but this seems to be a case of smart code (git) working on a relatively dumb data structure (the pack format).

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 15:55 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

My impression is that a big chunk of git is that it gets the filesystem to do a lot of the heavy lifting for it. It also makes a different space/speed tradeoff from most other version control software by storing full copies of each version of a file rather than trying to save space by using diffs. You can call that a dumb data format, but it's an appropriate design choice, which is really the point. An appropriate data representation will greatly simplify your code relative to an inappropriate one.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 18:32 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

git conceptually stores the full contents of the files instead of diffs, but within a few months of being created it gained the ability to optimize it's storage with pack files, which compress many versions of files into one file using diffs.

But the data model of how git functions is both very simple and very powerful, that model has survived, with implementation optimizations being added without changing the basic model.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 20:57 UTC (Tue) by nix (subscriber, #2304) [Link]

Indeed, one of the important distinctions that almost nothing else has got is the distinction between the data model (commit / tree / blob / tag, header + object, SHA-1 hashes) and the data representation (gzipped loose objects in hash-named tree versus xdeltaified packfiles with associated indexes). That alone has allowed git to improve in countless ways without having to worry about breaking the world every time they made a change. No svn/bzr upgrade here!

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 6:34 UTC (Tue) by marcH (subscriber, #57642) [Link] (9 responses)

> I find it especially interesting how Linus had git working well so quickly because of good design

... and even more interesting is Linus explaining where this good design came from: the use and study of bitkeeper and other previous attempts.

As Bernard de Chartres once said...

Posted Apr 7, 2015 13:07 UTC (Tue) by rvfh (guest, #31018) [Link] (8 responses)

"standing on the shoulders of giants"...

As Bernard de Chartres once said...

Posted Apr 7, 2015 18:12 UTC (Tue) by marcH (subscriber, #57642) [Link] (6 responses)

Great: I did not realize Newton was standing on the shoulders of the giants *even when saying it*! A very early form of recursive acronyms?

Now let's all go back to filing some patents for $DAYJOB @ $BIGCORP

As Bernard de Chartres once said...

Posted Apr 7, 2015 21:01 UTC (Tue) by nix (subscriber, #2304) [Link] (5 responses)

No no no, Newton was writing to Hooke, and he bore a massive grudge against Hooke (and other grudges against, it sometimes seemed, half the human race). I hold with the (fairly recent) view that he's basically slamming Hooke here. Hooke was not tall, and the implication is that Newton had been standing on the shoulders of people who were very specifically *not Hooke*, oh and also Hooke didn't see as far as he did, nyah nyah!

Newton was not a very nice man.

As Bernard de Chartres once said...

Posted Apr 7, 2015 22:55 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

There's a fair chunk of evidence, I believe, that Newton (like many greats) suffered from Aspergers.

The trouble with being very clever is that in your normal daily life, you are always right compared to the people around you. And it can be a very nasty shock when you come up against someone as clever or cleverer than you, and you find it very hard to admit - even to yourself - that they are right and you are wrong.

That's why science tends to advance in fits and starts - the bright young things come up with new ideas, and when they become the next old farts, they can't accept that the new generation of bright young things are overturning their ideas. (I believe Dalton is touted as a classic example - in his youth he advanced science greatly, in his old age he did great damage).

Cheers,
Wol

As Bernard de Chartres once said...

Posted Apr 14, 2015 10:57 UTC (Tue) by nix (subscriber, #2304) [Link]

There's a fair chunk of evidence, I believe, that Newton (like many greats) suffered from Aspergers.

What you mean I think is 'Baron-Cohen has speculated that Newton suffered from Aspergers'. There is no evidence. There can be no evidence. The man is dead, and when he was alive the term had not been coined and nobody existed who could assess him for it.

The most you could say is that it is not inconsistent that he might possibly have suffered from it -- but, to be honest, the degree of viciousness the man routinely exhibited is not consistent with Aspergers (not from my internal viewpoint anyway, though naturally that viewpoint is biased). Would someone trying to control and prevent painful extremes of emotion, as aspies normally do, really let it out all the time? (Mind you, that was the English way back then -- the stiff-upper-lip thing is a Victorian artifact.)

As Bernard de Chartres once said...

Posted Apr 8, 2015 1:15 UTC (Wed) by rahvin (guest, #16953) [Link] (2 responses)

Newton didn't like Hooke because he claimed Hooke stole his ideas and published them. According to the story in the new Cosmos Newton basically locked himself in a room for several years because he claimed Hooke stole one of his papers and claimed the ideas as his own. Hooke claimed for years he'd solved one of theories Newton is known for and it wasn't until other professors got involved and did a put up or shut up where Hooke presented nothing while Newton supplied a proof that made him famous.

Given the other historical accounts of Hooke he doesn't sound like a very nice person IMO. And if Hooke did steal Newtons ideas and publish them as his own he was the slimiest of slimiest and I completely sympathies with Newton's hatred of him. On of the other interesting bits in the Cosmos stuff on Newton was that Newton probably burned the only painting of Hooke that ever existed.

People have speculated about Newton's sanity and whether he had Aspergers but I personally believe his erratic behavior is tied to Mercury poisoning. He actually had a stint in a hospital (mental hospital) once that many people blame on this experiments with Mercury though at the time he blamed it on working too hard. People had no idea how poisonous mercury was at the time and freely handled it without any protections. Newton in particular experimented with it heavily.

As Bernard de Chartres once said...

Posted Apr 8, 2015 3:23 UTC (Wed) by rodgerd (guest, #58896) [Link]

> Newton didn't like Hooke because he claimed Hooke stole his ideas and published them.

Which seems rather unlikely, given that Hooke basically worked full-time at the Royal Society on science, while Newton veered in and out of scientific research and alchemy. He certainly suggested the inverse quare law before Newton. His obscurity has more to do with Newton using his political power later in life to reshape the official histories of the Society than anything else.

As Bernard de Chartres once said...

Posted Apr 9, 2015 7:20 UTC (Thu) by paulj (subscriber, #341) [Link]

If I remember accounts of Newton's childhood correctly, he was displaying heavy signs of intellectual brilliance and social disinterest/disengagement from early days. I don't know when he started experiments with mercury though.

As Bernard de Chartres once said...

Posted Apr 8, 2015 3:17 UTC (Wed) by zblaxell (subscriber, #26385) [Link]

I'd always thought Git's design was more like "seeing the giants' shoulders in the distance and running in the opposite direction." ;)

Successful copy

Posted Apr 7, 2015 15:09 UTC (Tue) by ncm (guest, #165) [Link] (1 responses)

Indeed, git was copied straight across from Graydon Hoare's Monotone, ditching the SQLite storage apparatus Linus recognized as slowing down the most essential operations. He treated Monotone as a prototype that had flushed out the interesting problems. This doesn't detract from his achievement -- to re-implement from scratch in a few days without depending on libraries was no small thing, but that was only possible at all because the architecture was already settled.

Soon after, Monotone got the attention to performance it needed (and now keeps up with Git) but by then the baton had been passed.

Successful copy

Posted Apr 7, 2015 22:09 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

Soon after? In 2010 or 2011, the instructions still were "download this seed database then update it" (at least for pidgin). And the update took a non-negligible amount of time. Something like 20 minutes *after* I downloaded the updates.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 6, 2015 22:21 UTC (Mon) by Sesse (subscriber, #53779) [Link] (39 responses)

One thing I didn't see addressed at all is how poorly git scales to big projects. People think of the Linux kernel as big, but then you have e.g. Android, where the kernel is only one of _lots_ of parts… So they work around it with repo (ick).

This appears to be a fundamental scaling issue of git; you have to clone basically all of history down to your local drive. (Sure, there are things like grafts, but they are also painful.) I would love to see Linus address that. Maybe he will if the kernel just gets big enough :-P

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 6, 2015 23:16 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (29 responses)

> One thing I didn't see addressed at all is how poorly git scales to big projects. People think of the Linux kernel as big, but then you have e.g. Android, where the kernel is only one of _lots_ of parts… So they work around it with repo (ick).
That's another issue, it's really not a job for one VCS to control multiple interdependent modules. Something like meta-git is needed and so far 'repo' tool is the best we've got (yeah, ick).

> This appears to be a fundamental scaling issue of git; you have to clone basically all of history down to your local drive.
That's rarely a problem. Even old projects have at most 3-5x their current size in history. And of course, there are shallow clones.

One significant exception is artwork and other types of media. Something like more integrated git-annex is really needed for them.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 6:15 UTC (Tue) by marcH (subscriber, #57642) [Link]

> That's another issue, it's really not a job for one VCS to control multiple interdependent modules.

It think it is because meta-git needs fairly intimate knowledge of git. See git submodules for (another) instance.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 6:41 UTC (Tue) by Gollum (guest, #25237) [Link]

I've been conceptualising an equivalent to Perforce, for use as a replacement for SharePoint (when used purely for sharing files, particularly binary files).

Basically, Perforce allowed for checking out of only a partial tree of the repository, as well as limiting the amount of history it transferred. This made it suitable for use as a distributed file repo.

It seems to me that this is something that is relatively easily achievable using shallow clones, as well as sparse checkout. Keeping track of what has changed should be relatively easily done by cloning the commits and associated tree data, which should be comparatively small, especially if it is limited to the portions of the tree that you have indicated that you are interested in. This could take the form of a flag at a particular location of the tree, indicating interest, or disinterest in that node, whether a subdirectory or a file. Interest would typically be inherited from its parent, but could obviously be overridden if desired.

And once the interest in a particular portion of the tree is no longer relevant, that portion would be marked as disinterested, and could be purged from the local copy if necessary/desired.

The biggest piece of work involved in this would actually just be in the UI/GUI to make this easy to use. The building blocks are there already.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 16:34 UTC (Tue) by elanthis (guest, #6227) [Link] (3 responses)

git-annex is a partial solution at best.

When you have 100's of GB of 3DS Max files, 8k x 8k uncompressed source textures, source audio files, etc. that are being hammered on by a hundred or so artists and designers...

You go Perforce to handle that, and let the code handling suffer. And no, you can't just easily use separate SCMs for data and code, because there very well might be dependencies (e.g. complicated Flash files that interact with native UI integration code and hence must stay relatively in sync with each other, as well as e.g. making sure the switchover to Max 2014 only happens once the branch that upgrades the Max SDK for the import/export tools is integrated, etc.).

Then there's the SDKs themselves, which often are proprietary (you can't just have a git submodule to a github repo) and filled with binary stuff of their own (test data if nothing else). And then the compiled binaries; even for FOSS code it's often way better to check in binaries than to expect the engineers to build those modules every time they check out a new working copy (for really the same reasons most users prefer installing RPMs over rebuilding SRPMs every time), so you might very well have to deal with tens of GBs just in .a/.lib/.so/.dll/.dylib files.

As skeptical as I was of the concept, even file locks start coming in handy once you're working on unmergeable data files at the kind of scale that a AAA game company or movie production company deals with.

In theory Platic SCM is a merger of Perforce and Git that designed for specifically this use case, and Perforce is pushing Git integration pretty hard these days, but both are imperfect and there's still no workable FOSS solution.

git's Windows support is still kinda awful (it works, but it's several times slower than Mercurial on Windows file systems, and good git GUIs that are non-uber-nerd-friendly are still hard to come by) which is kind of a big deal. Most game companies use Windows as their primary OS for all development (in fact, the game console SDKs from multiple vendors require it as they are only delivered as Visual Studio plugins) and most movie companies still have a heavy mixture of Linux, OSX, and Windows.

git is a fantastic solution for many problems, but certainly not all of them, and those other problems are still only served by proprietary software. Which is quite unfortunate.

Denying that git is flawed or that git-annex solves the problems doesn't help. The first step in addressing a problem is admitting that you have one, after all. :)

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 16:43 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> You go Perforce to handle that, and let the code handling suffer. And no, you can't just easily use separate SCMs for data and code, because there very well might be dependencies (e.g. complicated Flash files that interact with native UI integration code and hence must stay relatively in sync with each other, as well as e.g. making sure the switchover to Max 2014 only happens once the branch that upgrades the Max SDK for the import/export tools is integrated, etc.).

But git-annex should handle that, it simply pushes large files into a separate blob storage. All the regular git functionality still works (branching, merging, etc.). It's not well integrated and merges are sometimes tricky.

The major missing piece is lock manager - you really need locking for non-code assets.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 12:36 UTC (Wed) by nye (guest, #51576) [Link]

>the game console SDKs from multiple vendors require it as they are only delivered as Visual Studio plugins

Wait, you mean they don't provide their own compilers? Or their compilers are dependent upon other bits of VS toolchain?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 8:29 UTC (Thu) by henrikb (guest, #58898) [Link]

Git Large File Storage looks interesting and might solve the problems you mention.

https://github.com/blog/1986-announcing-git-large-file-st...

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 2:03 UTC (Wed) by roc (subscriber, #30627) [Link] (22 responses)

Some smart companies that operate at huge scale (Facebook, Google) have decided that a single unified repository works better than multiple repositories, for them. Having a single repo --- if you can get it to scale --- simplifies a lot of problems that are otherwise extraordinarily difficult. "repo" is horrific and I haven't met anyone who likes git submodules. So, there seems to be real demand for a VCS that can scale to incredibly large repositories. The fact that git can't do it is a real problem, not an indication that there is no problem.

Currently it seems that git won't solve this problem, and hg will take over this space instead.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 2:53 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

I think having some *separate* project to manage the build (gathering dependencies, setting flags, making a deliverable, etc.) of these mega projects is the way to go. But most of our stuff is also FOSS (so we cater somewhat to actually having external dependencies for distros) and maybe ~1% of their total repo size (though I'd like to know how large the projects which would be separate repos in a saner world would be), so there's a difference as well. The projects I have worked on that used a commit-the-world strategy would have been *much* easier to develop as standalone packages (and development was indeed done by setting install paths in standalone versions rather than in-repo because it was faster and easier). I think that project was at least ~20M SLOC all told (other than third-party).

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 4:17 UTC (Wed) by dlang (guest, #313) [Link] (8 responses)

you'd be amazed at the number of people inside Google who use git for working on the Google codebase.

Also, As Google starts opening up it's internal APIs for non-google people to use them, the biggest argument for a single repo (the ease of changing both an API and all it's users) weakens significantly, because you now have users of the APIs who you can't change, so you have to have more stable APIs. With stable APIs you no longer need to have everything in one repo so it can all be changed at once.

You should also remember that at these 'huge scale' shops, you also have very large numbers of machines running the same thing, so their codebase, while large, does not scale with the number of machines they are running, but rather with the number of different things they are doing.

Also, these companies tend to not have a QA organization, the code gets reviewed by the programming team and then pushed into beta/production (with whatever automated testing that the programmers put in place)

There are a lot of things they do well, but there are also a lot of things that they wish they had done differently years ago.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 4:51 UTC (Wed) by roc (subscriber, #30627) [Link] (7 responses)

Stable APIs are costly. It makes sense to pay for them when, as you say, you need to open APIs up to external users. It does not make sense to pay for them just because your VCS doesn't scale.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 19:08 UTC (Wed) by dlang (guest, #313) [Link] (6 responses)

> Stable APIs are costly.

"The implementation is the API" is also costly, things become part of the API that were never intended to be API items, and you frequently get trapped due to past implementation details.

It's 'easy' to import version X of an opensource project into your massive codebase (into an appropriate subdirectory) and then you know that your other stuff will continue to work. But when you can no longer put off updating that project (usually because of some security vulnerability or a new feature that you really need), updating to it when you've ignore it's API definition and haven't used or tested any of the intermediate versions becomes a huge task.

It's much harder to keep that same opensource project separate, and update it as the project releases new versions, but you avoid building up the technical debt of the postponed upgrades.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 9:56 UTC (Thu) by roc (subscriber, #30627) [Link] (5 responses)

By "the implementation is the API", I assume you mean the absence of a thorough definition of the interface's behavior. That's mostly orthogonal to interface stability. You can have poorly defined interfaces whose behavior is nevertheless stable (e.g., Win32). On the other hand, you can have well-defined interfaces which are unstable and evolve over time, with their users being updated as the interface changes.

Interface stability imposes maintenance costs, because you have to keep supporting the old interface even in situations where it would be better for both the implementation and the users if you changed the interface. Organizational structures and VCSs that let you update the implementation and the users together avoid some of those situations and reduce those costs.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 10:37 UTC (Thu) by nye (guest, #51576) [Link] (3 responses)

>You can have poorly defined interfaces whose behavior is nevertheless stable (e.g., Win32)

Win32 has extensive and detailed documentation. What makes you say it is poorly defined?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 13:28 UTC (Thu) by dgm (subscriber, #49227) [Link]

It is also a very big API. There are many quirks and corner cases that are not covered by the documentation.

Ask the Wine developers if you don't want to take my word.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 18:16 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

From a practical standpoint, the Windows API is defined by the implementation, not by the documentation. From day 1, third party developers have made awful hacks that depend on Windows implementation details that were never intended to be part of the formal API definition. Microsoft has bent over backward to continue supporting them, even when that means cluttering up their codebase with compatibility code that should never have been necessary. It isn't code, but the version jump from Windows 8 to Windows 10- to avoid giving misleading version information to badly written code that treats Windows 9 as a sign that it's dealing with 95 or 98- is an outstanding example.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 20:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

Oh, there's no "version jump". They just didn't tell anyone they're counting in base 9 now. Can't make a pattern in the release names more than 2 long after all. ;)

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 17:06 UTC (Thu) by dlang (guest, #313) [Link]

in any substantial environment, you need to support backwards compatibility for your API for at least one version back (so that you don't have a flag-day upgrade of everything on your network.

Personally, I care more about the APIs between various components than the coding/language of the components. If the API is good, then the implementation can be improved without breaking things. If the API is always changing based on what's more convenient (which isn't always done when all the code is in one repo, but is an extremely easy trap to fall into when it's all in one repo), then changes in one component tend to ripple through and impact everything.

And as the discussion of trivial patches on the l-k list in the last few days shows, even 'trivial' changes can introduce bugs, so changing other people's software to match your API changes is not always a safe thing to do.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 4:43 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

In this approach problems usually start when you need to versioning more complicated than 'change dependency and recompile the world'.

If you try to solve them directly on the level of the VCS then you get a bad case of ClearCase.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 7:00 UTC (Wed) by marcH (subscriber, #57642) [Link] (7 responses)

> So, there seems to be real demand for a VCS that can scale to incredibly large repositories.

Like a few other people around here, I think what's really missing is a good meta-VCS more than an "VCS than can scale". In other words: a better repo/submodules and two different levels of versioning instead of just one.

While git does not scale up, any simplistic, "commit the world" VCS does not scale down. Yes there are times where you want to lock and commit-the-world to avoid paying for stable APIs, but no it's not something you want to be forced to do all the time, and no you typically don't want the history of your local project to be interleaved and interlocked and polluted with hundreds of commits for stuff completely unrelated to you - even if you avail good filtering features. Even in the lack of stable APIs, modularity is still a very useful and Good Thing.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 6:41 UTC (Thu) by roc (subscriber, #30627) [Link] (5 responses)

*You* want a meta-VCS. Google and Facebook don't. You can argue they're just wrong, but it's not very convincing since (unless you happen to work there) they almost certainly understand their needs better than you do.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 7:05 UTC (Thu) by marcH (subscriber, #57642) [Link] (3 responses)

A good meta-VCS does not even exist yet, so no, they almost certainly don't understand their needs - because no one does. They're only using what's best *and* available now.

Before decentralized version control even existed almost every company, project and individual thought they "understood" their version control needs. And the answer was of course centralized.

Even today many professionals don't understand version control basics - decentralized or not. Just look around you. In this very interview Linus named it the second most boring computer topic after databases.

I'm not saying a meta-VCS is a silver bullet and that everyone needs one. I'm just saying many people will want one if/when there is a good one. At least all the people currently suffering with repo or git submodules and likely more.

> (unless you happen to work there)

I know a chef and a recruiter at Google, I'll try to have a chat about meta-VCSes next time I see them.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 10:01 UTC (Thu) by roc (subscriber, #30627) [Link] (1 responses)

> They're only using what's best *and* available now.

That's not really true. Google and Facebook have done and are still doing development work both on meta-VCSes (gclient, repo) and regular VCSes (hg). But as far as one can tell from outside, for their internal work they prefer the latter.

Meta VCS

Posted Apr 9, 2015 17:08 UTC (Thu) by marcH (subscriber, #57642) [Link]

> Google and Facebook have done and are still doing development work both on meta-VCSes (gclient, repo) and regular VCSes (hg). But as far as one can tell from outside, for their internal work they prefer the latter.

What I was just (and very cryptically) hinting at: they're huge companies with a gazillions of different projects, some doing things this way, others the other way. Some very high-profile Google projects are still heavily relying on repo (what a poor name) and do not show any sign of migrating to a single giant... repo.

I don't think the concept of a "company-preference" makes much sense, and I don't think it should either. It's more "the best tool of the moment for the job".

Meta VCS

Posted Apr 9, 2015 17:23 UTC (Thu) by marcH (subscriber, #57642) [Link]

> At least all the people currently suffering with repo or git submodules and likely more.

By the way Portage, Bitbake and Maven are also meta-VCSes when you think about it. They just happen to be more focused on the build engineer workflow (building everything from scratch) than on the coder workflow (downloading everything pre-built as fast as possible except for one project).

If you look behind the scenes, you'll find ad-hoc re-implementation of these in many companies.

There is proliferation of not much loved meta-VCSes, which I think shows clearly there is a gap and a lack of a good, developer-friendly one.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 9:22 UTC (Thu) by tao (subscriber, #17563) [Link]

Considering that Google (a friend of mine there, actually) wrote repo, they at least at some point wanted a meta-VCS...

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 15:30 UTC (Thu) by marcH (subscriber, #57642) [Link]

> no you typically don't want the history of your local project to be interleaved and interlocked and polluted with hundreds of commits for stuff completely unrelated to you

... and I forgot to mention the most obvious issue: a single giant repo makes forking and merging third party code a nightmare.

Modularity has a cost but it has value too.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 16:38 UTC (Wed) by joey (guest, #328) [Link] (2 responses)

Facebook came onto the git list, pointed out that git scaled badly in the some million files range, and nobody seemed to care to answer them. So they went somewhere else.

git does have scalability problems in that file range. IMHO it needs a smarter index file format, at least as the first step to scaling better. Writing out an index of 1 million files each time you stage a change is not ideal.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 19:12 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

Since that point people have been working on that issue.

Just reporting a problem doesn't result in a solution. They ended up writing their own fix for a different VCS, they could have done something similar for git instead. (unless you can show that they made similar comments on the other VCS list and got help instead of just doing all the work internally, which is what I understand happened)

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 6:45 UTC (Thu) by roc (subscriber, #30627) [Link]

From the Facebook post:
> Our engineers were comfortable with Git and we preferred to stay with a
> familiar tool, so we took a long, hard look at improving it to work at
> scale. After much deliberation, we concluded that Git's internals would
> be difficult to work with for an ambitious scaling project.
So they would have kept using git if they thought "they could have done something similar for git instead".

"unless you can show that they made similar comments on the other VCS list and got help":
> Just as importantly, the Mercurial developer community is actively
> helping us address our scaling problems by reviewing our patches and
> keeping our scale in mind when designing new features.
So yeah.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 14:01 UTC (Tue) by epa (subscriber, #39769) [Link] (7 responses)

Isn't there a way of using git in "svn style" - that is, keeping the remote repository for all commit operations and only copying the current version of files to your local disk? Some tool that lets you have multiple checkouts of the same repository, effectively. I forget what it is called.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 21:56 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

It might have worked, but git doesn't allow you to alias or shadow the built-in commands anymore. You're going to have to use aliases now.

As for multiple worktrees, there's a tool in git's contrib called git-newworkdir or something.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 8:21 UTC (Wed) by dgm (subscriber, #49227) [Link] (5 responses)

I use this style occasionally. You make a "bare" (git init --bare) repository in a networked filesystem that acts as a central repo, and then shallow (git clone --depth 1) clones in your workstation.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 19:06 UTC (Wed) by epa (subscriber, #39769) [Link] (4 responses)

I think that's not quite it - because you still have a cloned repository. When you commit you are committing to that repository, and then you have to push.

What I mean is to keep an existing repository, don't clone it, but create a fresh 'checkout' of it. You then work on that checkout and commit back to the repository as you wish. I know this is possible somehow but what I am less sure of is how well git copes if the repository has changed in the meantime - so it might not be that practical after all.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 19:24 UTC (Wed) by dlang (guest, #313) [Link]

> what I am less sure of is how well git copes if the repository has changed in the meantime

it depends on if the changes conflict or not.

Look at the kernel development process, developers pull down a copy of the code, work on it locally, and then publish updates.

the -next tree pulls the updates from many different developers into one tree, usually with very little manual work needed.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 20:10 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (2 responses)

It sounds like what you want is similar to what the git-new-workdir script provides. I've never used it myself, and it sounds a bit fragile, but in principle it provides the ability to have multiple working directories with a common object store by making symlinks in the .git directory to the original repository, with a few independent files like .git/HEAD to track the current branch.

The main .git directory could even be on a network share to work remotely, a la SVN, but I wouldn't count on great performance. The network filesystem would need to support O_EXCL semantics to avoid corruption if concurrent access is a concern.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 13, 2015 10:32 UTC (Mon) by epa (subscriber, #39769) [Link]

Thanks, git-new-workdir is what I was thinking of but couldn't remember the name.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 14, 2015 11:00 UTC (Tue) by nix (subscriber, #2304) [Link]

I use git-new-workdir extensively (I keep long-term branches under active maintenance in new-workdirs so that the make products don't need to be pointlessly regenerated all the time) and have never had a problem with it. Everything I've tried, right up to largescale history rewrites, works fine from new-workdirs.

It's quite slow over NFS, but no slower than anything else is over NFS :P

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 6:37 UTC (Thu) by pabs (subscriber, #43278) [Link]

I prefer myrepos to repo for a meta-git style thing. It does more than just git and is cross-project too.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 6:29 UTC (Tue) by marcH (subscriber, #57642) [Link] (5 responses)

> just look at how long CVS and even RCS have stayed around, but at some point git just took over.

RCS (or something like similar) is going to stay around for longer than CVS because it does something that git is extremely poor at: versioning files *in place*, that is: embedded in directories of other, non-versioned files. Typically: /etc and other config files - sometimes embedded in... git repos even.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 6:59 UTC (Tue) by lab (guest, #51153) [Link] (1 responses)

> RCS (or something like similar) is going to stay around for longer than CVS because it does something that git is extremely poor at: versioning files *in place*, that is: embedded in directories of other, non-versioned files

ESR wrote SRC to address this usecase:
http://www.catb.org/esr/src/
http://esr.ibiblio.org/?p=6502

It's a modern wrapper around RCS.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 7:11 UTC (Tue) by marcH (subscriber, #57642) [Link]

> ESR wrote SRC to address this usecase:
> It's a modern wrapper around RCS.

Great idea, great web site with great explanations... everything looks great. Unfortunately:
- The name is unsearchable (I actually searched for it before my previous post above and gave up). Not too late to change for something sexier?
- RCS is never further than one yum or apt-get install away.
- RCS almost never requires knowing more than 3-4 commands and it's too late: they're hardcoded deep in my brain by now.
- As said on the SRC site, Emacs abstracts all commands away anyway.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 7:40 UTC (Tue) by jezuch (subscriber, #52988) [Link] (2 responses)

> it does something that git is extremely poor at: versioning files *in place*, that is: embedded in directories of other, non-versioned files

Can you elaborate on that? I've been using etckeeper with git for a while at it seems to work better than extremely poorly. We're also doing versioning of non-/etc configuration files at work and it too seems to work well...

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 7:51 UTC (Tue) by marcH (subscriber, #57642) [Link] (1 responses)

> > embedded in directories of other, *non-versioned* files

> I've been using etckeeper

I did not know about etckeeper: isn't versioning the entire /etc ?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 7, 2015 9:38 UTC (Tue) by mchouque (subscriber, #62087) [Link]

> I did not know about etckeeper: isn't versioning the entire /etc ?

Yes, that's what it does.

Watch out for the selinux policy files if you use that: they're big and they'll be versioned, taking space for nothing (you can change that by customizing your .gitignore but still, it doesn't do what you want: per file SCM).

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 7:52 UTC (Wed) by NAR (subscriber, #1313) [Link] (8 responses)

the concept of 'merging' was generally considered to be something really quite painful and hard in most SCM's.

I don't think it depends that much on the SCM - it depends more on the actual code to be merged. I've worked on a project using CVS and merging was usually done in minutes. Now I work on a project using git and merging takes half day. In the CVS project people barely touched the same files, but in the git project everybody edits the same 10 files, so a merge is essentially a rewrite due to the conflicts.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 12:23 UTC (Wed) by mathstuf (subscriber, #69389) [Link] (7 responses)

This sounds like you're comparing two projects using completely different workflows (or even different stages in their life cycle). Yeah, when files changed are disjoint merging is easy even with cp. Backing out a merge is dead simple in git; I can only imagine CVS. It sounds like your project could do with splitting those 10 files a little more.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 18:03 UTC (Wed) by Wol (subscriber, #4433) [Link] (6 responses)

Actually, I doubt splitting the files would do any good. git is very good at merging files with loads of changes. The problem arises when all the changes are to the same few lines of code ...

Cheers,
Wol

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 8, 2015 19:10 UTC (Wed) by dlang (guest, #313) [Link] (5 responses)

what VCS is any good when you have different developers making lots of changes to a few lines of code?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 7:11 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

Any that has good integration with a good, graphical merge tool. I have tried a few and was never fully satisfied. Any git-friendly recommendations?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 13:12 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I've found vimdiff sufficient for my needs so far, but threesome[1] might be a better way to go.

[1]http://sjl.bitbucket.org/threesome.vim/

Meld

Posted Apr 12, 2015 9:06 UTC (Sun) by drothlis (guest, #89727) [Link]

Have you tried Meld?

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 10, 2015 21:23 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> what VCS is any good when you have different developers making lots of changes to a few lines of code?

Certainly in a company, I'd say that's a management problem, not a VCS problem.

I know I know - in a Free Software project the manager doesn't have the authority to prevent the problem, but really, shouldn't the developers be developing separate forks and not treading on each others' toes?

(If you've got several developers all changing the same few lines of code, isn't that indicative of a problem where nobody knows what the code is actually meant to do?)

Cheers,
Wol

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 14, 2015 10:49 UTC (Tue) by nix (subscriber, #2304) [Link]

The code might be in APL so virtually all the work is being done in a few lines ;)

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 9, 2015 13:11 UTC (Thu) by dennisdjensen (guest, #25165) [Link] (4 responses)

Nice interview. Since Linus mentions BitKeeper, which is based on the same data representations as SCCS, I am surprised that nobody has mentioned Schily SCCS yet (http://sccs.sf.net). There are some interesting comparisons to RCS there, even though the real interesting part is how good the delta file format is, apropos choice of good data representations. Linus chose to go in an entirely different direction from BitKeeper. It's interesting how Linus' main concern of merging has affected the design choices in Git.

It'll be interesting to see what happens with Git and Mercurial over the next few years, especially now that Google and Facebook both intend to improve Mercurial for large and central repositories.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 14, 2015 10:53 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

Oh that page is a hoot, an absolutely classic example of Joerg's relentless lie-through-omission argumentative process. If you believed it, SCCS was and remains the best version control system ever: we all know this is not true (God knows I do, I fought with it for fourteen years). SCCS's actual, manifold deficits are barely touched on, while tiny insignificant improvements are lauded to the skies.

The thing is... in this case, there is no advantage at all to this hilarious mendacity -- except, perhaps, that Joerg has clearly invested time in this historical curiosity, wants it to appear not to be a complete waste, and hugely overdoes it. So, the same as the way he treats cdrecord etc then.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 15, 2015 19:28 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

It's probably just as well – he's likely to be the only person interested in the thing, so there's little chance of a flame war about how the incompetent Debian sccs maintainers have done terrible things in their package, how the Linux kernel is set up all wrong to support sccs, etc.

As long as Jörg plays with his version of sccs he's unlikely to bother other people about other things, so I hope he has many interesting ideas for improving sccs and will stick with it for a very long time.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 21, 2015 21:45 UTC (Tue) by nix (subscriber, #2304) [Link]

Quite. What I find interesting about this is that the underlying causes of this -- things like loss aversion -- are all cognitive flaws that we all feel in ourselves, and try (and sometimes fail, God knows) to suppress -- but in Joerg's case, he seems to think they're *good* things and try to amplify them.

10 Years of Git: An Interview with Git Creator Linus Torvalds (Linux.com)

Posted Apr 14, 2015 15:25 UTC (Tue) by paulj (subscriber, #341) [Link]

Just to fill out the DVCS family tree a bit:

Sun Microsystems built a DVCS around SCCS for its internal use (solaris). You would 'bringover' (pull) a workspace from the parent, work on it, bringover and merge any updates in the parent as/when needed, then push back stuff when done. It was called "Teamware".

Larry McVoy worked on a DVCS at Sun that influenced TeamWare (to what extent there seems to be differing accounts of IIRC).