Linus on the BK withdrawal [LWN.net]

Seriously... version control is hard

Posted Apr 7, 2005 0:46 UTC (Thu) by gstein (guest, #3612) [Link]

Larry is right. Version control is hard. Believe me, I was part of the team that built Subversion. It took us four years to get to 1.0. It was useful long before then. But polishing the edges and making dead sure that you would never lose a single piece of data... that takes a long time.

We were self-hosting in 1.5 years. The next 2.5 years were the fit and finish.

Go ahead and try to build a version control system. To some extent, yes: it isn't all that hard. You'll have the basics done in a month. We are all so familiar with the functionality of the system that we understand the needed functionality very well. But reducing that to practice is really, really frickin' difficult. Have you considered a delta algorithm? network protocol? working copy metadata? merge system? storage system? authentication, encryption, authorization? The list goes on and on.

I've said it before at CodeCon 2003: Larry's right. Version control is hard. I'm sad to see what's happening [to the kernel] right now, and I don't think Larry's been acting all that nicely towards OSDL, but I will agree with him on this topic.

Hard, but apparently not impossible :-)

Posted Apr 7, 2005 12:55 UTC (Thu) by emk (guest, #1128) [Link]

I'm truly impressed by the work you've done on Subversion, and I know that any kind of serious storage software requires a scrupulous attention to detail.

In a previous life, I hacked on compilers. Sure, they're hard--especially good ones--but I'd never doubt the ability of Cygnus or Ximian (and their many volunteer hackers) to write a world-class compiler.

Maybe I'm being unfair to McVoy, but it always seemed that he considered version control software to be somehow unique: harder than compilers, harder than kernels, harder than X11--harder than all these terrifyingly complex projects which the open source community does so well.

Certainly, building a good version control system takes years, and requires many skilled programmers. But the open source community can presumably do that, and do it well, without reverse-engineering BitKeeper.

Hard, but apparently not impossible :-)

Posted Apr 7, 2005 13:41 UTC (Thu) by hppnq (guest, #14462) [Link]

It's an interesting question: how does one (the Open Source community in general) enforce the creation of a technically superior piece of software? I'd say the distinction between something that "just works" and something that is extremely well engineered (and therefore much more succesful than its peers) is largely attributable to factors that cannot really be controlled: brilliance being the most obvious one.

Compare this to the way we ended up with the theory of special relativity. The problem solved by this theory had been known for decades before one spark of genius solved it in a revolutionary way of thinking. In retrospect it is hard to imagine that several Nobel prize winners spent so much time on theories that high school students now would find offensively naive. However, there's no guarantee that we would view the world in the way we do now if Einstein hadn't been around.

(Actually, the current state of physics demands that someone like Einstein step up to the plate..)

I'm sure dwheeler has much more to say about this. ;-)

Hard, but apparently not impossible :-)

Posted Apr 7, 2005 23:08 UTC (Thu) by lm (guest, #6402) [Link]

Perhaps it's because I've done significant work on kernels, X11, and written the better part of two different SCM systems. Oh, and compilers long ago. But not optimizers which may be where the really hard compier work lies. Working on BK is harder than any of those including multi-threading kernels.

It's certainly harder than all the things you listed in _my_ experience. And history has shown it not to be easy for anyone else either. Greg Stein's post nailed it: it's easy to get something that looks like it works, it's a lot harder to get something that always works.

We've counted more than 10,000 replicas of the Linux kernel in BK. More than 150,000 changesets over the 2.4/2.5/2.6 series. Making that work, where all of them synchronize with each other, using versions of BK that differ by years without having it all fall apart, yes, that's hard.

I'm sure one of the many open scm systems will get there some day but it's going to take a while and it's going to be no fun when they issue a new release that breaks your old repos, etc. Or they aren't around to answer the phone when it gets corrupted, etc.

I'm not saying you can't do it, you can. It's just hard. And not much fun once the thrill of having people use your code wears off and you have to stick around and support them. Please don't take this as looking for strokes or whatever else it is that I'll be accused of next, it's simply a statement that it is really hard and if you want a good answer it is going to take a lot of people a lot of time to get there. I don't care how good they are, I think I'm very good at this and I've done it twice already and it would take me years if I had to start over with everything I know now. It's hard, there are a zillion corner cases. None of which are very important in isolation but all of them added up are somewhat overwhelming.

You'll see. Whoever emerges as the person doing this work for you deserves a lot of your support. Please give it to him or her.

--lm