Linus Torvalds' BitKeeper blunder [LWN.net]

From:		Bradley Chapman <kakadu-AT-gmail.com>
To:		neil_mcallister-AT-infoworld.com, letters-AT-infoworld.com
Subject:		Linus Torvalds' BitKeeper blunder
Date:		Wed, 4 May 2005 10:20:57 +0100
Cc:		letters-AT-lwn.net

Mr. McAllister,

After reading your short InfoWorld article* about Linus Torvalds'
so-called 'blunder' with regards to the termination of the free
BitKeeper client, I am curious as to why you chose the term 'blunder'.
If the removal of BitKeeper had caused the complete collapse of the
kernel development community, the term 'blunder' would have been
justified - however, it did not, and now the kernel is being developed
using git, a tool written by Linus in direct response to the loss of
BK.

Personally I don't feel that Linus fell for the free beer argument
when he chose to migrate to BitKeeper; I feel that he cast aside any
thoughts on ideology and chose to use it because it was the best tool
for the job, despite the criticisms of others who felt that
introducing closed-source software into such a critical role was
dangerous to the health of kernel development.

Now that the 'free beer' version of BitKeeper has been removed due to
the so-called reverse engineering done by Andrew Tridgell, Linus's
decision is better seen as a well-intentioned mistake, not as a
blunder. Your comments about the dangers of being forced to switch
gears due to software withdrawal are valid, but in this case the gear
switch was not as traumatic as it may have been for other software
projects - the creation of git is proof positive of that.

In my opinion, Linus Torvalds' choice to use BitKeeper was made
entirely for technical reasons, and therefore, at best could only be
considered a mistake after the fact.

Yours,

Bradley Chapman

* - http://www.infoworld.com/article/05/05/02/18OPopenent_1.html

to post comments

Desire for freedom is practical self-interest

Posted May 5, 2005 6:22 UTC (Thu) by bignose (subscriber, #40) [Link] (7 responses)

> I feel that [Linus] cast aside any thoughts on ideology

I don't see where ideology comes into it. Linus cast aside freedom from vendor control. That's a blunder.

> and chose to use [BitKeeper] because it was the best tool for the job,

Despite the fact that, now that his requirements are understood better, a tool to keep the workflow going has been written within mere days and is being improved rapidly.

> despite the criticisms of others who felt that introducing closed-source
> software into such a critical role was dangerous to the health of
> kernel development.

The kernel project change history -- Linus's main reason for using the tool -- is held in a database that the vendor claims ownership of. That's the type of outcome easily predicted, and actively warned against, at the time Linus chose to abandon practical software freedoms.

Desire for freedom is practical self-interest

Posted May 5, 2005 6:36 UTC (Thu) by TheOneKEA (guest, #615) [Link] (6 responses)

> I don't see where ideology comes into it. Linus cast aside freedom from
> vendor control. That's a blunder.

That's one way of looking at it.

> Despite the fact that, now that his requirements are understood better, a
> tool to keep the workflow going has been written within mere days and is
> being improved rapidly.

Keep in mind that the tool was written based on the knowledge of what BK did for Linus and co. and how it improved their working practices - if they had never used BK, they would have never really known how to write git.

> The kernel project change history -- Linus's main reason for using the
> tool -- is held in a database that the vendor claims ownership of. That's
> the type of outcome easily predicted, and actively warned against, at the
> time Linus chose to abandon practical software freedoms.

Maybe so, but it is only metadata. What matters is that the vendor did not claim ownership of the kernel itself, only the metadata. Irregardless of the mistakes of choosing BK in the first place, the fact that Linus was able to walk away at all, much less with access to the metadata retained, indicates that the outcome was not as terrible as it could have been.

Desire for freedom is practical self-interest

Posted May 5, 2005 7:42 UTC (Thu) by dlang (guest, #313) [Link] (4 responses)

and note that the metadata people are complaining about is the stuff that bitkeeper uses to track the patches, not the patches themselves, where they came from, or any comments related to them (in other words, nothing that anyone ever typed in is being withheld)

also note that several people have recreated the full kernel history in various tools in the last few weeks. Linus choose not to import all of that data into his Git repository, not becouse of any lack of ability to get the data, but to allow git to settle down and go through any changes that are found to be needed before the repository gets too large (we've already gone through one rebuild of the git repository, and others are possible, it's far easier to do this if you are dealing with 10's of megs of data then if you are dealing with gigs of data)

Desire for freedom is practical self-interest

Posted May 5, 2005 15:42 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (3 responses)

dlang wrote:

and note that the metadata people are complaining about is the stuff that bitkeeper uses to track the patches, not the patches themselves

It was my understanding that information available via other means such as the BK-CVS gateway was never a full equivalent of what one got directly with the BK client, though I might have been misinformed. In any event, ask yourself what this "stuff that BitKeeper uses to track the patches" turned out to be: SCCS files! So, BitMover claims to have proprietary rights to SCCS? That will come as a heck of a shock to the Bell Labs guys!

In effect, however, what BitMover was really claiming to "own" was a monopoly right to require everyone to use nothing but its proprietary client to talk to its repositories, obliging those who declined its licence to have second-class access to hosted data, or none at all. And it never had that right.

Rick Moen
rick@linuxmafia.com

Desire for freedom is practical self-interest

Posted May 6, 2005 0:22 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

just shortly before this fiasco there was an update to the CVS info and I saw at least one of the die-hard opponents fo bk who had been screaming about the missing info agree that the info was now complete enough.

What exactly was risked by the telnet and netcat trick?

Posted May 6, 2005 1:01 UTC (Fri) by rickmoen (subscriber, #6943) [Link] (1 responses)

dlang, thanks for clarifying that. It's one of the aspects of the former situation that had been left hanging for a long time. (Just for completeness, I note in passing a potentially importance difference in kind between direct open-source access to a repository's contents and access through a gateway that exists courtesy of a third party's goodwill and willingness to fund it.)

Many long ago clearly wearied of this entire subject, but I remain interested in seeing whether some key factual questions can resolved with a reasonable degree of objectivity. For example: What was BitMover's real concern with Tridgell's activity? The legal basis for their complaint was at best thin (attempting to control what a licensee's contractor did in his spare time), but that's not the point: What actual proprietary interest was threatened?

(I pose that as a real, not rhetorical, question. One might well have existed and been at risk, but what specifically was it?)

You say the metadata in question was the repository's internal data structures (how I'm interpreting your phrase "the stuff that bitkeeper uses to track the patches", anyway -- correct me if I'm wrong). And it would seem reasonable to assert that those BK repository data structures were and remain a trade secret -- like BK's advanced merge algorithm, etc. But, unless I'm missing something vital, Tridgell's activity would seem to have given him no discernable access to repository data structures, SCM merge algorithms, etc. Those remained squirreled away in, variously, the BitMover-hosted repositories and the proprietary, binary-only client software. What Tridgell's perfunctory telnet-and-netcat "client" fetches is... SCCS files.

Given that distinction, if "BitMover-proprietary metadata" denotes, as you say, something much more sensitive than a pile of SCCS files -- which sensitive information would logically exist in the hosted repository and proprietary client but not the SCCS files -- then what company-proprietary information was McVoy upset over?

I have no objection to how you construe the term "metadata" in this context -- but then that's not what Tridgell retrieved, is it?

Rick Moen
rick@linuxmafia.com

What exactly was risked by the telnet and netcat trick?

Posted May 6, 2005 16:38 UTC (Fri) by ksmathers (guest, #2353) [Link]

Larry sells licenses on a per-developer-seat basis. Add a competing client, and Larry loses money. Improve a competing client to the point where it can emulate most or all of the standard client functionality (not by really doing local version control, but by passing all of the hard work on merging and branching to the 'real' BK 'server', and you end up with all of the BK
functionality at a fraction of the cost. One server, lots of clients -- totally destroys Larry's business model.

I'm guessing that what gave Larry the scare was realizing how easy it would be to make a dumb client instead of a smart peer.

Desire for freedom is practical self-interest

Posted May 29, 2005 21:41 UTC (Sun) by dag- (guest, #30207) [Link]

> Keep in mind that the tool was written based on the knowledge of what
> BK did for Linus and co. and how it improved their working practices - if
> they had never used BK, they would have never really known how to write
> git.

I don't agree with the fact they would never really know how to write git. It definitely may have taken longer to write git if they didn't have the experience BK gave them.

Just like BK development would have taken longer if they didn't have the experiences of those people that used BK for kernel development. (Remember that Linus was important for part of the feature set that made BK useful to kernel development).

I don't think Larry was stupid when he arranged for BK to become the kernel developers' tool, aside from the media-attention and the licenses, he got a lot of experience feedback and set of requirements out of it. Of course when the advantages weared out it was again in his best interest to find an exit strategy.

Linus Torvalds' BitKeeper blunder

Posted May 5, 2005 15:15 UTC (Thu) by gdt (subscriber, #6284) [Link]

As a once and future software development manager (boo, hiss!) I've got to say that Linus blundered. Not so mucb in the area of technology (bk was good) or politics (bk wasn't free, and that bit him, but the huge effort and good progress with git/cogito seems to be remediating that issue) but in handling the people involved.

How major this is I've no idea, since I don't have day-to-day contact with any of the principals and people's reactions to poor treatment vary so much. Given time, doubtless their actions will inform us.

Linus Torvalds' BitKeeper blunder

Posted May 5, 2005 15:31 UTC (Thu) by rickmoen (subscriber, #6943) [Link] (1 responses)

Bradley Chapman wrote:

the 'free beer' version of BitKeeper has been removed due to the so-called reverse engineering done by Andrew Tridgell

This claim reflects a rather eccentric and polemical theory of causation, rather like saying "It's your own fault I broke your wife's arm. You should have handed over your wallet."

I doubt that, e.g., Perforce would ever cease letting open-source developers have free-of-charge licences for their SCM (which they do, by the way), for no better reason than someone telnetting into a repository, hitting return, and following the resulting suggestion of typing "help" -- which is literally what Tridgell did. Yet, BitMover decides differently, and you call it "due to" Tridgell's small typing spree? Weird.

McVoy finding unbearable people manually issuing BitKeeper directives using telnet and netcat, and getting to see SCCS files by doing so, may or may not have been a rational response -- you be the judge -- but calling it all Tridgell's doing seems a bit warped.

Rick Moen
rick@linuxmafia.com

Linus Torvalds' BitKeeper blunder

Posted May 6, 2005 0:34 UTC (Fri) by khim (subscriber, #9252) [Link]

Gosh. Of course not. Think about it: limit for 64K changesets was in BK from the day one. Why it was there ? 16bit word is not native size for contemporary processors. I'm pretty sure Larry planned for this from the very beginning. Not as for the outcome perhaps but for quite real possibility.

And in the end he even tried to blame open source community for not forcing everyone to play by his (constantly changing) rules: "This is really an open source community problem and I have to say that the open source community couldn't have failed more than they have. Unlike the Marine corp, the open source community is more than willing to ignore their bad apples as 'not my problem' (the Marine corp punishes the group for the behavior of the bad apples, pretty soon there are no bad apples)."

Gosh.

Linus Torvalds' BitKeeper blunder

Posted May 7, 2005 0:03 UTC (Sat) by njhurst (guest, #6022) [Link]

Doesn't Larry read LWN? Why hasn't he defended his decisions to all these ungrateful whining hippies? Perhaps because he knows he was right and we'll all come round to his viewpoint soon enough...