Following up after a day in the shop [LWN.net]

Following up after a day in the shop

Posted Sep 27, 2004 2:21 UTC (Mon) by lm (guest, #6402)
In reply to: So, as an outcome of the above arguments by gallir
Parent article: An Interview with Tom Lord of Arch (O'ReillyNet)

First of all, I have to say that metal shavings are way more fun than arguing with you about our business model. I had a blast in the shop, my new (very old, from the 1950's) mill works great after I cleaned it up. I have new pictures up at http://bitmover.com/lm/mill/ and before you start blasting me that that isn't great, try and remember that what I've done is sort of the "hello world" of metal working. I know it sucks but it's damn cool anyway.

OK, moving on...

To Felix.Braun, thanks very much for the acknowledgement that the goal was to help Linux and that we have succeeded beyond our wildest dreams. When we started I was scared stiff that we would lose Linus to burnout and he is still here, kicking butt, he's great. We helped make that happen, he has no problem admitting that, we're very proud of that. Yeah, yeah, wouldn't it be great if we did it with the GPL...

To kevinbsmith, we _don't_ change the license that much. That's just FUD that the GPL fanatics spread. The last change was more than a year ago and the last change before that was more than a year before that. So we are averaging a change a year. Not exactly frequent.

And the bit about me not being part of the open source community warrants comment. That hurts. I've been a long term member of this community, I've done a ton of work for this community, the founders of redhat acknowledge my work, my copyright is in several GNU utilities, and I created benchmarking tools that Linus and others use to this day to make sure they are doing the right thing (it's worth noting that I have suppressed every port of LMbench to windows for more than 10 years simply because if the data is out there then they will fix it).

But the real bummer is that BK exists to help the FLOSS community. It's the way that I could help the most. I've walked away from so much money, you can't even imagine (4th guy at Google? You think I'd be talking to you if I had stayed?) because I wanted to help Linus. It's pretty depressing that you don't realize that the choices I've made have been to help you. If your definition of "help" is "GPLed" then you need to broaden your horizons. There are a lot of ways to help and some pretty famous people have pointed out that what we have done has helped more than most.

To hppnq, sure we have considered the GPL. Of course we have considered that. I used to think that if we GPLed BK you'd all be happy but I think some of you are so fanatic that we could GPL BK and you'd still be pissed at us. Sigh. Anyway, we are in negotiations with a company who is 100% committed to the GPL. We discussed with them the idea of GPLing BK and they looked at us and said "are you nuts? You'd go out of business in a week!" And they are correct. We spend more in a week on payroll than any company has ever made on an open source SCM system in a year. That's a 50x problem. We need to pay our people and because all of you are so sure that you could reimplement BK in a week with some shell scripts there is zero, repeat, zero, nada, no friggin way, that we could survive with a GPLed product.

I encourage you to keep on thinking about this, it's way cool to have you thinking about ways to make it work. Many of you don't believe this but my goal is to help you. It's a huge bummer for me that in order to do that we have to be a typical corporate company, I would have loved to have done this with an open source product. Perhaps this is a good time to remind you that once upon a time BK was open source with a license that said "you can't remove the openlogging part" and you guys promptly removed that part of the code. The reason BK isn't open source is because your friends cheated. Let's keep that in perspective, OK?

(Log in to post comments)

Following up after a day in the shop

Posted Sep 27, 2004 4:55 UTC (Mon) by iabervon (subscriber, #722) [Link]

I agree entirely with the need for BK to be proprietary. I have no doubt that it is ahead of everything else due to having a full-time team of really good people. OSS can get full-time paid developers, but it doesn't tend to happen until a project is as big as Linux or Apache. I'd guess that someday arch will reach the point of being important enough to enough major projects that it will get enough development effort to surpass BK, but I'd bet that you and your current team will retire before then. Ironically, I think that the existance of people prohibited from using BK forces projects to use arch instead, and that gives arch market share and attention, meaning that it improves more quickly. If anyone could use BK, I bet arch would progress about as quickly as Hurd, and nobody but Tom Lord and RMS would really care about it.

On the other hand, I doubt that a competitor could learn anything significant from using BK. Actually, that's not quite true. I doubt that a competitor could learn anything more from personally using BK than from reading Documentation/BK-usage and similar documents. I'm fairly certain that everything that is really clever about BK would require as much effort to identity as something that could be reverse-engineered from BK as it did to work it out in the first place. You are, after all, constantly having to tell people that there are hard problems in version control; these people are hardly going to identify the hard problems and the solutions to them simply be using a tool which hides from the user just how hard the problems are. Linus has, at this point, used BK probably more intensely than anyone who doesn't work for BM (and quite possibly anyone who does). Linus is good enough to write a C compiler just because it seems like a good idea. Linus clearly cares a lot about version control and having it work right. But Linus hasn't decided to write his own version control system. Chances are that, if he hasn't done it, nobody else will.

A number of years ago, I used Purify from Rational. It was pretty easy to tell what it was doing from what it produced (plus, there were whitepapers provided which explained a bit more). On the other hand, there was absolutely no danger of anyone reverse=engineering and reimplementing it from using it (Years later, valgrind came out, which has a similar effect but using a very different method). Rational was perfectly willing to let whoever wanted to buy and use Purify to work on whatever they wanted. For that matter, they even gave me tech support, despite me using a trial license at a point when my employer's license hadn't been renewed (they were fanatical about the quality of the software; I was just telling them that -fomit-frame-pointer should be mentioned as a reason you can't debug a program; they wanted to make it work for my case). As far as I can tell, Rational did well on Purify. So there's at least that example of a successful model: let anyone of no commercial significance use your software for free; let anyone whose budget is noticable buy a license for a lot of money; don't worry about the competition. Just focus on getting as many people as possible, whoever they are, to pay you and your developers.

With 20/20 hindsight, I bet Google would now be even more thrilled with BK developed at Google and released under the GPL than they are with Orkut. Too bad there wasn't any way at the time to know it would work out that way...

Following up after a day in the shop

Posted Sep 27, 2004 7:21 UTC (Mon) by darthmdh (guest, #8032) [Link]

Bravo! When BK first appeared I loved the fact they made it open source, it meant if I ever found any bugs I could fix them and contribute this back, or merge in fixes from others without having to rely on a vendor release (since most commercial software houses like to roll up a bunch of stuff over time and their support track record was yet to be proven). This is what open source is all about. Not that you get something free of charge (which so many people mistakenly believe), or even that you can rip off other's ideas - essentially fork a codebase and try to "compete" (be it financially or simply in mindset). Of course, like most ventures there's always some village idiot who wants to spoil things for everyone else, and so it was with BK and Bitmover were forced to make it more and more closed. Unfortunately by some of the very people they were trying to help by having it free in the first place. People who want to rip off open source software simply for the credits are IMO no better than software pirates who want their credits on their 0-day warez. Open source is about improving the open source software, not providing 50 billion nearly-identical alternatives to what currently exists, with someone else's name/brand on them, and nothing improved that you would write home about.

I buy software because its a tool that helps me solve real problems. Little annoys me more than computers that make tasks slower than doing it manually - what's the point? Why should I bother with arch (for example) when 6 months down the track I'm still not able to do any useful work with it because I still don't understand how the heck it works, its poorly documented, the syntax is woeful and confusing, and the mailing lists are full of egotists that want to enforce this "little boys club" mentality? Bitkeeper had a bit of a learning curve (mainly getting my head around the ChangeSet versus revision thing - I was used to CVS and PRCS) but it was well documented and the support is amazing. And I say that as a gratis user - what they must do for their paying customers blows my mind. I deal with a lot of commercial vendors and there are few that come even close - to *paying* customers. That being said, to compare the two, BK improves my productivity and arch kills it dead and then some, when I'm writing code. I don't use an SCM tool because its a cool thing to do or the latest buzzword, or is licensed to me under the terms of the LLOTW (Libre License Of The Week). I want a tool that will assist me with the *real* task at hand - providing decent software in an agreeable timeframe. Same goes for Window Managers, editors, debugging tools, command shells, you name it.

Don't get peeved at Larry because he had to make his baby closed, get peeved at the morons who forced him to do that. They're the real enemy of open source.

Following up after a day in the shop

Posted Sep 27, 2004 10:06 UTC (Mon) by hppnq (subscriber, #14462) [Link]

Larry, I'm with you on this one, setting aside some of the practical stuff (how you run your business is entirely up to you).

I think that it is a shame that releasing BitKeeper under the GPL does not work, and I think that members of the Open Source community should be able to discuss this problem from a pragmatic point of view. We're trying to solve a real problem here, and solving real problems usually calls for compromises. Stuffing Open Source down people's throats as an attempt to rid the world of non-free software (or at least the problems associated with it) is just not going to work. Nor is saying: forget the GPL because there are some people out there trying to rip me off. (They will rip you off anyway.)

I didn't mean you should just GPL BitKeeper, I was saying that there are already several hybrid solutions that might also work for you. Think Red Hat or MySQL. Companies are not waiting for a hairy hacker who might help solve a problem, they want the full blown support, added value and services. (Mind you: this is *not* my point of view, I think the hairy hacker solution is brilliant. ;-) It's not a question of giving everything away and hoping that some lunatic will still pay you for it -- it is a question of deciding exactly what it is you're selling. Given the fact that you want people to be able to use BitKeeper, and still make money off it, the hybrid model just seems to be the most appropriate.

Use the strength of the GPL. Your customers might be interested to know that, if you are gone or they want to switch to another system, they do not run the risk of losing their software, data, or investments therein. Like someone else said here, who needs another superior SCM system anyway, if we can have a GPL'ed BitKeeper? The risk that you lose against a competitor seems to be much higher if you don't GPL BitKeeper, because people will start scratching their itches.

(Not every avid Open Source supporter is a fanatic, by the way, though I imagine your path has crossed some real nutcases over the years. I would advise you to ignore them so we can have a normal discussion. There is no reason to throw at me what someone else might say or do. ;-)

Following up after a day in the shop

Posted Sep 27, 2004 23:58 UTC (Mon) by lm (guest, #6402) [Link]

The mill is way nifty and if you are local to the Bay Area drop me a mail and you can come over and get covered in shavings too :)

Following up after a day in the shop

Posted Sep 27, 2004 11:15 UTC (Mon) by lolando (subscriber, #7139) [Link]

A few points:

> And the bit about me not being part of the open source community warrants
> comment. That hurts. I've been a long term member of this community [...]

Yeah, you as Larry McVoy, author of free tools at some point in time. I don't know them, but I have no reason to deny you any credit for them. You as BitKeeper's keeper seems to be a different matter.

> once upon a time BK was open source with a license that said "you can't
> remove the openlogging part"

Doesn't seem to be open source to me. Open source is not just about seeing the code. I agree people who did remove that stuff did violate the license, though, but don't call the license open source while it's not.

> BK exists to help the FLOSS community

The way I see it, BK exists to provide funding for BM, which in turn devotes resources to pay engineers to 1. improve BK, 2. provide support for it. BK helps (some) community members do stuff (the second S in FLOSS), but as far as I can see it doesn't help the "FLOS" part.

Oh, and refusing to sell a product to a customer (for whatever reason) is such a kind of "Uh, you what?" that it shouldn't be necessary, but you know us French, a law for everything and all, well, it's forbidden in France. Never thought that would have any actual influence on any but theoretical cases, seems I was wrong. Oh well. I suppose you'll have to do without the patronage of my (rather large) company then, as I've already written my first patch to Arch (and will probably write others).

Following up after a day in the shop

Posted Sep 27, 2004 19:39 UTC (Mon) by kevinbsmith (guest, #4778) [Link]

Larry,

I appreciate your thoughtful response. Other people have already addressed most of the points, but I wanted to add just a couple notes.

Perhaps Bitmover doesn't change the wording of the license that often, although yearly significant changes seem somewhat frequent to me. However, it seems to me (and I could be wrong) that the interpretation of the license changes more often than that. It seems like every few months I hear about another person or group who is somehow excluded from being able to use BK. For example, I was not previously aware that small businesses who employ advocates of FLOSS BK competitors would be denied the option to purchase BK licenses.

The scariest part of the licensing is that someone who already has a license can have it suddenly pulled from them. With most software, once you have a license for a particular version, you can at least continue using that version forever, as long as you obey the license you accepted when you started using the software. With BK, it seems that is not the case, based on what I have read.

As far as your personal FLOSS contributions: I have not followed your career outside Bitmover, so I was unaware of your GPL work. For whatever true free/open software you have written, tested, or documented: Thank you. Seriously. And thank you for doing what you think is best for the Linux kernel. In contexts other than BK, you may indeed be a valued member of the FLOSS community, and I did not intend to take that away from you.

So let me rephrase my statement as: I still believe that in the context of BK, Bitmover is not part of the FLOSS community, any more than other proprietary companies who merely provide gratis tools (and there are many). In fact, most such companies do not discriminate against specific FLOSS contributors or advocates.

Following up after a day in the shop

Posted Sep 27, 2004 23:48 UTC (Mon) by lm (guest, #6402) [Link]

Hi Kevin,

It's a pleasure to actually have a conversation rather than a flamefest, thanks for doing so, I mean it.

On the topic of excluding people, this is just FUD raised by people who want everything in the world to be GPLed. Other than Zenaan, in our entire history I am aware of exactly one other person who we excluded, Ben Collins, and we later worked out an agreement with him and he's off using BK.

It's true that we could use the license as a baseball bat but it's also true that we don't do that. Lots of people at Red Hat use BK and have for years, in spite of the fact that in theory they are not allowed to do so. If one of them started working on arch and we saw arch getting BK features one by one then we might do something about it. But they haven't and so we don't bother them.

Bottom line: all the fuss that has been made about the non-compete clause excluding people is just FUD by the GPL police. All sorts of people use BK, we've counted more than 10,000 active branches of the Linux kernel in BK and even if each developer had 10 branches that's still 1,000 developers. I'm sure you could make a case that we could exclude lots of those developers but that's missing the point. The point is that we don't. Not unless one of them tells us that they are going off to copy BK. I wasn't involved with Zenaan but I got the impression he didn't leave us any choice.

As for pulling the license, again, we could but we haven't. What the license says has to do with support, not beating people up. We have more than 50,000 free users - that's a lot of support and as others have pointed out, we take support very seriously. The deal we want is that you stay current so that we don't have to handle the same old bug 50,000 times. The license says you have to upgrade when we issue a new release that has regression tests that wouldn't pass on the older release[s] (which is pretty much true every release, we have a very extensive regression test suite). While we have the legal right to force you to upgrade, we have never done so. People are aware of our reasons for wanting you to stay current and by and large they respect that.

It's worth noting that all the FUD spread about our license is just that, FUD. While we believe we can make our license stick if need be the truth is that if the license is genuinely unreasonable you can take us to court and win, the courts will side with you if we are being jerks. That doesn't mean you get to win because you think I'm a jerk, it means you get to win if we are not allowing you fair use or reasonable enjoyment or whatever the legal mumbo jumbo is.

Finally, your last comment is the one I'd really like to address. It's most unfortunate that you feel that way because BK exists simply to help the Linux effort, specifically Linus. Joe Buck suggested that I did that as a marketing move. There are two flaws with that theory: it doesn't account for the 5 or 6 years that I worked with Linus on Linux before writing a line of code on BitKeeper and it doesn't account for the fact that I couldn't market my way out of a wet paper bag. Look at me, I can show up with a free $100 bill and manage to piss you off in the process of giving it to you. And I'm the guy that came up with this brilliant marketing plan? If I'm a marketing genius my goal must be to piss off the world because I do a great job of it. Face it, the "BK + Linux == marketing" theory doesn't hold water and anyone who has been around for the last ten years knows that. The idea of me in marketing is a joke, a bad joke.

BK was created to help you. I'm not looking for your thanks or gratitude but I am looking for you to realize that what I'm telling you is the truth. I picked the path where I could make the most positive difference to the open source world. I'm a good engineer, maybe a great engineer, but there is no way that I'm as good as Linus+Alan+Dave+Greg+Andrew+.... If I can help 1000 engineers be 5% more productive that's a 50x more good than I could have done by writing kernel code. You don't like that BK isn't GPLed. What you don't get is that a GPLed answer wouldn't have helped. It costs way too much money to do something like BK, there is more than 100 man years of top quality, full time, 60-80 hours/week, engineering in BK. You can wait for an open source replacement but you will wait for a long, long time. On the other hand, BK is here right now, helping, and has been for years. And there is no replacement for it in sight.

I've been saying the same things for years, this stuff is hard, it costs a lot of money to get it right, and there is a big payoff for the end users. Look at http://www.bitkeeper.com/press/2004-03-17.html and you can see the payoff, nobody disputes that. So *you* are benefitting right now by the use of BK on the Linux kernel, on MySQL, etc. Maybe some day there will be a suitable GPLed replacement but right now there isn't. And BK is helping the development of lots and lots of open source.

Can you see how perhaps I might not agree with your view that we're not part of the FLOSS community? Maybe you still disagree with me but that's not going to help you, it's going to hurt you. You are rejecting people who are trying to help because they don't want to help in your narrow definition of help. That's unfortunate.

Following up after a day in the shop

Posted Sep 28, 2004 8:07 UTC (Tue) by lolando (subscriber, #7139) [Link]

There's a difference between "helping the FLOSS community", which I think nobody can reasonably deny BK the software and BM the provider of hosting are doing, and "being part of the FLOSS community", which is a criterion only the license can fulfill.

Following up after a day in the shop

Posted Sep 30, 2004 17:07 UTC (Thu) by rgoates (subscriber, #3280) [Link]

Larry, thanks for taking the time for this discussion. I've found it
very educational.

Following up after a day in the shop

Posted Oct 1, 2004 1:40 UTC (Fri) by lm (guest, #6402) [Link]

My pleasure, hopefully it's been helpful. It's definitely not an easy path that we've choosen, lots of people are very unhappy BK is used in the open source world at all, they view it as a failure of the open source model and a very visible failure at that. I can easily see that people would take the position that they should use the open source stuff and if it isn't good enough then they should fix it. Makes perfect sense, I'd argue the same thing in their shoes.

However, this particular product space is really difficult. Every engineer thinks they can do a better job in a few weeks with some scripts and CVS. Unfortunately, that's just not even close to being true. I've done this before and I knew how much work it was before I started again. And to tell the truth, I had a little of that "oh, this will be easy" feeling. BitMover was created to do Linux clusters, not source management. I told my people "we just have to hack out this little source management tool for Linus and then we'll get on the clusters, OK?" Yeah, right. It became clear in about of 6 months that this was a far bigger effort than I had anticipated and we had to have a business model or we'd never get to anything good.

If there had been any way I could have done this as a GPLed product I would have. I get it. I get it more than most of the people flaming me. But even if I were wealthy enough to do this for free (which I'm not) it wouldn't have worked. This job is a lot bigger than me, a lot bigger. I needed help and in order to attract that help I needed something more than my stellar personality and the possibility of helping the OSS world. People wanted to get paid, they wanted stock options, and they wanted those options to be potentially worth something. You just can't make that happen with an open source product. But what about Red Hat? Red Hat isn't an open source product, it is about 1000 open source products, and Red Hat wrote less than 1% of those products. Red Hat is a packaging company by and large. We aren't. We have one product and the support model doesn't work for us, the packaging model doesn't work for us, that was obvious. We needed an answer which generated enough revenue to pay us to make a better product for you. And for our commercial customers.

We had to have a commercial model that generated revenue. It sucks, all of us would rather be hacking away on GPLed source, that would be cool. But we have to eat, we have to support our families, and most importantly, we have to do a good job for you. We simply can't do that with an open source business model.

Following up after a day in the shop

Posted Oct 1, 2004 10:18 UTC (Fri) by hppnq (subscriber, #14462) [Link]

Larry, thanks for your time explaining all this. It is too bad that an Open Source business model doesn't cut it for you, I really believe that you are missing out on a lot of the good stuff that Open Source can do for you. But I am aware that you know this. ;-)

So, I wish you all the best! Thanks again.