LWN.net Weekly Edition for January 31, 2008

Ten-year timeline part 4: the end and the beginning

When your editor started this series, the idea was to have four installments covering the ten-year life (so far) of LWN. Well, this is the fourth installment, and it gets less than halfway there. This is not, it seems, a topic which inspires brevity. So this series will continue past the anniversary, though your editor anticipates picking up the pace a bit for the second five years. There is less to be learned, arguably, by looking at events in the relatively recent past.

Anyway, at the end of the third installment, LWN had been unacquired by Tucows and was, once again, on its own. The worst of the dotcom bust may have passed, but it was still a somewhat scary environment in which to be attempting to restart a business. It was, in fact, even scarier than we had thought when we so naively set out to show that we could do a better job of bringing in the cash than Tucows did.

• February 7, 2002: Linus tries BitKeeper at last.

• February 14, 2002: Sun states that it will "ship a full implementation of the Linux operating system." Dave Whitinger joins LWN.net.

Dave Whitinger was, of course, one of the founders of LinuxToday. He joined LWN with the intent of helping us develop the advertising side of the business. That did not work out as intended, but it is hardly Dave's fault; it was a terrible time to be trying to sell advertising.

• February 28, 2002: Sun cuts off free access to StarOffice, but we had OpenOffice.org by then and didn't mind. BitKeeper starts to settle in as the kernel's source management system.

Linus stuck with BitKeeper after his initial trial, setting a number of things in motion. For the next few years, the use of proprietary software at the core of the kernel development process would be a constant source of unhappiness and worry - and, in fact, the story had just the sort of unhappy ending that some observers had feared. But this was also the move which rationalized the kernel work flow and made the whole system scale; the incredible rate of change we see now would not have been possible without it. The use of BitKeeper also made the community aware of what distributed source control could do and, eventually, inspired the creation of a number of free programs with the same essential features. One could say that the community would have eventually developed these systems on its own without the push from Larry McVoy and BitKeeper, and that's probably true. But the fact is: we didn't do it at that time, so we had no real alternative to BitKeeper.

• March 7, 2002: Martin Dalecki's "IDE cleanup" patches start to raise concerns among kernel developers, who have this strange notion that their disks should actually work. A petition against the use of BitKeeper circulates on the net. Eric Raymond goes around telling the world that the kernel development process is "in crisis."

• March 14, 2002: Richard Stallman claims that the GNU HURD will be ready by the end of the year. MandrakeSoft pleads for donations to keep the business alive - and LWN does too. Martin Dalecki officially takes over IDE maintenance - and breaks more systems.

We got about $5,000 from our initial plea for donations. It was a real act of generosity on the part of our readers, but one does not keep a business with five employees going for very long with that sort of money.

• March 28, 2002: The proposed "consumer broadband and digital television promotion act" would require DRM technology in all software which touches digital media. Lineo lays off more staff.

• April 25, 2002: More BitKeeper flames. Lineo goes through a "recapitalization" effort to be able to do things like pay its employees.

• May 2, 2002: OpenOffice.org 1.0 is released.

• June 6, 2002: LWN switches to the "new" site code. Red Hat applies for a few software patents. ADEOS, a real-time system which avoids the RTLinux patent, is released. UnitedLinux launches. Mozilla 1.0 is released.

It is amazing how many readers hated the new code. Certainly there were a lot of silly things in the initial version of the site; we fixed a number of them in a hurry. Many readers disliked the ability to post comments - often posting comments to that effect. The addition of comments was something we thought about carefully for a long time; we were quite concerned that they could ruin the feel of the site. In the end, it seems, trusting our readers has paid off; the quality of the conversation here is often quite good.

UnitedLinux was a cooperative effort between Caldera, Conectiva, SuSE, and Turbolinux; the idea was to join together to create a common base from which each could then craft a separate product. The effort was never all that successful, and the presence of Caldera would, of course, doom it outright in the end. But it was a big deal at the time. It is interesting to see that Mandriva (despite MandrakeSoft's refusal to join UnitedLinux) and Turbolinux are now attempting a very similar sort of arrangement.

• June 13, 2002: Secure Computing Corporation claims patents on SELinux.

• June 27, 2002: The 2002 kernel summit sets October 31 as the date for the 2.6 feature freeze. GNOME 2.0 is released.

• July 4, 2002: Darl McBride takes over at SCO.

• July 25, 2002: LWN announces "the end of the road." The "IDE cleanup" patch series (up to number 100) causes system lockups and file corruption. Debian GNU/Linux 3.0 ("woody") is released. Version 1.0 of the Ogg Vorbis codec is released.

By the end of July, we had come to realize that the advertising business was not going to work out for LWN, and we were short of other ideas. The bank account had reached a point where we could not pay even very small expenses. So we concluded that it was time to throw in the towel and try something else - though we had no clue of what "something else" might be. It was with a heavy heart that we announced our plan to shut down the site.

What happened next is that our donation box, which had sat mostly empty after the initial announcement, was suddenly topped up to the tune of about $35,000. Many of the donations came with notes to the effect of "use this to throw a big party." This, shall we say, got our attention. We decided that, just maybe, the subscription idea was worth a try after all, and decided to make a go of it. It was not the end after all.

• August 1, 2002: A new beginning. HP tries to use the DMCA to shut down disclosure of security holes.

• August 15, 2002: Distributions from MandrakeSoft, Red Hat, and SuSE are certified to be compliant with the Linux Standard Base.

This was when our credit card merchant bank at the time decided that all those donations might just be fraudulent. So they seized the money back out of our bank account. That, too, got our attention. It took a few months and some lawyer time to get the money you all had sent in our direction; during that time, it was money from PayPal (the subject of everybody else's horror stories) that kept the lights on while our main source of cash was blocked.

Needless to say, we got a new merchant bank, which we still use to this day. The new bank exhibits a rather higher clue level than the old one did, but we also learned a valuable lesson: don't mess with the credit card money pipeline. Every now and then, somebody asks why we don't accept pure donations; this is why.

• August 22, 2002: Martin Dalecki quits and the entire series of 115 "IDE cleanup" patches is deleted from the 2.5 kernel.

• August 29, 2002: British Telecom's attempt to patent the web dies in court. The BitKeeper license changes. Caldera becomes the SCO Group.

• September 12, 2002: Some patches get dropped after Linus starts running his mail through a spam filter.

It's hard to believe that, only 5+ years ago, somebody with an email address as well distributed as Linus's could get by without spam filtering. There are a lot of free "productivity" applications, but, arguably, few have actually increased productivity to the extent that SpamAssassin has.

• September 26, 2002: The first development release of the "Phoenix" browser is announced. UnitedLinux upsets the community by releasing a closed beta.

Phoenix was the Mozilla Foundation's answer to (relatively) lightweight browsers like Galeon, which had managed to turn the Gecko engine into something which was truly usable. The Phoenix browser proved popular, and eventually became the tool now known as Firefox.

• October 3, 2002: The first subscriber-only weekly edition. Eldred v. Ashcroft is argued in the U.S. Supreme Court.

Eldred v. Ashcroft, argued by Lawrence Lessig, was an attempt to roll back copyright extension in the US; it eventually was unsuccessful. To this day, there still has not really been a successful challenge to the extensions to copyright passed over the last few decades - though some especially nasty attempts to make things even worse were defeated.

With the October 3, 2002 edition, LWN adopted the new policy of requiring subscriptions in order to read our original content prior to the publication of the weekly edition. That policy has stayed essentially unchanged since then, despite the occasional temptation to increase the subscriber-only period. Subscription rates have also stayed unchanged, even though raising them is also tempting.

Subscriptions have certainly been successful, in that they have kept the operation going in the years since then. And there is a real joy associated with being truly answerable to our readers instead of advertisers. Nonetheless, it is a challenging business; people do not like to pay to read web-based content. The fact that so many of our readers are willing to do so is most gratifying. Trends in other parts of the net are moving away from this approach, though, with formerly subscription sites moving to pure advertising models. So it will be interesting to see how it all plays out in the future.

Meanwhile, next week's installment will look at how things went for Linux (and LWN) starting toward the end of 2002. Stay tuned.

Comments (39 posted)

LCA: Bruce Schneier on the two sides of security

The conference portion of linux.conf.au opened on Wednesday morning with a keynote by Bruce Schneier. LCA is a sold-out event; in fact, there are rather more attendees than can be fit into the hall where the keynotes are held. Thus the room was packed, with the second-class citizens - those with yellow badges who put off registration until late - watching a remote feed in a separate room. Those folks may have had a more distant experience, but it was almost certainly a cooler one too.

Bruce's key point is that we need to rethink how we try to achieve security, though it took a while to explain just why that is. Security, he says, has two components:

• The feeling of security: that which helps us to sleep well at night.

• The reality of security: whether we are, in fact, secure.

These two aspects of the problem are entirely separate from each other, but they both have to be addressed if our security goals are to be achieved.

Security is always a set of tradeoffs which we are all making every day. As an example, consider that, in all likelihood, nobody in the audience was wearing a bulletproof vest. It's not that the vests do not work; instead, nobody feels that the cost of wearing a bulletproof vest is justified given the risk. On a bigger scale, the answer to the question of how to prevent more 9/11-like attacks is clear: ban all aircraft. In fact, that was done in the US for a few days after those attacks, but, in the longer term, that is not a tradeoff that people are willing to make.

So the fundamental question for any security tradeoff is: is it worth it? As it happens, we are quite bad at making that decision. We tend to respond to feelings rather than reality. Spectacular risks drive us more than everyday risks. We fear the strange over the familiar and the personified (think Osama bin Laden) over the anonymous. Involuntary risks are seen as being bigger than those entered into voluntarily. In the end, evolution has equipped us quite well for making tradeoffs in the small communities we lived in many, many thousands of years ago. We are less well equipped for the world we live in now.

Since we respond to feelings more than reality, there are strong economic incentives for solutions which address feelings. The result is snake-oil products and security theater. Sometimes people notice that they are being sold bad security (later Bruce mentioned a US survey which indicated that the Transportation Security Agency is now less trusted than the taxation agency), but, all too often, they don't. They have a poor understanding of the risks and the costs involved, and there are plenty of people with strong interests in confusing the issue.

The security market is a lemons market, one where buyers and sellers have asymmetric access to information. Economic research shows that, in such markets, the bad products tend to drive the good ones out of the market. There is no easy way to evaluate the work which has gone into the creation of a truly secure product, so buyers respond to other, less reliable signals. Things like price, sales claims, or the Gartner Group. These signals are sloppy and prone to manipulation. When security is outsourced to outside agencies - governments, say - the problem gets even worse.

In the business world, information eventually brings some order to a lemons market. As businesses learn about what really works, access to information evens out - though there is always a problem with very rare, high-cost events where information is not available. In the individual world, though, it is much harder, because fear plays a much bigger role.

The fact of the matter is that fear is wired deeply into how we work - it is a result of a very old part of our brain. As humans, we have the ability to override our fears when reason indicates that we should, but it is a hard thing to do. The default state is that fear rules. So this is Bruce's core point: the feelings matter. All that security theater out there is not entirely stupid; any security solution must address the fears that people feel. We must address both aspects of security.

The problem is where the feeling of security and the reality of security diverge from each other. If only feelings are addressed, security has not really been achieved. If only the reality of security is addressed, people feel insecure and may make bad decisions. Either way, the full problem has not been solved. Addressing this all-too-common problem is hard, though; Bruce knows of no better way than the spreading of good information.

Your editor's perspective follows - nothing from this point on was said during the talk. It seems that he has a point here. Consider some common situations in the free software world:

• A large number of security updates from a distributor may be an indication that the reality of security is being achieved: problems are being found and fixed before they are exploited. But all those updates can undermine the feeling of security. The seemingly endless stream of Wireshark updates is a case in point; most of these problems are found through proactive auditing by the developers and have never been exploited by the Bad Guys. But the feeling of insecurity associated with Wireshark can be strong. This feeling can push users toward other software which, while not having that long history of security updates, is actually less secure.

• A system running SELinux may, in fact, be highly secure. But many administrators still turn it off. SELinux does not make them feel secure because they do not understand it, and they fear (rightly or wrongly) that it will interfere with the proper operation of the system. But, by turning it off, they undoubtedly expose themselves to a number of attacks which SELinux would block.

We should hear Bruce's point and think a bit more about how we can ensure that free software creates the feeling of security - but a feeling which is backed up by real security. It's a hard problem, one which lacks technical solutions. But we'll find ourselves less secure than we would otherwise be if we do not address that side of the issue.

Comments (17 posted)

A ten-year retrospective from LWN's other co-founder

Hello to all LWN readers! For the tenth anniversary of LWN, I've been dragged out of my closet to say a few words. Am I stunned that LWN is still going after 10 years? Not really. Much more stunning to me is the realization that the number of years LWN has been published without me are now almost double the number of years it was published with me. That is much harder to get over. As a result, all new readers from 2002 on have no reason to know who I am or what I've written in the past. For those of you that remember me and have asked about me, thank you and rest assured that I haven't forgotten you either.

My name is Elizabeth Coolbaugh (Liz) and I was there for the very first issue as well as many issues that followed in 1998 through 2001. I've always said it was the very best job I ever had. I wish for all of you, if you haven't experienced it yet, a job where your first weeks of work are greeted with happy, enthusiastic letters. As the years went by, letters of praise, though much sparser, never totally ceased. You couldn't have a better incentive to work harder and harder!

Jon has done an excellent job of going over the history of the first few years already, so all I can add is some tidbits or personal viewpoints. I'll mention that for me, the start of LWN was actually back in the early 1980's, when Jon, Becky and I came together as a programming team in the then infamous "Assembly Language Programming" class offered through the Engineering School at CU Boulder. We got a chance to experience lots of late nights, interesting hardware experiences and how to keep going with pizza, chocolate, caffeine, etc. That is a good way to get to know your future business partners. Jon and Becky never let me down and we all found different strengths to add to the mix. Forrest was around, too, though not working with us directly at the time.

Jon mentioned that I was between jobs at the time we began. In fact, I had left NCAR three months pregnant. I loved working at NCAR for many, many years, but I had always said that I would leave it when the work stopped being fun. It actually stopped being fun about two years before that, but I had weathered rough times before and waited to make sure the situation wasn't going to turn-around before choosing to move on. The challenge of a new baby on the way (and the continuing challenge of the Multiple Sclerosis that eventually led to my departure from LWN) finally made it "the right time".

So I'd actually had most of a year off to recuperate, re-organize, have a baby and test the job market waters. What I wanted was a job that used my professional skills and yet was part-time, to help me keep the health I'd regained. What a pipe-dream! Companies that would have gladly recruited me full-time just tossed my resume into the nearest recycle bin. The nicer ones told me to go out and find someone else with identical skills who wanted to job-share a full-time job and they would be willing to consider the possibility. Not bloody likely.

So when Jon and I were having lunch and he suggested we might be able to work together to create something giving me what I wanted and allowing him to eventually leave NCAR, it seemed to be the right idea at the right time. I never regretted the decision, but in fact, I had a full-time working spouse to cushion the decision. Brandon's reaction (my husband) to becoming the sole support of the family and a new father in one fell swoop was a little different -- much like a deer full-blinded by headlights.

In the spirit of true confessions, though I had fifteen years experience in the computing field and had worked with many different operating systems, VMS and Solaris being primary, I'd never actually touched a Linux system. Jon's unwavering belief in my ability to pick it all up in a heartbeat was both daunting and encouraging at the same time. So I installed my first Linux system only three or four months before we first started publishing. It did give me a fresh, unbiased view of the whole community, though. Okay, not totally unbiased. I did sit on the emacs side of the whole emacs/vi war.

To get started, I subscribed to say, a hundred different newsgroups and mailing lists full of people I'd never met, topics I'd never heard of and flame wars I didn't care to read. It was truly a new skill to develop to learn to skim through them searching for the topics people cared about, the posts that actually carried real information and gently lift each little kernel of "news" out and place in into the newsletter, then wait to hear how well I'd done.

The response was totally overwhelming. I will never, ever forget the emails we received those first couple of months. New people were finding us each week and so the responses kept coming in. They drove me to try and make my contributions worthy of the praise they sent. It is because of those emails that I'm not surprised LWN is still out there today. People wanted and needed what we had to offer. Jon's vision of what people liked and wanted has always been clear and that is another important piece of why LWN is still going strong.

My take on the Red Hat Support fiasco: I have no hard feelings. Although my work as a systems administrator had always included supporting people and I had enjoyed the interaction, I had no idea what I was getting into offering 24 hour support from my home. Just as my daughter was getting old enough to give me a full-night's sleep, I was getting phone calls at 2am and 3am, having to wake up to a fully alert state and go into emergency fix-it mode. I'm surprised I survived until all the contracts we had sold finally expired. In the long run, Red Hat's ideas gave us the courage to start our own business and since writing for LWN was what I learned to love, I consider the end result to have worked out for the best. I also carefully noted for the future that telephone support work was definite going to be a last resort for any future career moves.

Meanwhile, since the few contracts we had didn't bring in enough to pay the bills, let alone enough to support Jon's full-time entry, I also did contract work as a technical writer, remote or on-site administration of Linux for some local companies and I don't even remember what else. Eventually, Jon had to take the risk, forgo waiting for a reliable income and quit his day job in order to increase the income stream. Note that his early work on LWN was always done in addition to continuing his full-time job and trying to increase our income stream at the same time. No wonder he got grumpy if I was out sick or worse, got to head to a fun Linux conference, leaving him to pick up the slack! Of course, it was terrifying in turn for me when the situation reversed and Jon was unavailable. Picking up the kernel page for the week? Ack! I didn't usually complain. Instead, I kept my head low, worked hard and hoped not to see too many corrections or criticisms come in.

It was wonderful for both Jon and I when we were finally able to add Becky to the mix. I think initially we were only able to scrape up enough to pay her for 10 hours a week, but every hour helped. I haven't forgotten, Becky (okay, it should be Rebecca, but she'll always be Becky to me), the hours you put in at a very low rate of pay. Of course, we did pay you first -- the downside to being the business owners for us.

Over the course of the next couple of years, we continued to bring in our income from other sources. We did actually initiate putting some advertising on our site and it brought in a tiny amount of money, but the bread and butter of the company continued to be contract work done in addition to the weekly publication. That included our most successful side foray, building and teaching Linux classes.

What else did I love about LWN? I so enjoyed the friendships I made throughout so many different communities. Will Rogers once said he never met a man he didn't like. Well, I've met many! But truly, in all the years I worked for LWN, I never met anyone I didn't like. Sometimes people I liked said things or did things that I didn't like, but underneath it, they were all good people, smart, idealistic and very strongly opinionated. That was part of what I liked and enjoyed, so I never held people's opinions against them.

The conferences I attended and at which I spoke were like the icing on the cake. I got to meet in-person people I had only come to know through newsgroups and mailing lists or occasionally personal correspondence. I got to meet even more people and share in the excitement. And yes, I do remember the late nights going out for food, drink and conversation with you -- the Atlanta Showcase, LinuxWorld San Jose, Embedded Systems Conference San Jose, LinuxWorld New York, the Colorado Linux Info Quest and the Singapore Linux Conference. Each one provides me with rich memories. My trip out to Singapore was one high-point. So many good and wonderful people and such a wonderful experience. I thought it was to be the first of many international conferences that I would be attending and I am still so sad that it was my last. I particularly regret never making it out to any of early Linux conferences in India, despite invitations.

Professionally, though, the highlight of the work was actually developing myself as a journalist, rather than a computer expert. I enjoyed researching more in-depth articles. When rumors floated my way, I loved actually going out and contacting the people involved first hand by telephone -- short-circuiting email and the rest, to discuss the issues and get their first-hand viewpoints. Since our community wasn't exactly hounded by the media back then, everybody actually wanted to talk to me and was more than happy to give me the straight scoop, instead of just seeing themselves misquoted elsewhere the next day, with the resultant flames. Best of all, I was occasionally able to get the sources of both sides of a controversy together and talk. I can think of at least twice where problems got resolved as a result, people got together and I got the scoop on a story the next day that had literally changed as a result of my work. Very heady stuff.

Jon has already done an excellent job of covering our experience with the dot-com bubble, so I won't add to his description. It was truly a unique life experience that we enjoyed to the fullest, knowing that another like it was unlikely to come by us again. We were very fortunate in our decisions and I agree that the people at Tucows were extremely good to us.

Well, at this point, all this happened a long time ago. I had a great time and regret nothing I did, only the things I didn't get time to do. For those who have asked after me personally, be assured that health-wise, giving up my job was again the right choice at the right time and I'm doing much, much better than I was in August of 2001. You're still not likely to see me back any time in the near future. I focus my research skills now-a-days on tracking traditional and alternative medical discoveries, implementing what seems good to me and serving as an ad-hoc resource for other family members. Oh yes, and serving as a chauffeur to my daughter, who is now ten years old, just as LWN is. Take care, all of you, remember to be proud of what you are achieving and *always* have fun doing it. I stand by my opinion that when work ceases to be fun, it is time for a change.

Comments (12 posted)