LWN Weekly Edition
Front pageSecurity
Kernel development
Distributions
Development
Linux in the news
Announcements
->One big page
This page
Previous weekFollowing week
| |||||||||||||
Leading itemsLWN.net: a ten-year timeline (part 1)LWN is about to celebrate a birthday. Picking the true anniversary of an enterprise like LWN can be a bit tricky - there are many points which could be said to mark the true birth of the organization. After some thought, we have decreed that LWN.net was born on January 30, 1998. So we have a tenth anniversary coming up. That's a long time - far longer than any of us thought we would be doing this. Life is funny that way, somehow.One cannot let a date like this go by without at least partially taking advantage of its hype-creation possibilities. So there will be a few things happening to celebrate our decade of writing about Linux, culminating with some sort of celebration on the 30th, when your editor will be speaking at this year's (sold-out!) linux.conf.au in Melbourne, Australia. One of those will be a short series of articles - starting with this one - looking back at those ten years. What a long, strange trip it has been. Back in early 1997, your editor was the manager of a software development, system administration, and data delivery group at the National Center for Atmospheric Research. He had, at that point, been using Linux for a few years. It was running on a number of servers, of course, but we had also deployed it on desktops and used it for the acquisition and display of meteorological data, including high-bandwidth (for the time) doppler radar data. Don't let anybody tell you that real-time Linux is a new thing. At this time, your editor was seeing two futures: (1) an increasingly dilbertesque life spent mostly in meetings, and (2) the clearly bright future of Linux. So he was actively looking for ways to move out of conference rooms and toward Linux, and talking over schemes with a number of friends. An early idea - to commercialize one of the first weather stations ever put on the World Wide Web with LWN editor Forrest Cook, never quite took off. But that thought process continued. During that same time, Elizabeth Coolbaugh had just left a very similar position at the same institution; she was looking for a new project for the next phase of her life. After some discussions, Liz and your editor settled on a business idea which seemed to have some promise. It was not to be the last silly decision they were to make. You see, at that time there was a struggling Linux distributor named Red Hat which was beginning to get the sense that there might be a market for its boxed Linux product in the corporate world. But companies need support, and Red Hat lacked the ability to provide that support. So the company's management came up with the "support partner" concept. Upon being accepted into this program, partner companies would be able to sell Red Hat-backed support certificates, which Red Hat would help to market. This widespread network of Linux experts would be able to provide local support to clients and would, for the hardest problems, be able to get help from Red Hat itself. It looked like a winner for everybody involved. That program was not yet operational at this time, though - but Red Hat promised it would be Real Soon Now. Your soon-to-be editors, not yet having done much business with Red Hat beyond ordering an occasional CD, believed this promise. But it still made sense to do something productive while waiting. The idea that emerged after some talk was to put up a regular newsletter about what was happening in the fast-evolving Linux community. Even back then, keeping up with everything was hard, so we figured that the service would be valuable. As an added bonus, it would attract attention to this new support company (called Eklektix) and show just how blindingly smart and up on Linux we were. Discussion of details occurred slowly through much of 1997. On January 22, 1998, the first issue of LWN was posted; it talked about the 2.1.79 kernel, the brand-new spinlock mechanism, the devfs debate, the creation of Red Hat Advanced Development Labs, and attempts to bring Java to Linux. The January 29, 1998 issue changed the format and led off with Netscape's announcement that it would be releasing the source code for its browser. We also found all of two news articles about Linux (we posted every one we found in those days) and talked about NFS problems, the devfs debate, the Debian 2.0 release roadmap, and gcc 2.8 problems. At this point, we had posted two issues, but had not actually told anybody about them. Unsurprisingly, traffic was low. That changed on January 30, when our announcement made it out to the comp.os.linux.announce newsgroup - the best way to get the news out at that time. As promotional text the announcement was rudimentary at best, but it had the desired result - we got over 1000 page views on that first day, which seemed like a lot at the time. LWN was off and running. Some highlights from the early days of LWN:
It's fair to say that we didn't entirely grasp the significance of the events reported in the April 2 edition. The hiring of Alan Cox was one of the first in a long series - before then, almost nobody actually had a job which involved developing Linux. The Open Group's attempt to relicense X was thoroughly defeated by the existence of a free version with an active development community - a story which would be repeated a number of times in the coming years.
The Oracle announcement seems mundane now, but the existence of Oracle products for Linux was a specific indicator that many people were looking for. It was an indication that Linux was a "serious" platform. Richard Stallman, of course, thought that Oracle's announcement was terrible news.
Let us now pause for a moment. From this distance, it may be hard to appreciate just how big the news of the Red Hat investments was. For all that had happened, Linux was still a somewhat obscure phenomenon, unknown to much of the information technology world. When Intel put money into Red Hat, it became clear to all that both Linux and Red Hat were headed toward success. This was, in some real sense, the point where Linux entered the dotcom bubble, though the real action was still a year away. The 2.1.123 release failed to compile as a result of some merging errors; developers got upset about the state of affairs and a long, inflammatory discussion resulted. Linus stormed out of the virtual room and took a vacation. It was a somewhat scary series of events which foreshadowed more to come; getting the kernel development process to scale as the community grew was a multi-year process. During this time, LWN was also growing in both readership and size; it was taking increasing amounts of time. We eventually had to move the server from its initial location (behind an ISDN line in your editor's basement) to a proper hosting facility. But, remember, LWN was not the main endeavor; it was an attention attractor for the support services offered by Eklektix, Inc. This business plan was not going particularly well. Those who dealt with Red Hat in that era know that, as a company, it was a rather chaotic place. The marketing for the support partners never happened, and the backup services for the support plans the partners were able to sell themselves were, shall we say, less than the customers thought they deserved given what they had paid. The support partner program was not a big success for anybody involved. As a result, one of the first things Red Hat did with its new pile of cash was to cancel this program and start building its own, internal support operation. Eklektix continued to push its own support offerings for a while, but the fact of the matter is that it was not a fun business: it seemed to mostly consist of cleaning up after low-budget ISPs which could not be bothered to install security updates. So the search for alternatives began. Meanwhile:
About this time, Eklektix announced that its new line of business would be training - and Linux system administration training in particular. The announcement was timed for the first ever LinuxWorld conference; both LWN editors spoke there, with Jon delivering a system administration tutorial to 450 attendees. It was the start of a new phase - though it was not much more successful than the one which came before. If the investments in Red Hat were the beginning of the Linux bubble, LinuxWorld was where the inflation began in earnest. The amount of money on display there was impressive to say the least. The Red Hat party will live forevermore in the memory (or lack of memory, as the case may be) of all who attended. LinuxCare, which was supposed to be the big support success story for Linux, was unveiled at this conference. Never had there been so much overt commercial interest around Linux.
It is interesting to note that, during this time, LWN got its first acquisition offer: from Red Hat. We turned it down: the terms of the offer looked much like indentured servitude under firm Red Hat control. But we did work a deal with the company to supply news items for its portal site. Yes, during this time, Red Hat's business model was aiming toward becoming the dominant network portal for Linux-related information. Remember, this was 1999.
The Red Hat IPO was the beginning of a new phase: clearly somebody was making a lot of money from Linux, even if who wasn't exactly clear. What was clear is that Eklektix was not on the list. When we planned out the training offering, we had a set of spreadsheets with some truly wonderful numbers on the income which was sure to result. Somehow reality failed to match the spreadsheets. So we came to realize that we needed to look in other directions. At this time, advertising was beginning to bring in some actual money. But, more to the point, as the market heated up, companies were showing increasing amounts of interest in anybody who had any sort of Linux credibility or mindshare. We had some of that credibility at that time. So we decided to see what would happen if we let the word out that LWN was for sale. Suffice to say that the result was a far wilder ride than we could have ever anticipated. But that will be the topic of next week's installment.
Development issues part 1: Project communicationFree software projects, like all projects, live and die by their communications; developers must be able to talk to each other easily so that a consistent, coherent result emerges. But developers have differing ideas about what methods to use. A discussion on the Emacs development list provides a nice contrast between two of the main communications methods used by projects today. Traditionally, developer communications have been handled by the venerable mailing list, but that is changing, at least for some projects. Internet relay chat (IRC) has become the tool of choice for newer projects, which may leave those who are not inclined towards realtime communication out of the loop. Development methodologies are evolving, and some are adopting the new ways more quickly than others – some may never adopt them at all. The difference between communicating in IRC or via a mailing list is in some ways like the difference between text messaging and email. Email has its advantages, in that the recipient chooses the time to read and respond to the message, but it is often seen as slow. Text messaging or IRC have the advantage of speed; people receive a message and generally respond immediately. But that speed comes at a cost – interrupting the recipient. It also requires a full-time internet connection. While email archives are somewhat cumbersome to use, they are usable. IRC logs are exceedingly painful as they are not subject-based; they just cover a specific time span of all conversation on the channel. Email conversations may play out over days or weeks, but they are generally easier to follow compared to the multiple interleaved conversations that occur on IRC channels. It is in the nature of the medium: IRC conversations are meant to be used immediately, not reread weeks later. It is, in some ways, a culture clash. Younger developers tend to be more inclined towards realtime communications, while older hackers tend to be more comfortable with mailing lists. In what would seem to be an uphill battle, Eric S. Raymond has been advocating a more "modern" development style for GNU Emacs. His messages, appearing on Emacs-devel, champion a development style that includes IRC communication, a bug tracking system, and a version control system (VCS) more advanced than CVS. Raymond's experiences working with the Battle for Wesnoth development team exposed him to some of the newer techniques used in project communication, particularly IRC. He reached a somewhat surprising conclusion about IRC:
And far from finding I can't keep up, I've discovered that I like the
stimulation. I grok how the kids feel about this, because
mailing-list-only
projects have started to seem slow and boring to me, too.
The Wesnoth project uses IRC for all day-to-day design and development decisions, leaving the mailing list for more complicated discussions and white papers. This has the effect of excluding interested developers who are not able or willing to monitor an IRC channel throughout their day, but that is unlikely to be the intent. The reverse is also true: the perceived slow pace of mailing-list only projects has the effect of excluding those with a strong preference for a faster style of development. As Raymond shows, though, there is hope that members of one school can retrain – if they wish – for the other. While decision making by IRC does not seem to be in the cards any time soon for Emacs, an upgrade to something other than CVS seems to have gained more traction. Richard Stallman has been asking a lot of questions about git while other developers discuss other distributed version control systems (DVCS), like darcs, monotone, arch, and Mercurial. Raymond is working on a survey of the VCS landscape that, once completed, he and others hope will guide the project into a better VCS choice. One of the main DVCS features that seems of interest to Stallman is the "offline" capabilities. Having the entire history of a project and being able to do commits of work in progress while being disconnected from the internet are features that CVS does not have. Stallman is adamant that the tools used to develop Emacs be usable by those who are not always connected to the net which makes a DVCS rather attractive. The Emacs project is one of the oldest free software projects in existence; it is, like its founder, fairly resistant to change. While Emacs itself is used by hackers everywhere, it is increasingly falling behind its competitors, at least partially because of the slow pace at which it is developed. Raymond's belief is that by upgrading the tools used to take advantage of advances made since CVS and mailman were new, the time between Emacs releases could be reduced to something more sane. Doing that could go a long way towards making Emacs more relevant to younger hackers:
When
those Eclipse fans pointed and laughed because we're still stuck on
CVS and don't have a bug tracker, what counter could I have had? They
know these are bad choices and they know that I know it -- so when
they write off Emacs as old, tired, and irrelevant to anything they're
interested in, I find it increasingly difficult to reply.
It is unlikely that just some tool changes will be enough to resurrect the flagging popularity of Emacs, but there are hopeful signs. Some of Raymond's suggestions met a warmer reception than one might have expected. It is clear that a fair number of Emacs fans and developers are frustrated with the current state of affairs. It may be that "just some tool changes" are enough to reinvigorate the project to a point where it attracts more developers and users. That can only be a good thing for Emacs.
Development issues part 2: Bug trackingOnce upon a time, free software was a relatively rare commodity, and there was a real novelty in being able to run a free package for a specific purpose. The availability of a free C compiler, for example, was cause for celebration. The fact that said compiler was not always the most reliable program on the system did little to reduce enthusiasm; many of us persisted in irrational endeavors (like trying to use gcc to build the X Window System) despite the occasionally painful (and predictable) consequences. And, in the process, we helped to make both programs more reliable.There comes a time, though, when even the most die-hard free software proponent wishes that things would just work. As our software finds its way into more situations where failures are unwelcome (at best), the level of tolerance for bugs is falling. The desire for fewer flaws, however, runs counter to the desire for increasingly capable (and thus more complex) software. Somehow we have to find ways to simultaneously grow our systems and reduce the total number of bugs. To this end, a few projects have been having some interesting discussions on the tracking and fixing of bugs. As has been discussed in this companion article, Eric Raymond has been busily stirring up trouble on the Emacs development list. His point, deemed reasonable by your editor, is that Emacs must adopt a number of relatively modern development practices if it is to have any hope of remaining relevant at all. One of his key points is that Emacs needs to have a real bug tracking system. Says Eric:
Now I consider Emacs: 1100K lines, a COCOMO estimate of over 328
years, and no issue database. I think I think I understand much
better now now why the team has only been able to ship one release
in five years. Trying to converge on a releasable state with as
poor a view of the Emacs bug load as we have must be damn near
impossible.
While some of Eric's suggestions appear to be non-starters - imagine trying to get Richard Stallman to hang out on an IRC channel - the bug tracker suggestion might just go somewhere. Certainly it could only be an improvement for a project of that size to have some sort of idea of what the current list of outstanding bugs looks like. It might even help bring about another Emacs release before the end of the decade. Bug trackers are not a magical solution to the bug problem, though; in fact, they can create some problems of their own. The Fedora project, which does have a bug tracker, is currently trying to figure out what to do with the contents of that tracker. It seems that said tracker contains over 13,000 bugs, almost 10,000 of which apply to Fedora 7 and later. A bug database of this size is simply overwhelming to anybody who tries to do something about it. As a result, Fedora users are filing bugs, only to see nothing happen in response. Not even a "thanks for your report" message. This situation is discouraging for everybody involved, causing Fedora users to give up on reporting bugs and developers to fear looking at the tracker. In the Fedora case, there appears to be a near-consensus that the biggest problem is in triaging bug entries. This is not a job which can be automated; somebody has to go through bug submissions, weed out the duplicates, identify those which are really "features," figure out which developer should be notified, etc. Tying bug entries to those found in upstream trackers would be a highly useful bonus. Without this sort of effort, the bug tracker quickly fills with low-quality entries which help nobody. For the most part, nobody is doing this job for Fedora now. Red Hat is not paying for a staff member to triage bugs, and the wider community has not filled this gap. In the short term, any sort of solution looks like it will have to come from the community, so the Fedora folks are wondering what can be done to encourage more participation. Simply asking for help is the obvious first step, as is making sure that the process is easy. Then they may consider the tactics adopted by other large projects - Mozilla's policy of expressing its appreciation by sending a T-shirt, for example. As an aside, one of the more useful bits of information to come from this discussion was the existence of this family of URLs:
http://bugz.fedoraproject.org/<package-name>
Fill in the name, and the result is an immediate list of open bugs for the given package. Thus, for example, a visit to bugz.fedoraproject.org/gcc yields a list of compiler bugs. This result can be had directly from bugzilla, of course, but this interface is faster and easier. The Fedora developers have discussed a number of related issues, such as whether the Fedora bug database should be separated from the RHEL system and what can be done to make Red Hat better appreciate the value of doing more of its quality assurance work in the Fedora repository. But the core problem is just getting human attention applied to the bug reports. Digging through bug databases is a relatively unglamorous job; it is not an easy path toward rock-star hacker status. But it is an important and relatively easy way to help make free software better. Just in time to serve as an example of how well bug management can work, the GNOME project has posted its annual bugzilla statistics. It seems that over 110,000 GNOME bugs were filed in 2007, almost 109,000 of them were closed. The top bug-closers for the year were:
It is worth pondering for a moment on the amount of energy required to close over 14,000 bugs in a year - that's almost 40 per day, every day, without a break. This kind of energy does exist within our community, and some projects are putting it to very good use. While it is easy to get a contrary impression, the kernel does, in fact, have a bug tracker; there is also, in the form of Natalie Protasevich, somebody who handles the care and feeding of that tracker. But, as a recent episode shows, that still is not always sufficient to actually get the bugs fixed. On November 13, 2007, a bug in the SCSI subsystem was reported to the linux-kernel mailing list. It was put into the tracker as bug 9370 on the same day. Some developers looked at it over the next few days, but, even though a specific commit which appeared to cause the bug had been identified, no solution was forthcoming. Discussion eventually died out. At least until January 2, when Ingo Molnar decided to stir the pot by posting a patch to revert the seemingly guilty commit. At that point the discussion picked up and a reliable way of reproducing the bug was found. The commit which was said to have caused the problem was, in fact, not guilty; it had just caused an older bug to come to light. The discussion did not stop there, though. A number of charges went back and forth which do not require discussion here. But one core point is this: as long as the bug report sat in the tracker, nothing much appeared to be happening with it - though, it seems, the SCSI developers had not forgotten it and were trying to figure out what was really going on. But once the problem came back to the linux-kernel list in the form of a brute-force solution, the root cause was found in short order. The key here was bringing the problem to the attention of a wider group of people; the crucial recipe for reproducing the problem came from a developer who had not been looking at the problem previously. In the kernel context, at least, giving wide exposure to a bug often helps immensely in getting that bug fixed. That is especially true for the sort of hard-to-reproduce bugs which tend to come up in kernel programming. So, while bug trackers are a useful tool for ensuring that problems do not fall through the cracks, it seems that one of the most potent anti-bug tools we have - discussing the problem via a widely-distributed email list - is the same tool we have been using for decades.
Yet another advertising updateIn our continuing efforts to keep our readers informed, we wanted to update you on our recent advertising initiative. We are focusing our efforts this year (and hopefully beyond) on banner (or image) advertising. We won't neglect other opportunities, but we do want to more fully explore banner ads. To that end, we are currently running ads in a new location on the daily page, just to the right of the second entry. We also have plans to add more locations for banner ads of various sizes throughout the site. Unfortunately, the need to "keep the lights on" here requires us to generate more income than we currently do. To start with, as with any business, our income must be greater than our expenses. Even with a great deal of fiscal restraint, low salaries, and very low overhead, that is not, yet, happening. We would like to see the business grow beyond just a minimal, break-even operation – we think our readers agree – which will take some time and experimentation. We hope to strike the right balance between revenue generation and annoying our readers; we feel sure that you will let us know if we cross the line. We are always open to constructive suggestions (to lwn@lwn.net) about advertising and its placement on the site, but the most common suggestion, so far, is not particularly workable. A "no animated ads" policy becomes, essentially, a "no ads" policy. For better or worse, image ads are almost always animated. Readers do have the ability to change things at their end. Firefox provides a means (by setting the image.animation_mode in about:config to "none") to turn off animations – other browsers do as well. Firefox plugins (or add-ons) give even more control over the display of images and ads. In addition, subscribers at the project leader level have the ability to turn off all ads on the site. We have always tried to treat our readers with respect – as we would want to be treated – and will continue to do so. We do, however, need to find a way to make this enterprise sustain itself financially. We want to keep bringing you the excellent Linux and free software content that you have come to expect from LWN for many years to come.
Page editor: Jonathan Corbet |
|||||||||||||