LWN Weekly Edition
Front pageSecurity
Kernel development
Distributions
Development
Linux in the news
Announcements
->One big page
This page
Previous weekFollowing week
| |||||||||||
|
| |||||||||||
LWN.net Weekly Edition for July 3, 2008More DTrace envy Nearly a year ago, we looked at the status of SystemTap in the context of Sun's much-hyped DTrace tool. Since that time there has been progress, but the basic problem still remains: Linux does not have a good, ready-to-run answer to those wanting the equivalent functionality of DTrace. Due to an apparent disconnect between the developers of SystemTap and the kernel hackers, tracing for the Linux kernel—never mind user space programs—is not up to the competition. Both SystemTap and DTrace are tools meant to help administrators track down performance and other problems on production systems by instrumenting the kernel. Because SystemTap has not matured to the point of easy usability, DTrace is often seen as a prime differentiator between Linux and Solaris. In a posting to the ksummit-2008-discuss mailing list—where Kernel Summit topics are considered—Matthew Wilcox brought up the subject based on his experience at a recent PostgreSQL conference:
There was a lot of
buzz around DTrace. Sun and a couple of other companies have put DTrace
hooks into postgres, so they now have some really useful canned queries.
If you're running Solaris or MacOS, of course.
So there was a lot of talk about switching away from Linux. This can't possibly be a good thing for us. I don't personally know what the state of our competing projects are, but clearly they haven't got their hooks into postgres ... at least not upstream. Typically Linux has been in the forefront of interesting new technologies for free operating systems. When Sun opened up Solaris, though, a few features jumped ahead of their Linux counterparts, in particular the ZFS filesystem and DTrace. SystemTap is supposed to provide the tracing functionality while Btrfs is the leading candidate for a "next generation" filesystem. But, so far, SystemTap has not lived up to its potential. There are a few reasons for disappointment with SystemTap, some of which were pointed out by James Bottomley:
When I go around end users, I find people in two camps: The ones who've
drunk the sun coolaid and won't take anything on linux that isn't a
fully replicated dtrace (sort of like windows people who demand the
availability of outlook on linux) and people who are migrating to Linux
and trying to use systemtap for tracing. These latter seem to have a
number of genuine concerns including latency, the time it takes to
actually go from command executing to functional trace, the inability to
trace user programs (dtrace can) and concerns about the amount of
perturbation the probes actually place inside the kernel.
Those are all valid concerns, but the biggest problem for users is that, unless they are knowledgeable about kernel internals, it is difficult to know how to use SystemTap. A more simplified interface, one that is less reliant on kernel internals, needs to be created; the way to do that is through the placement of static trace points in the kernel and the creation of "tapsets" to make them easily usable. The SystemTap developers think the kernel hackers are in the best position to do that work. Ted Ts'o agrees but sees some barriers:
The big thing that are missing are the tapsets
— the macro libraries that allow a system administrator to use it to
find and solve performance problems without being a kernel developer,
and more importantly, the documentation for said macro libraries so a
system administrator can actually use it.
[ ... ] the real problem isn't as much kernel developers, it's that (a) it's too hard for many kernel developers to use (and so many kernel developers are [not] using it), and (b) there aren't enough tapsets. The latter is something that kernel developers can help solve, but unfortunately I'm not sure discussing it at the Kernel Summit will necessarily lead to making forward progress. If the kernel developers have trouble using SystemTap, they are unlikely to add the tapsets that would make it more usable for system administrators and others who have some general kernel knowledge but not enough to sensibly instrument it. For people using distribution kernels—at least for the enterprise distributions and Fedora—it is only somewhat painful to get SystemTap up and running. But kernel hackers tend to run their own kernels, often many different versions in a short period of time, so they need to be able to be easily build one that works with SystemTap and includes all of the debugging information that it requires. SystemTap developer Frank Ch. Eigler has a long reply to many of the complaints in the thread. It seems clear that the SystemTap folks and the kernel hackers have not been communicating—there are solutions to many of the problems that were cited. They are in various states of readiness, but are mostly working. So SystemTap is most of the way there for kernel tracing as long as you are well-versed in kernel internals, but that has been true for some time. In order to get SystemTap to where it needs to be, the kernel hackers need to be involved. Building the infrastructure and waiting for tapsets to magically appear is not a recipe for success. The SystemTap hackers need to be engaging the kernel community, as well as distributions, to make the tool into something that gets used. SystemTap can use static probe points, kernel markers—merged into 2.6.24—but it is notable that no one has, as yet, made use of them. A concerted effort needs to be made to make the tool more usable for the kernel developers who can, in turn, help make it more usable for others. There is a clear problem when folks like Ts'o regularly try, but find it too difficult to be useful:
But maybe as more people try using it, they'll discover some of these
rough edges, and will start trying to fix it. Every couple of months,
I've tried using it, and because it [h]as so many rough edges, I've
normally found it less work to debug the kernel using manual methods
rather trying to make Systemtap work on my system and with my kernel
development workflow.
It is a commonly heard complaint that while SystemTap is difficult to use, DTrace "just works" for Solaris; Eigler responds:
Yeah, so I hear, but think about how different their target
environment is. Their kernel hardly changes (several fixed APIs,
ABIs): this has huge implications. Their kernel was willing to
insert probes (~ markers), a bunch of build system changes (debug
info subset transcribing). Here in linux land, we suffer
multifaceted tensions and it is hard to go toward a goal without
obstructions (well-meaning as they may be).
A bunch of third-party scripts are often conflated with "dtrace", which is just a matter of growing the user community enough, and giving them a good tool to build on top of. A growing set of runnable end-user scripts is already packaged with systemtap, intended for use by nonexperts, more help (e.g. concise problem statements about what you'd like to measure/see) would be welcome. Many administrators and other users of tracing facilities are not necessarily interested in kernel-level tracing, but would really like to be able to use the instrumented versions of things like PostgreSQL. That is in the plan according to Eigler: "We aim to piggyback on these efforts by reusing the dtrace instrumentation calls embedded into postgres etc., if at all possible." Until the rough edges can be smoothed on the kernel side, Bottomley wonders if it even makes sense to start considering user space:
Although there are
differing opinions about what systemtap could and should do, it's clear
that it's not working incredibly well for its design space: the kernel,
so talking about extending it to userspace is a premature.
DTrace sounds like a nice working solution that has many uses and many happy users. If one can ignore the self-congratulatory postings from its lead developer, it might be worth having in Linux, but that simply is not going to happen. Paul Fox is working on a port of DTrace to Linux, but that ignores the licensing realities that would never allow it to become part of Linux. It also ignores the difficult path a DTrace port would face getting merged into the mainline. (We hope to have an article from Mr. Fox on his DTrace porting work soon, stay tuned). For all of the talk out of Sun about how they would love to make DTrace a part of Linux, they clearly made a choice to ensure that could not happen. Even if any technical barriers were lifted, the CDDL is not compatible with the GPL. It is perfectly fine as a free software license, but if you wish to get things into Linux, they must be licensed in a GPL-compatible way. This was well understood at the time Sun freed Solaris, so this must have been a conscious decision. Given how much their marketing organization likes to tout DTrace, it would seem to be a choice that Sun is quite happy with. Linux will eventually get the tracing support it needs, in a way that is easily accessible to users, but it may take some time. Conversations like the recent one on ksummit-2008-discuss are an important part of getting there. It would appear that better support for the use cases of kernel developers will be forthcoming. It is mostly a matter of documentation along with simplifying some of the building and installation issues. Once the kernel hackers actually start using it, progress is likely to be fairly swift. This is the way free software development works; it generally does not track a straight path to a solution, but often wanders about in the solution space for a while. It is highly unlikely that a development like DTrace could have come about in the way that it did in a true community-developed operating system. For that you need everyone pulling in the same exact direction, which may be why Sun is reluctant to turn over much of the governance of Solaris to the community. That may help them develop things more quickly, because there will be fewer barriers, but it won't help them to foster the kind of development community that characterizes Linux.
Mozilla plans for Firefox 3 and beyond The gift wrap is scarcely off Firefox 3 and the Mozilla community is already looking toward its next update. The first alpha release of Firefox 3.1, codenamed Shiretoko, may be released as early as this month, while its final release might see the light of day by year's end. Let's take a look at where this popular Internet browser is headed in the coming months, and what new features users can expect to see. Several features were nearly included for Firefox 3.0 but didn't make the cut because they weren't completely ready. New features expected to be in version 3.1 include a history and bookmark organizer with unified search and smart folder capabilities, and visual tab switching that shows thumbnail images of the web sites opened in each tab when moused over, both of which were abandoned in lieu of other, more critical features. According to an email sent to the mozilla.dev.planning mailing list, Mozilla's Vice President of Engineering, Mike Schroepfer, says there are other features expected to make it into version 3.1. For instance, native JSON DOM bindings (preferred by web developers over its JavaScript counterparts), an improved Awesomebar, support for cross-site XMLHttpRequest for the development of more powerful web applications, and better system integration are a few of the features Mozilla is anxious to get into the hands of users. Schroepfer says, "This, along with the overall quality of Gecko 1.9 as a basis for mobile and the desire to get new platform features out to web developers sooner has [led us] to want to do a second release of Firefox this year." In the event a feature isn't ready for version 3.1's targeted ship date, Schroepfer says rather than hold the release, it will simply be included in the next major release instead. In a recent blog post, Schroepfer says the new decision to aim for shorter, date-driven release cycles is in large part due to Mozilla's desire to "deliver releases of the quality and impact of Firefox 3 with much greater frequency." More frequent indeed; the gap between the release of Firefox 2.0 and 3.0 was almost two years. Not surprisingly, Firefox 4 is expected to usher in a whole host of changes, not the least of which is the introduction of Mozilla2, "an extensive update to the Mozilla platform to feature highlights like ActionMonkey, the merge of Mozilla's JavaScript engine (SpiderMonkey) and Tamarin, Adobe's JavaScript virtual machine open-sourced in late 2006." Details of the features expected to ship with Firefox 4 are sketchy, but the Vice President of Mozilla Labs, Chris Beard, has two projects currently under development that he'd like to see included: Weave and Prism. Weave is similar to the wildly popular browser synchronization add-on, Foxmarks. While Foxmarks only syncs an individual's bookmarks across machines, Weave's goal is to replicate a user's entire browsing experience — including bookmarks, favorites, passwords, and preferences — no matter where they access the Internet. Prism takes aim at Google Gears by making browser functionality available even while offline. Previously known as WebRunner, Prism is based on an idea called site specific browsers (SSB) and is already implemented in Fluid for Mac OS X, Adobe Air, and Microsoft Silverlight. Prism team member Matthew Gertner explains, "Rather than running programs in normal web browsers like Firefox or Safari, wedged in a tab between New York Times articles and TechCrunch posts, each app is given its own dedicated browser, which is customized to include many of the desktop features that users know and love." For a taste of what Prism can do within Firefox 3, download this extension. Of course, one of the biggest questions on the minds of many people these days is: what's up with the mobile version of Firefox? Although it looks like there's a ways to go before Mobile Firefox turns up on your Razr or BlackBerry, the rapid release cycle of Firefox will help push the project along. Schroepfer says, "There are already devices shipping with early versions of Gecko 1.9 at the core. More are coming soon and we'll be releasing milestones of full branded versions of Firefox (with XUL and the Firefox team taking a lead in the user experience) later this year. This lines up well with Firefox 3.1 and a synchronized release schedule will make everything run more smoothly." The development team is working on sorting through some of the basic differences among mobile devices such as a touch screen versus non-touch screen interface, virtual versus tactile keyboards, and so on. If you're interested in trying out the prototypes, they're available on the team's wiki page. Firefox 3 has been downloaded more than 8 million times since its release on June 17th, and more than 90% of users download the latest version of the browser within 7 days of its release. Clearly, Firefox has a large and growing user base, no doubt due in large part to Mozilla's willingness to offer new and useful features in a timely fashion.
Netgear's open router Your editor was recently reminiscing about an early stage of his career, which involved the administration of a VAX 11/780 computer. The VAX was a highly successful product, as was its native operating system VMS. Quite a few VAX customers chose to do without VMS, though, and put early versions of BSD Unix on them instead. Digital Equipment Corporation never entirely appreciated those customers. To DEC, every BSD installation looked like a lost VMS service contract.The company should, instead, have seen those installations as an extra sale gained as a result of the VAX's ability to run a nice operating system. Almost 30 years later, some parts of the computing industry have come to understand that there is value in selling hardware which can run operating systems provided by others. Microsoft made that point in a big way, of course, but there are also significant parts of the industry which benefit from making systems which can run Linux - and, in particular, a version of Linux which is not necessarily supplied by the vendor. But other sectors still seem to see the ability for the customer to put (or replace) Linux on their systems the way DEC saw Unix in the early 1980's. They see no value in letting their customers make changes to their systems, choosing instead to lock those systems down and keep total control. Embedded systems are often singled out as an example of this type of behavior, and vendors of small routers tend to be especially inclined in this way. It is not a coincidence that a substantial portion of the high-profile GPL-enforcement cases to date have involved consumer-level routers. Some vendors, at least, are getting smarter and doing what they need to do to avoid licensing problems. But relatively few of them welcome customers who want to replace the software on "their" devices. There are exceptions, though, and their number just grew with this announcement from Netgear. The WGR614L router looks like a fairly straightforward consumer wireless router, with the usual set of features. LWN readers will doubtless be glad to hear that it is "Works with Windows Vista" certified. It has a four-port Ethernet switch, an 802.11g access point, and a mighty 240 MHz CPU and 16MB of RAM. All of the stuff one would expect from an inexpensive desktop device. But what makes this device interesting is that it's designed to be open and hackable. The source code for the factory-installed firmware is available from Netgear's community web site; it's amusingly packaged as a zip file containing a single, compressed tarball which, in turn, holds a bleeding-edge 2.4.20 kernel tree. But anybody wanting something a bit more contemporary and community-oriented can replace that firmware altogether with a package like Tomato or DD-WRT; indeed, Netgear almost seems to encourage its customers to do so. Every one of those customers then gets the benefit of the effort which has gone into the development of those router distributions - with little effort required on Netgear's part. Those customers can improve this platform and make their changes available to other customers; that makes Netgear's hardware more valuable. If there are bugs in the system, a single motivated customer can fix them and make those fixes available to everybody else. And all of this comes at almost no cost to Netgear. It is always fun to see Linux turn up in new places. It's now a routine experience to realize that one's new television, camcorder, music player, or automobile runs Linux. But locked-down, Linux-based devices are not far removed from the fully proprietary systems which preceded them. Whether or not one agrees that locking down systems in this way is legally or morally defensible, it's easy to conclude that it is undesirable. A Linux system which is cast in concrete loses a part of the vital energy which makes Linux what it is. So it is always a welcome development when a vendor decides to take a more open path. With any luck at all, the wider public will eventually realize that more open devices are more powerful devices, and, as a result, such devices will prove more successful. That is the path that brings us more control over our systems and, eventually, to World Domination.
Page editor: Jonathan Corbet Inside this week's LWN.net Weekly Edition
|
|||||||||||