LWN Weekly Edition
Leading itemsSecurity
Kernel development
Distributions
Development
Announcements
->Multipage format
This page
Previous weekFollowing week
Recent Features
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for June 3, 2011A conversation with Linus at LinuxCon JapanLinus Torvalds rather famously does not like speaking in public, so his conference appearances tend to be relatively rare and unscripted. At LinuxCon Japan, Linus took the stage for a question-and-answer session led by Greg Kroah-Hartman. The result was a wide-ranging discussion covering many issues of interest to the kernel development and user communities.
3.0The opening topic was somewhat predictable: what was the reasoning behind the switch to 3.0 for the next kernel release? The problem, said Linus, was too many numbers which were getting too big. A mainline kernel release would have a 2.6.x number, which would become 2.6.x.y once Greg put out a stable release. Once distributors add their build number, the result is an awkward five-number string. Even so, we have been using that numbering scheme for about eight years; at this point, the "2.6" part of a kernel version number is pretty meaningless. Once upon a time, Linus said, making a change in the major number would be an acknowledgment of some sort of major milestone. The 1.0 kernel was the first to have networking, 1.2 added support for non-x86 architectures, 2.0 added "kind of working" SMP support, and so on. We used to think that incrementing the major version number required this kind of major new feature, but, in the 2.6.x time frame, we have stopped doing feature-based releases. The current development process works wonderfully, but it has caused the 2.6.x numbering scheme to stick around indefinitely. As we approach the 20th anniversary of the Linux kernel, we had a good opportunity to say "enough," so that is what Linus did. "3.x" will not stay around forever - or even until the kernel is 30 years old; Linus said he expects to move on around 3.20 or so.
Linus noted that some people were thinking that 3.0 meant it was time to
start with big new features (or removal of old code), but that's not what
is meant to happen. It is just a number change, no more. Trying to have
the kernel be stable all the time has, he said, worked very well; that is
not going to change.
Greg asked: what recent changes did he like the most? Linus responded that he tends to like the boring features, things that people don't notice. Performance improvements, for example; he called out the dcache scalability work as one example. There is no new interface for users, it just makes the same old stuff go faster.
Features and bloatIs the kernel, as Linus famously said on a 2009 panel, bloated? Linus acknowledged that it is still pretty big; it couldn't possibly run on the machine that he was using to develop it 20 years ago. But even phones are far more powerful than that old machine now, so nobody really cares. The kernel has been growing, but that growth is generally necessary to meet the needs of current hardware and users. What about the addition of features just for the heck of it - new stuff that is not strictly driven by new hardware? Is that something that we can still do? Linus said that there are certainly developers working on features with no current users, thinking five years or so into the future. Sometimes that work succeeds, sometimes we end up with code that we regret adding. Linus said that he is increasingly insisting on evidence that real users of a feature exist before he is willing to merge that feature. Greg asked about control groups, noting that a lot of kernel developers really object to them. Linus responded that control groups were a feature that did not initially have a whole lot of users, but they do now. Control groups were initially added for certain specific server setups; few others had any interest in them. Developers were unhappy because control groups complicate the core infrastructure. But control groups have begun to find a lot of users outside of the original target audience; they are, in the end, a successful feature. Symmetric multiprocessing (SMP) was also, at the beginning, a feature with few users; it was a "big iron" feature. Now we see SMP support used across the board, even in phones. That illustrates, Linus said, one of the core strengths of Linux: we use the same kernel across a wide range of platforms. Nobody else, he said, does things that way; they tend to have different small-system and large-system kernels - iOS and Mac OS, for example. Linux has never done that; as a result, for example, there has never been a distinct cut-down kernel for embedded systems. Since the full kernel is available even at that level, Linux has been a real success in the embedded world.
Embedded systems, world domination, and the next 20 yearsContinuing with the topic of embedded systems, Greg asked about the current furor over the state of the ARM tree. Linus responded that developers in that area have been a bit insular, solving their own problems and nothing more. That has resulted in a bit of a mess, but he is happy with how things are working out now. As a result of pushback from Linus and others, the ARM community is beginning to react; the 3.0 kernel, Linus thinks, will be the first in history where the ARM subtree actually shrinks. The embedded world has a history of just thinking about whatever small platform it's working on at the moment and not thinking about the larger ecosystem, but that is changing; this community is growing up. Many years ago, Greg said, Linus had talked about the goal of "total world domination" and how more applications were the key to getting there. Is that still true? Linus responded that it's less true than it used to be. We now have the applications to a large degree. He also no longer jokes about world domination; it was only funny when it was obviously meant in jest. At this point, Linus said, we are doing really well everywhere except on the traditional desktop. That is kind of ironic, since the desktop is what Linus started the whole thing for in the first place - he wanted to run it on his own desktop system. We have most of what we should need at this point, including a lot of applications, but the desktop is simply a hard market to get into. It's hard to get people to change their habits. That said, we'll get there someday. Can we do anything in the kernel to further that goal? Linus responded that he'd been thinking about that question, but he didn't really know. A lot of work has been done to get the kernel to support the desktop use case; kernel developers, after all, tend to use Linux as their desktop, so they are well aware of how well it works. But it's up to the distributors to target that market and create a complete product. Greg noted that 20 years is a long time to be working on one project; has Linus ever thought about moving on? Linus responded that he really likes concentrating on one thing; he is not a multi-tasker. He's really happy to have one thing that he is doing well; that said, he never had expected to be doing it for this long. When asked if he would continue for another 20 years, Linus said that he'd be fairly old by then. Someday somebody young and energetic will show up and prove that he's really good at this work. That will be Linus's cue to step aside: when somebody better comes along. What do we need to do to keep the kernel relevant? Linus said that relevance is just not a problem; the Unix architecture is now 40 years old, and it is just as relevant today as ever. Another 20 years will not make a big difference. But we will continue to evolve; he never wants to see the kernel go into a maintenance mode where we no longer make significant changes.
Moments, challenges, and licenses
A member of the audience asked Linus to describe his single most memorable
moment from the last 20 years. Linus responded that he didn't really have
one; the kernel is the result of lots of small ideas contributed by lots of
people over a long time. There has been no big "ah ha!" moment. He went
on to describe a pet peeve of his with regard to the technology industry:
He did acknowledge that there have been some interesting moments, though, going back to nearly 20 years ago when Linux went from a personal project to something where he no longer knew all of the people who were involved in it. At that point, he realized, Linux wasn't just his toy anymore. There have been exciting developments; the day Oracle announced that it would support Linux was one of those. But what it really comes down to is persistence and hard work by thousands of people. Another person asked whether the increasing success of web applications would mean the end of Linux. Linus responded that the move toward the browser has, instead, been helpful to Linux. There used to be a whole lot of specialized, Windows-only applications for tasks like dealing with banks; those are now all gone. When applications run in the browser, the details of the underlying operating system don't matter, at which point it comes down to technology, licensing, and price - all of which are areas in which Linux excels. The next question was: are you happy with Ubuntu? Linus suggested that Greg should answer that question for a more amusing result. He went on to say that Ubuntu is taking a different approach and is getting some interesting results. It is helpful to have a distributor working with a less technical, more user-centric approach. Ubuntu has been successful with that approach, showing the other distributors a part of the market that they were missing. Greg added that his main concern is that he wants to see the kernel community grow. Things are, Greg said, getting better. What is the toughest technical problem that Linus has ever had to deal with? Linus answered that the biggest problems he faces are not technical. In the end, we can solve technical issues; we make bad decisions sometimes, but, over time, those can be fixed. When we have serious problems, they are usually in the area of documentation and help from hardware manufacturers. Some manufacturers not only refuse to help us support their hardware; they actively try to make support hard. That, Linus said, irritates him, but that problem is slowly going away. What is really hard, though, is the problem of mixing the agendas of thousands of developers and hundreds of companies. That leads to occasional big disagreements over features and which code to merge. If Linus loses sleep, it tends to be over people and politics, not technical issues; the interactions between people can sometimes frustrate him. We usually solve these problems too, but the solution can involve bad blood for months at a time. The linux-kernel mailing list, Linus said, is somewhat famous for its outspoken nature; it is seen as a barrier to participation sometimes. But it's important to be able to clear the air; people have to be able to be honest and let others know what they are thinking. If you try to be subtle on the net, people don't get it; that can lead to developers putting years of work into features that others simply hate. In the long run, Linus said, it can be much healthier to say "hell no" at the outset and be sure that people understand. Of course, that only works if we can then admit it when it turns out that we were wrong. The final question was about the GPL: is he still happy with the license? Linus said that, indeed, he is still very happy with GPLv2. He had started with a license of his own creation which prohibited commercial use; it took very little time at all before it became clear that it was making life hard for distributors and others. So he has always been happy with the switch to the GPL which, he said, is a fair and successful license. He feels no need to extend it (or move to GPLv3); the license, he said, has clearly worked well. Why change it? [Your editor would like to thank the Linux Foundation for assisting with his travel to Japan.]
MeeGo 1.2 on the N900As discussed last week, the ongoing absence of a mass-market handheld device running MeeGo remains a thorn in the project's side. While the set-top box and in-vehicle platforms have signed on major OEM adopters, the general consumer market remains focused on smartphones. Several rumors were circulating around MeeGo Conference San Francisco last month, hinting that Nokia had intended to launch its long-planned MeeGo-based phone at the event, but had been forced to push back the release date again (the stories conflict as to why). Nevertheless, a community effort exists to develop a user-ready MeeGo distribution for the N900 handset, and that team did make a release at the conference. The project is known as the MeeGo Developer Edition (DE) for N900, and its latest release is named MeeGoConf/SF in honor of its unveiling at the conference. The SF release is based on MeeGo 1.2, but includes a lengthy set of customizations that have been shared back upstream with the MeeGo Handset user experience (UX) project (although not all of the changes are destined to be merged into the official MeeGo releases). The MeeGo DE software is currently in a working and remarkably stable state, though the developers caution that it still should not be used as a daily replacement for the N900's default Maemo OS — there are the to-be-expected bugs, plus the loss of warranty protection. Still, the team has set very user-oriented goals for the project, and it is a pleasant surprise to see just how well some of them work. Inside the Developer Edition![[Jukka Eklund]](/images/2011/meego-eklund-sm.jpg)
Jukka Eklund led a panel session presentation about the MeeGo DE release on the final day of the conference. In it, he outlined four use-cases that the team established as its goals: working cellular calls, working SMS messages, a working camera, and a browser working over WiFi. The camera goal was added to the list along the way, he noted, because the camera code was working so well. In addition to that core functionality, the team decided to include certain add-ons on top of the base MeeGo Handset distribution, including additional applications (the Firefox For Mobile browser, Peregrine instant messenger, the QmlReddit news reader, etc.), improvements to core MeeGo Handset applications (such as a heavily patched and ported-to-QML dialer, and improved volume-control), and some entirely new functionality. The new functionality includes USB modes not yet supported upstream, plus the ability to lock and unlock a SIM card with a PIN code — including a user interface to do so. Although the MeeGo DE project works entirely in the open, within the existing MeeGo project infrastructure (IRC, mailing lists, and bug tracker), Eklund described its mindset as working like a "virtual vendor," following the same product management life-cycle as a commercial hardware/software supplier would — but sharing the experience in the open, and providing its changes back to the community. That is a novel and interesting approach: the Linux Foundation makes much out of MeeGo's ability to shorten product-development times and costs, but typically with real-world examples where a product emerges only at the end. The WeTab team was on hand during the MeeGo Conference keynote to say that their MeeGo-based tablet went from design to production in a scant four months, for example. There is certainly no reason to doubt those numbers, but actually seeing the process documented is even better. On top of that, Nokia has long defended its right to keep its own UX layer proprietary because it is the only way to "differentiate" its MeeGo products in the market. That mindset, it seems, has led partly to the project's current UX strategy, where MeeGo releases drop with a set of "reference quality" UXes that vary considerably in usability and polish. It is to the MeeGo DE team's credit that they have improved on the reference UX and sent their patches upstream. Some (such as the dialer) have already been merged; others (such as the theme and UI layer) have a less-than-clear fate awaiting them. Eklund covered the current status of the SF release, gave some time for fellow panelists Makoto Sugano, Harri Hakulinen, Marko Saukko, and Carsten Munk to make comments, then proceeded to demonstrate the new code running on an N900. The demonstration included the revamped dialer, which he used to call Tom Swindell — the developer who ported the dialer to QML — there in the audience. He also showed off the basic application set, and demonstrated the N900's ability to run the MeeGo Tablet UX (although the tablet interface is not quite usable, as it assumes larger screen real estate). Testing![[N900]](/images/2011/meego-n900-sm.jpg)
As nice as the demo was, I decided that I must try out the code on my own N900 in order to get the full experience. Installation instructions are on the MeeGo wiki, which proved to be the only challenging part of the process — not insurmountable, but probably due for a cleanup. The wiki spreads out download, flashing, memory card preparation, and installation instructions over a large number of pages, some of which are not properly updated to reflect the current state of the release and show signs of multiple editors crossing paths. For example, the "Install image" table lists two separate processes as the "recommended way" to install MeeGo DE, and the most-recommended-process is listed as "Dual-boot install," which is a separate entry from "MMC Installation" even though it in fact requires installing MeeGo DE to the phone's MMC card. In addition, there are a few places where the instructions say that a 2GB memory card is sufficient, but this is no longer true as of the SF release, which takes at least 4GB of storage. The simplest option is to install the uBoot bootloader on the phone (which necessitates activating the unstable Maemo Extras-devel repository first, at least long enough to add uBoot), then to write the raw MeeGo DE image to an un-mounted microSD card. On the SF release download page, this is the large .raw.bz2 file. It is theoretically possible to download the raw image from the N900 itself, then pipe it through bzcat into dd to write it directly to the memory card, but I wimped out and did the card-preparation step from a desktop Linux machine instead. With the newly-written memory card inserted into the phone, all that remains it to power it on. The uBoot loader will recognize the bootable volume on the card, and load it by default, booting the phone into MeeGo DE. To boot the phone into Maemo again, you simply reboot with the memory card removed, or type Browsing around, I have to say I was quite impressed with MeeGo DE. There are a few areas where DE is a noticeable improvement over vanilla MeeGo Handset UX: the UI widgets are more consistent in appearance, the text is higher-contrast and more readable, and the UI navigation cues (such as the "return to home" button and the application grid's page indicator) easier to pick up on. Loading applications is slow, although this is no doubt largely dependent on the speed of the memory card I used. Once launched, the interface runs smoothly, including the quick QML transitions. As for the application set itself, the dialer is (as all reports had indicated) a vast improvement over the stock Maemo dialer. It is easier to find contacts and enter numbers, and the on-screen buttons respond faster to finger-presses. There are similar usability improvements in the network and general configuration settings — Maemo's interface for connecting to a new WiFi access point or Bluetooth device borders on arduous. The camera seems faster to focus, and is much faster to adjust to lighting changes and to display the most recent shot. On the other hand, there are pain points. I could not get the video player to work at all (the SF release ships with both Big Buck Bunny and a stalwart, YouTube-caliber kitty video in 240p); it alternated between displaying only a black screen and playing wildly stretched-out-of-proportion. The audio player has a nicer navigation structure than Maemo's (which relies on large, vertically-scrolling lists oddly displayed only in landscape format), but the playback buttons are tiny — about one-fourth the size of the application launcher buttons. For some reason, MeeGo DE will sense the device orientation and rotate the display into three of the four possible directions, but not the fourth, upside-down-portrait. I don't have any need to use my phone in upside-down-portrait orientation, but surely it is more useful than upside-down-landscape, which is supported, even though it leaves you with an upside-down keyboard. Finally, I could not get the hang of the top status panel; I kept wanting (or expecting) it to open up a quick-settings menu, like it does in Maemo, for common tasks like enabling or disabling Bluetooth or the data connection. The third-party applications are a mixed bag, as you might expect. I am a big fan of Firefox For Mobile, and Peregrine seems nice, but I was less impressed with the Kasvopus Facebook app. Some of the apps use their own widget styles, which can be awkward. That is certainly not MeeGo DE's fault, but it raises some questions about QML. One of Maemo's strong suits is that all of the applications use the same toolkit, so feature buttons, sliders, and notifications are easy to pick out. The more applications that write their own crazy QML interfaces, the more potential for user confusion there is. On the whole, however, the SF release packs a good selection of applications. If you have purchased a phone recently, you no doubt remember that the first step is always removing the glut of sponsored apps and vendor-provided tools that only work with a single service. MeeGo DE doesn't go overboard in its service offerings, it just provides you with a flexible set of utilities: an IM client, not a GoogleTalk-only or Skype-only app; a real browser, not a Yahoo-only search tool. All that said, MeeGo DE is not yet ready to serve as a full replacement for Maemo. The big blockers at the moment are mobile data and power management, but it will need improvements in SMS and MMS support, call notification, and a few other key usability areas. Eklund and the other team members are straightforward about the intended audience of the DE builds: they are a showcase for MeeGo on handsets, they demonstrate that the code can run comfortably well even on hardware several years old, and they allow the community to push the envelope where commercial phone-makers are not. On all of those fronts, one must consider the MeeGo DE SF release a success. I experienced no crashes, and I both made and received phone calls without incident. Better yet, the UX layer (including both the interface and the application suite) is an improvement over the reference design. If the MeeGo steering group wants to keep the reference UXes in an unpolished, demo-only state, they still can. I personally feel like that is a mistake: no OEM is prevented from replacing them in order to differentiate its products, but the gadget-hungry public always sees a demo-quality design first. But MeeGo DE for the N900 shows that the project doesn't have to continue down that road, because the community is ready and willing to pitch in. [ The author would like to thank the Linux Foundation for sponsoring his travel to the MeeGo Conference. ]
PGCon 2011, the PostgreSQL developer conferencePGCon is the international PostgreSQL contributor conference, held in Ottawa, Canada for the last four years. It serves the same purpose for the PostgreSQL community that the Linux Plumbers Conference does for Linux, attracting most of the top code contributors to PostgreSQL and its extensions and tools. Nearly 200 contributors from around the world attended this year, including hackers from as far away as Chile and the Ukraine, as well as a large contingent from Japan. PostGIS knows where you are"Everyone is being followed, all the time," said Paul Ramsey, beginning his talk on PostGIS. Ramsey is the creator of PostGIS, the geographic data extension to PostgreSQL, and works for OpenGeo as an architect. He talked about how changes to the way we live are causing an explosion in the adoption of geographic and spatial technologies. "PostGIS knows where you are. Should you be worried? No. Should you be worried about Apple? Probably," he said. He also went briefly into the history of the Geographic Information System (GIS), which was invented in Canada, and then talked about the PostGIS project development. PostGIS has been advancing rapidly, and according to Ramsey is now advancing GIS itself. "We ran out of things to implement in the standard, so we moved to a new standard that had more things to implement." Plans for PostGIS 2.0 include raster support, 3D and 4D indexing, and volumetric objects. The first requests for raster support and volumetric objects have come from archaeologists and water system planners. These people need to represent three-dimensional objects underground; not only their shape but their relative position in three dimensions. Another area where volumetric shapes are essential is airspace; one attendee demonstrated an SVG and PostGIS application which showed Ottawa airport traffic in real time. "4D indexing" means indexing objects not just in space but in time as well so that people and objects can be tracked as they move. The proliferation of public video cameras and geographic data from our cell phones is supplying a flood of data about motion as well as location. Currently no software exists which can represent this data and search it on a large scale. The PostGIS project plans to develop it. The primary limitations PostGIS is struggling with today are write-scalability and lack of parallel processing for GIS data. The latter is an issue because rendering, transforming, and testing spatial objects — such as for answering the question "which building parcels border this one?" — are CPU-intensive. Currently you cannot use more than one core for a single GIS query. Recently Foursquare had to turn away from PostGIS because it couldn't keep up with the hundreds of thousands of updates per second that the service gets. Both the transactional overhead and the time required to write serialized geometric objects were too much. They switched to custom software on top of Lucene and MongoDB. Ramsey hopes that PostgresXC will supply clustering that PostGIS can use. Conference SessionsSince PGCon consists of 34 sessions in three parallel tracks, I couldn't attend everything. Some of the most interesting talks included:
There were many other good sessions which I had no time to attend. However, I did make it to: Hacking the query planner
"My name is Tom Lane, I've been hacking the query planner, and I'm looking for some help."
Possibly the most popular sessions at PGCon was Tom Lane's review and explanation of how the PostgreSQL query planner works. Tom Lane is the lead hacker on the PostgreSQL project, and his session, which ran overtime into a second session, was packed. The query planner is part of answering database requests; it decides how to retrieve data most efficiently. Since even a moderately complex query can involve making decisions between millions of possible execution paths, the query planner code itself is a bit ... opaque. Some of PostgreSQL's more than 25,000 lines of planner code is 20 years old. Many parts of the planner are best described as "kludgy," but Tom Lane and other lead developers are reluctant to modify them because they currently work fairly well. More importantly, the query planner is a holistic system with parts which interact in ways which are hard to predict. This has led to a lack of developers interested in making improvements due to the difficulty of the task. Lane gave his talk in hopes of changing that. The planner is the most complex part of the four stages which go into answering a PostgreSQL query. First, the Parser decides the semantic meaning of the query string, separating out object identifiers, operators, values, and syntax. Second, the Rewriter expands views, rules, and aliases. Thirdly, the Planner chooses indexes, join plans, "flattens" subselects, and applies expressions to different tables in order to find the best query plan to execute. Lastly, the Executor executes the plan which the planner prepares. The three goals of the planner are: to find a good (fast) query plan, to not spend too much time finding it, and to support PostgreSQL's extensibility by treating most operators, types, and functions as black box objects. The planner does this by forming a tree of "plan nodes," each of which represents a specific type of processing to be performed. Each node takes a stream of tuples (i.e. rows) in and outputs a stream of tuples. Plan nodes consist of three types: Relation Scans, which read data from tables and indexes, Join nodes, which join two other nodes, and Special Plan nodes, which handle things like aggregates, functions scans, and UNIONs of two relations. Each node has a data source (the input stream), output columns and calculations, and qualifiers or filters. Nodes also have a set of statistics in order to estimate how much time it will take to do all of these things. Plan nodes are mostly autonomous from each other, but not always; for example, the planner takes output sort order into account when considering a merge join. The first phase the planner goes through is an attempt to simplify the query. This includes simplifying expressions, "inlining" simple SQL functions, and "flattening" subselects and similar query structures into joins. This simplification both gives the planner the maximum flexibility in how to plan the query, and reduces repetitive re-processing for each tuple. The planner then goes through Relation Scan and Join planning, which roughly correspond to the FROM and WHERE portions of a SQL query. Of this planning stage, the most expensive part is Join planning, since there is a factorial expansion of the number of possible plans based on the number of relations you are joining (i.e. n tables may be joined in n! ways). For a simple to moderately complex query, the planner will attempt to plan all possible execution plans, or Paths, it could take to perform the query. For a very complex query — for example, one with a 25-table Join — the planner needs to resort to approximate planning methods, such as the "GEQO" genetic algorithm for creating query plans built into PostgreSQL. GEQO does not tend to produce very good query plans, however, and really needs improvement or replacement with another algorithm. There is also a set of simplification rules for complex Join trees, but since these are not estimate-based, they sometimes cause bad plans. All of the preceding is needed to determine the possible Paths for executing the query, so that the planner can calculate the cost of each path and find the "cheapest" plan in terms of estimated execution time. This estimate is created by combining a set of statistics about tables and indexes, which is stored in the pg_statistic system table, with a set of estimation functions keyed to operators and data types. For each join or scan with a filter condition (such as a WHERE clause), the planner needs to estimate the selectivity of the conditions so that it can estimate the number of tuples returned and the cost of processing them. Using both a histogram and stored selectivity estimates, the planner can usually calculate this fairly accurately, but often fails with complex expressions on interdependent relations or columns, such as AND/OR conditions on columns which are substantially correlated. The other large omission from this design which causes planning issues is the lack of estimate calculations for functions. Lane also went over the need for major future work on the planner. Two specific issues he mentioned were the need for parameterized scans for subselects and functions, and the need to support estimates on Foreign Data Tables in PostgreSQL 9.1. He also went over in detail some of the files and code structures which make of the query planner. His slides [PDF] are available on the PGCon website. Developer SummitsThe biggest change to the conference this year was the multiple summits and meetings held alongside and after the conference. Tuesday was the Clustering Hacker's Summit, Wednesday was the Developers' Meeting, and Saturday was the PL/Summit, a new event. The goal of these all-day meetings was better coordination of the hundreds of contributors in more than 20 countries who work on PostgreSQL The Clustering Summit was the second such event, sponsored by NTT Open Source, who also sponsored the previous summit in Tokyo in 2009. PostgreSQL supports multiple external replication and clustering projects, many of them having only one or two developers. The thirty attendees included developers on the Slony-I, Bucardo, Londiste, pgPool-II, and GridSQL projects. Several NTT and EnterpriseDB staff there were representing a newer project, called PostgresXC, intended for write-scalable clustering. For the last three years, twenty-five to thirty of the most prominent and prolific PostgreSQL code contributors have met in order to discuss development plans and projects. This is what has allowed the project to work on several features, such as binary replication, which have taken several years and several releases to complete. While there was a great deal of discussion at the summit, few decisions were made. The Developers' Meeting did set a schedule for 9.2 development, which will be very similar to last year's schedule, with CommitFests on June 15th, September 15th, November 15th, and January 15th, and a release sometime in the summer of 2012. This year's PGCon also included the first summit of contributors to the various Procedural Language (PL) projects, and included developers of PL/Perl, PL/Python, PL/Lua, PL/PHP, PL/Tcl, and pgOpenCL. ConclusionThere were some other announcements at PGCon. Marc Fournier, one of the founding Core Team members for PostgreSQL, retired from the Core Team. This follows several changes on the Core Team in the last year, including the retirement of founding member Jan Wieck and the addition of major contributor Magnus Hagander. Postgres Open, a new business-oriented PostgreSQL conference, was announced by its organizer. Lest you think that PGCon was entirely brainiac database hacking, the most popular track was probably the unofficial Pub Track held at the Royal Oak Pub near campus, where Aaron Thul introduced The PostgreSQL Drinking Game [PDF] at the conference. Next year you could go to PGCon, and learn both how to hack databases and how to drink to them.
Security Phones and permissionsThe Android permission system for applications ("apps" these days) is an all-or-nothing affair; one can either grant all the permissions that the app asks for, or deny them and not install it. While it is useful to know what permissions are being granted, it would be even more useful for security and privacy conscious users to be able to selectively deny certain permissions—especially those that have no clear connection to the proper functioning of the app. The CyanogenMod (CM) alternate Android firmware has had this ability in its tree since mid-May, but a newer patch that builds on that functionality has been met with resistance from a somewhat surprising direction. The original patch from Plamen K. Kosseff to add permission revoking was accepted after working out some difficulties with apps that crashed when they didn't have the permissions they expected. In addition, enabling permission revocation involves a setting in the "Performance Settings" menu of CM configuration, which means that the project is free to ignore any bug reports generated from the feature. Kosseff then went on to post a patch for review building on that earlier work, and allowing users to allow certain permissions in a "spoofed" mode. The specific example he used was to spoof the phone's International Mobile Equipment Identity (IMEI) number, rather than allow an app access to the real IMEI—that didn't sit well with some CM developers, including CM founder Steve Kondik. The choice to start by spoofing the IMEI was perhaps unfortunate, as Kosseff has ideas for other, probably less-controversial privacy features. For example, in the comments on the patch he describes other possible uses, including restricting what information in the contacts list gets handed to apps, or only showing a portion of the SD card contents to apps. Either of those are obvious improvements to privacy, and ones that shouldn't cause any problems for app developers. The main objection to returning a bogus IMEI value (or the related idea of returning a bogus SIM ID and phone number) is that app developers use that information for data-gathering purposes. While that data gathering might be used for malicious purposes, the clear sense from the comments on the patch is that most app developers are using it for demographic and usage information that is, at least relatively, benign. Kondik and others are concerned that creating a "hostile environment" for app developers will lead to problems for CM, either from the app developers themselves or from larger organizations like Google, handset makers, and cellular service providers. But, as Kosseff asks, shouldn't the user be able to make the decision about what information they share with apps? For IMEI and related information, the answer from the CM developers seems to be "no". It seems somewhat counter-intuitive that a phone distribution with the goal of unlocking the full potential of the hardware would draw that particular line. Others, perhaps the Guardian project for example, are likely to take a different stance. Part of the issue is that it is unclear what is "owed" to the app developers for use of their code. For paid apps, the line is a little less blurry, as one can expect that those developers aren't owed any more than was paid. For gratis apps, things get a little more hazy. If one grants permission to see the contact list to latest bouncing cow game, is it reasonable to revoke that permission, or to provide an empty list? In addition, many gratis apps use the network permission to grab advertisements to show within the app. That is part of the revenue model the developer is using to fund app development, so is it fair to turn that off? On the flip side, should the app refuse to run if it can't call home for ads? There aren't necessarily any easy answers to some of those questions. Avoiding apps that request more permissions than they really need is certainly one way around the problem, but the permissions aren't really fine-grained enough to prevent abuse. If one grants an ebook reading app permission to use the SD card (presumably to store any books that are being read), does that mean it should be able to go poke around and see what other ebook apps are being used? It will also presumably need network permissions to grab content from various places, can it also use them to phone home with a copy of one's reading habits? This is yet another area where free (as in freedom) software can help. There are certainly plenty of users who will be happy to play an ad-supported bouncing cows game, without disabling the network out from under it, if they are sure that the game isn't using its permissions for ill. Likewise, there are plenty of legitimate reasons that an app might need to access the contact list, so long as one can be sure that it isn't sending the contents to spammers (of the voice, SMS, IM, or email kind). For most consumers, any of these safeguards are essentially pointless. As we have seen in the consumer PC world, users will install almost anything, from anywhere, even overriding security warnings from the OS, if it will get them the latest game, mouse cursor, or video content. There's not much hope of changing that, but for the rest of us, who might just care about the data we store on our phones, having more control over the permissions we grant to apps will go a long way toward solving these kinds of problems. A rich ecosystem of free software apps would go even further.
Brief items Security quotes of the week
'apply jipsam algorithm'. This is a crypto module that isn't in mainline
(and apparently doesn't exist outside North Korea). I bet it's good
though. No backdoor master keys or anything similar.
-- Dave
Jones roots through the Red
Star Linux kernel changelog
I'm talking about instances where the government is relying on secret
interpretations of what the law says without telling the public what those
interpretations are, and the reliance on secret interpretations of the law
is growing.
-- US Senator Ron
Wyden in Wired on the "secret" Patriot Act
New vulnerabilities bind9: denial of service
chromium-browser: multiple vulnerabilities
citadel: denial of service
dovecot: denial of service, possible mailbox corruption
ejabberd: denial of service
eucalyptus, rampart: code execution
gdm: uncontrolled access to local filesystem
gimp: arbitrary code execution
gimp: arbitrary code execution
jabberd14: denial of service
kernel: multiple vulnerabilities
kernel: multiple vulnerabilities
libmodplug: stack overflow
mahara: multiple vulnerabilities
mumble: denial of service
pam: denial of service
perl-libwww-perl: man-in-the-middle attack
php-zendframework: SQL injection
rssh: privilege escalation
subversion: multiple vulnerabilities
systemtap: denial of service
unbound: design flaw
unbound: denial of service
wireshark: multiple vulnerabilities
Page editor: Jake Edge Kernel development Brief items Kernel release statusThe current development kernel is 3.0-rc1, released on May 29. Linus said:
So what are the big changes? NOTHING. Absolutely nothing. Sure, we
have the usual two thirds driver changes, and a lot of random
fixes, but the point is that 3.0 is *just* about renumbering, we
are very much *not* doing a KDE-4 or a Gnome-3 here. No breakage,
no special scary new features, nothing at all like that. We've been
doing time-based releases for many years now, this is in no way
about features. If you want an excuse for the renumbering, you
really should look at the time-based one ('20 years')
instead.
See the separate article below for a summary of the changes merged in the second half of the 3.0 merge window. Stable updates: The 2.6.38.8 and 2.6.39.1 stable updates were released on June 2. This will be the last 2.6.38 stable release, so users should move to the 2.6.39 series.
Quotes of the week
Users are a really terrible source of interface
specifications. "Hackers" are often not much better, but at least
if the interface is lousy the developer has the potential to be
accountable for it and its improvement.
-- Casey Schaufler
IMHO the key design mistake of LSM is that it detaches security
policy from applications: you need to be admin to load policies,
you need to be root to use/configure an LSM. Dammit, you need to be
root to add labels to files!
This not only makes the LSM policies distro specific (and
needlessly forked and detached from real security), but also gives
the message that:
-- Ingo Molnar
'to ensure your security you need to be privileged'which is the anti-concept of good security IMO.
Garrett: RebootingMatthew Garrett appears to be having some "fun" looking into how to reboot x86 hardware. He lists five different mechanisms to reboot 64-bit x86 hardware including: "kbd - reboot via the keyboard controller. The original IBM PC had the CPU reset line tied to the keyboard controller. Writing the appropriate magic value pulses the line and the machine resets. This is all very straightforward, except for the fact that modern machines don't have keyboard controllers (they're actually part of the embedded controller) and even more modern machines don't even pretend to have a keyboard controller. Now, embedded controllers run software. And, as we all know, software is dreadful. But, worse, the software on the embedded controller has been written by BIOS authors. So clearly any pretence that this ever works is some kind of elaborate fiction. Some machines are very picky about hardware being in the exact state that Windows would program. Some machines work 9 times out of 10 and then lock up due to some odd timing issue. And others simply don't work at all. Hurrah!"
The Wonderful World of Linux 3.0Once upon a time, Joe Pranevich made a name for himself by writing comprehensive release notes for major kernel releases. He got out of that business during the 2.6.x series, but he is now back with a draft version of "The Wonderful World of Linux 3.0". Readers who are curious about what has happened since the 2.6.0 release may be interested in giving it a look. "This document describes just a few of the thousands of changes and improvements that have been made to the Linux kernel since the launch of Linux 2.6. I have attempted to make it as accessible as possible for a general audience, while not shying away from technical language when necessary."
Kernel development news 3.0 merge window part 2In the end, 7,333 non-merge changesets were pulled into the mainline kernel before Linus closed the merge window and decreed that the next release would be called "3.0". There have not been vast numbers of exciting new features added since last week's summary was written, but there are a few. The most significant user-visible changes include:
Changes visible to kernel developers include:
With the -rc1 release, Linus tagged the kernel "3.0.0" (with a new name of "Sneaky Weasel"). His stated intent is to drop the last digit during the stabilization period so that the final kernel would be just "3.0", but that depends on getting various user-space scripts fixed. Either way, the stable updates that most people will actually run will start with 3.0.1. Linus is clearly hoping for a relatively smooth development cycle this time around; he has hinted that he may be fussier than usual about the fixes that he'll pull from now on. 3.0, it seems, is supposed to be boring, just to drive home the point that the version number change does not really mean much. The final release, boring or not, can be expected sometime in the first half of July.
Forking the ARM kernel?In the last few months, many people have suggested forking the ARM kernel and maintaining it as a separate project. While the reasons for forking ARM may seem attractive to some, it turns out that it really doesn't make very much sense for either the ARM community or the kernel as a whole. Here are the most common reasons given for this suggestion:
Let's have a look at these reasons. Time to marketThe time-to-market advocates reason that it takes less time to hack up a new driver than to get it accepted upstream. I know I'm old school and do not understand the rapidly changing world and new challenges of the semiconductor industry anymore, but I still have enough engineering knowledge and common sense to know that there is no real concept of totally disconnected "new" things in that industry. Most of an SoC's IP blocks (functionality licensed for inclusion in an SoC) will not be new. So what happens to time to market the second time an IP block is used? If the driver is upstream, it can simply be reused. But all too often, a new driver gets written from scratch, complete with a new set of bugs that must be fixed. Are vendors really getting better time to market by rewriting new drivers for old IP blocks on every SoC they sell? In addition, the real time to market for a completely new generation of chips is not measured in weeks. The usual time frame for a new chip, from the marketing announcement to real silicon usable in devices is close to a year. This should be ample time to get a new driver upstream. In addition, the marketing announcement certainly does not happen the day after the engineering department met for beers after work and some genius engineer sketched the new design on a napkin at the bar, so most projects have even more time for upstreaming. The one-off nature of embeddedIt's undisputed that embedded projects, especially in the consumer market, tend to be one-off, but it is also a fact that the variations of a given SoC family have a lot in common and differ only in small details. In addition, variations of a given hardware type - e.g. the smartphone family which differs in certain aspects of functionality - share most of the infrastructure. Even next generation SoCs often have enough pieces of the previous generation embedded into them as there is no compelling reason to replace already proven-to-work building blocks when the functionality is sufficient. Reuse of known-to-work parts is not a new concept and is, unsurprisingly, an essential part of meeting time-to-market goals. We recently discovered the following gem. An SoC with perfect support for almost all peripheral components that was already in the mainline kernel underwent a major overhaul. It replaced the CPU core, while the peripheral IP blocks remained the same - except for a slightly different VHDL glue layer which was necessary to hook them up to the new CPU core. Now the engineer in me would have expected that the Linux support for the resulting SoC generation B would have just been a matter of adjusting the existing drivers. The reality taught us that the vendor assigned a team to create the support for the "new" SoC and all the drivers got rewritten from scratch. While we caught some of the drivers in review, some of the others went into the mainline, so now we have two drivers for the same piece of silicon that are neither bug nor feature compatible. I have a really hard time accepting that rewriting a dozen drivers from scratch is faster than sitting down and identifying the existing - proven to work - drivers and modifying them for the new SoC design. The embedded industry often reuses hardware. Why not reuse software, too? The SoC diversityOf course, every SoC vendor will claim that its chip is unique in all aspects and so different that sharing more than a few essential lines of code is impossible. That's understandable from the marketing side, but if you look at the SoC data sheets, the number of unique peripheral building blocks is not excitingly large. Given the fact that the SoC vendors target the same markets and the same customer base, that's more or less expected. A closer look reveals that different vendors often end up using the same or very similar IP blocks for a given functionality. There are only a limited number of functional ways to implement a given requirement in hardware and there are only a few relevant IP block vendors that ship their "unique" building blocks to all of the SoC vendors. The diversity is often limited to a different arrangement of registers or the fact that one vendor chooses a different subset of functionality than the other. We have recently seen consolidation work in the kernel which proves this to be correct. When cleaning up the interrupt subsystem I noticed that there are only two widely used types of interrupt controllers. Without much effort it was possible to replace the code for more than thirty implementations of "so different" chips with a generic implementation. A similar effort is on the way to replace the ever-repeating pattern of GPIO chip support code. These are the low hanging fruit and there is way more potential for consolidation. Avoiding the useless workDealing with maintainers and their often limited time for review is seen as a bottleneck. The extra work which results in addressing the review comments is also seen as waste of time. The number of maintainers and the time available for review is indeed a limiting factor that needs to be addressed. The ability to review is not limited to those who maintain a certain subsystem of the kernel and we would like to see more people participating in the review process. Spending a bit of time reviewing other people's code is a very beneficial undertaking as it opens one's mind to different approaches and helps to better understand the overall picture. On the other side, getting code reviewed by others is beneficial as well and, in general, leads to better and more maintainable code. It also helps in avoiding mistakes in the next project. In a recent review, which went through many rounds, a 1200-line driver boiled down to 250 lines and at least a handful of bugs and major mistakes got fixed. When I have the chance to talk to developers after a lengthy review process, most of them concede that the chance to learn and understand more about the Linux way of development by far outweighs the pain of the review process and the necessary rework. When looking at later patches I've often observed that these developers have improved and avoided the mistakes they made in their first attempts. So review is beneficial for the developer and for their company as it helps writing better code in a more efficient way. I call out those who still claim that review and the resulting work is a major obstacle as hypocrites who are trying to shift the blame for other deficiencies in their companies to the kernel community. One deficiency is assigning proprietary RTOS developers to write Linux kernel code without teaching them how to work with the Linux community. There is no dishonor in not knowing how to work with the Linux community; after all, every kernel developer including myself started at some point without knowing it. But it took me time to learn how to work with the community, and it will take time for proprietary RTOS developers to work with the community. It is well worth the time and effort. What would be solved by forking the ARM kernel?Suppose there was an ARM-specific Git repository which acted as a dumping ground for all of the vendor trees. It would pull in the enhancements of the real mainline kernel from time to time so that the embedded crowd gets the new filesystems, networking features, etc. Extrapolating the recent flow of SoC support patches into Linux and removing all the sanity checks on them would result in a growth rate of that ARM tree which would exceed the growth rate of the overall mainline kernel in no time. And what if the enhancements from mainline require changes to every driver in the ARM tree, as was required for some of my recent work? Who makes those changes? If the drivers are in mainline, the drivers are changed as part of the enhancement. If there is a separate ARM fork, some ARM-fork maintainer will have to make these changes. And who is going to maintain such a tree? I have serious doubts that there will surface a sufficient number of qualified maintainers out of the blue who have the bandwidth and the experience to deal with such a flood of changes. So to avoid the bottleneck that is one of the complaints when working with mainline, the maintainers would probably just have the role of integrators who merely aggregate the various vendor trees in a central place. What's the gain of such an exercise? Nothing as far as I can see, it would just allow everyone to claim that all of their code is part of a mysterious ARM Git tree and, of course, it would fulfill the ultimate "time to market" requirements, in the short term, anyway. How long would an ARM fork be sustainable?I seriously doubt that an ARM fork would work longer than a few kernel cycles simply because the changes to core code will result in a completely unmaintainable #ifdef mess with incompatible per-SoC APIs that will drive anyone who has to "maintain" such a beast completely nuts in no time. I'm quite sure that none of the ARM-fork proponents has ever tried to pull five full-flavored vendor trees into a single kernel tree and deal with the conflicting changes to DMA APIs, driver subsystems, and infrastructure. I know that maintainers of embedded distribution kernels became desperate in no time exactly for those reasons and I doubt that any reasonable kernel developer is insane enough to cope with such an horror for more than a couple of months. Aside from the dubious usefulness, such an ARM fork would cut off the ARM world from influencing the overall direction of the Linux kernel entirely. ARM would become a zero-interest issue for most of the experienced mainline maintainers and developers as with other out-of-tree projects. I doubt that the ARM industry can afford to disconnect itself in such a way, especially as the complexity of operating-system-level software is increasing steadily. Is there a better answer?There is never an ultimate answer which will resolve all problems magically, but there are a lot of small answers which can effectively address the main problem spots. One of the root causes for the situation today is of a historical nature. For over twenty years the industry dealt with closed source operating systems where changes to the core code were impossible and collaboration with competitors was unthinkable and unworkable. Now, after moving to Linux, large parts of the industry still think in this well-known model and let their engineers - who have often worked on other operating systems before working on Linux - just continue the way they enabled their systems in the past. This a perfect solution for management as well, because the existing structures and the idea of top-down software development and management still applies. That "works" as long as the resulting code does not have to be integrated into the mainline kernel and each vendor maintains its own specialized fork. There are reasonable requests from customers for mainline integration, however, as it makes the adoption of new features easier, there is less dependence on the frozen vendor kernels, and, as seen lately, it allows for consolidation toward multi-platform kernels. The latter is important for enabling sensible distribution work targeted at netbooks, tablets, and similar devices. This applies even more for the long-rumored ARM servers which are expected to materialize in the near future. Such consolidation requires cooperation not only across the ARM vendors, it requires a collaborative effort across many parts of the mainline kernel along with the input of maintainers and developers who are not necessarily part of the ARM universe. So we need to help management understand that holding on to the known models is not the most efficient way to deal with the growing complexity of SoC hardware and the challenges of efficient and sustainable operating system development in the open source space. At the same time, we need to explain at the engineering level that treating Linux in the same way as other OS platforms is making life harder, and is at least partially responsible for the grief which is observed when code hits the mailing lists for review. Another area that we need to work on is massive collaborative consolidation, which has the preliminary that silicon vendors accept that, at least at the OS engineering level, their SoCs are not as unique as their marketing department wants them to believe. As I explained above, there are only a limited number of ways to implement a given functionality in hardware, which is especially true for hardware with a low complexity level. So we need to encourage developers to first look to see whether existing code might be refactored to fit the new device instead of blindly copying the closest matching driver - or in the worst case a random driver - and hacking it into shape somehow. In addition, the kernel maintainers need to be more alert to that fact as well and help to avoid the reinvention of the wheel. If driver reuse cannot be achieved, we can often pull out common functionality into core code and avoid duplication that way. There is a lot of low hanging fruit here, and the Linux kernel community as a whole needs to get better and spend more brain cycles on avoiding duplication. One step forward was taken recently with the ARM SoC consolidation efforts that were initiated by Linaro. But this only will succeed when we are aware of the conflicts with the existing corporate culture and address these conflicts at the non-technical level as well. SecrecyAside from the above issues the secrecy barrier is going to be the next major challenge. Of course a silicon vendor is secretive about the details of its next-generation SoC design, but the information which is revealed in marketing announcements allows us to predict at least parts of the design pretty precisely. The most obvious recent example is the next-generation ARM SoCs from various vendors targeted for the end of 2011. Many of them will come with USB 3.0 support. Going through the IP block vendor offerings tells us that there are less USB 3.0 IP blocks available for integration than there are SoC vendors who have announced new chips with USB 3.0 support. That means that there are duplicate drivers in the works and I'm sure that, while the engineers are aware of this, no team is allowed to talk to the competitor's team. Even if they would be allowed to do so, it's impossible to figure out who is going to use which particular IP block. So we will see several engineering teams fighting over the "correct" implementation to be merged as the mainline driver in a couple of month when the secrecy barrier has been lifted. Competing implementations are not a bad thing per se, but the inability to exchange information and discuss design variants is not helping anyone in the "time to market" race. I seriously doubt that any of the to-be-released drivers will have a relevant competitive advantage and even if one does, it will not last very long when the code becomes public. It's sad that opportunities to collaborate and save precious engineering resources for all involved parties are sacrificed in favor of historical competition-oriented behaviour patterns. These patterns have not been overhauled since they were invented twenty or more years ago and they have never been subject to scrutiny in the context of competition on the open source operating system playground. A way forwardThe ever-increasing complexity of hardware, which causes more complex operating-system-level software, caused a shortage of high-profile OS level software developers long ago which cannot be counterbalanced by either money or by assigning a large enough number of the cheapest available developer resources to it. The ARM universe is diversified enough that there is no chance for any of the vendors to get hold of a significant enough number of outstanding kernel developers to cause any serious damage to their competitors. That is particularly true given the fact that those outstanding developers generally prefer to work in the open rather than a semi-closed environment. It's about time for managers to rethink their competition models and start to comprehend and utilize the massive advantage of collaborative models over an historic and no-longer-working model that assumes the infinite availability of up-to-the-task resources when there is enough money thrown into the ring. Competent developers certainly won't dismiss the chance to get some extra salary lightly, but the ability to get a way more enjoyable working environment for a slightly smaller income is for many of them a strong incentive to refuse the temptation. It's an appealing thought for me that there is no "time to market" howto, no "shareholder value" handbook, and no "human resources management" course which ever took into consideration that people might be less bribable than generally expected, especially if this applies to those people who are some of the scarcest resources in the industry. I hope that managers working for embedded vendors will start to understand how Open Source works and why there is an huge benefit to work with the community. After all, the stodgy old server vendors were able to figure out how to work with the Linux community, so it cannot be that hard for the fast-moving embedded vendors. However, the realist in me - sometimes called "the grumpy old man" - who has worked in the embedded industry for more than twenty five years does not believe that at all. For all too many SoC vendors, the decision to "work" with the community was made due to outside pressure and an obsession with following the hype. Outside pressure is not what the open source enthusiasts might hope for: the influence of the community itself. No, it's simply the pressure made by (prospective) customers who request that the chip is - at least basically - supported by the mainline kernel. Following the hype is omnipresent and it always seems to be a valid argument to avoid common-sense-driven decisions based on long-term considerations. The eternal optimist in me still has hope that the embedded world will become a first class citizen in the Linux community sooner rather than later. The realist in me somehow doubts that it will happen before the "grumpy old man" is going to retire.
Object-oriented design patterns in the kernel, part 1Despite the fact that the Linux Kernel is mostly written in C, it makes broad use of some techniques from the field of object-oriented programming. Developers wanting to use these object-oriented techniques receive little support or guidance from the language and so are left to fend for themselves. As is often the case, this is a double-edged sword. The developer has enough flexibility to do really cool things, and equally the flexibility to do really stupid things, and it isn't always clear at first glance which is which, or more accurately: where on the spectrum a particular approach sits.Instead of looking to the language to provide guidance, a software engineer must look to established practice to find out what works well and what is best avoided. Interpreting established practice is not always as easy as one might like and the effort, once made, is worth preserving. To preserve that effort on your author's part, this article brings another installment in an occasional series on Linux Kernel Design Patterns and attempts to set out - with examples - the design patterns in the Linux Kernel which effect an object-oriented style of programming. Rather than providing a brief introduction to the object-oriented style, tempting though that is, we will assume the reader has a basic knowledge of objects, classes, methods, inheritance, and similar terms. For those as yet unfamiliar with these, there are plenty of resources to be found elsewhere on the web. Over two weeks we will look for patterns in just two areas: method dispatch and data inheritance. Despite their apparent simplicity they lead to some rich veins for investigation. This first article will focus on method dispatch. Method DispatchThe large variety of styles of inheritance and rules for its usage in languages today seems to suggest that there is no uniform understanding of what "object-oriented" really means. The term is a bit like "love": everyone thinks they know what it means but when you get down to details people can find they have very different ideas. While what it means to be "oriented" might not be clear, what we mean by an "object" does seem to be uniformly agreed upon. It is simply an abstraction comprising both state and behavior. An object is like a record (Pascal) or struct (C), except that some of the names of members refer to functions which act on the other fields in the object. These function members are sometimes referred to a "methods". The most obvious way to implement objects in C is to declare a "struct" where some fields are pointers to functions which take a pointer to the struct itself as their first argument. The calling convention for method "foo" in object "bar" would simply be: bar->foo(bar, ...args); While this pattern is used in the Linux kernel it is not the dominant pattern so we will leave discussion of it until a little later. As methods (unlike state) are not normally changed on a per-object basis, a more common and only slightly less obvious approach is to collect all the methods for a particular class of objects into a separate structure, sometimes known as a "virtual function table" or vtable. The object then has a single pointer to this table rather than a separate pointer for each method, and consequently uses less memory. This then leads to our first pattern - a pure vtable being a structure which contains only function pointers where the first argument of each is a pointer to some other structure (the object type) which itself contains a pointer to this vtable. Some simple examples of this in the Linux kernel are the file_lock_operations structure which contains two function pointers each of which take a pointer to a struct file_lock, and the seq_operations vtable which contains four function pointers which each operate on a struct seq_file. These two examples display an obvious naming pattern - the structure holding a vtable is named for the structure holding the object (possibly abbreviated) followed by "_operations". While this pattern is common it is by no means universal. Around the time of 2.6.39 there are approximately 30 "*_operations" structures along with well over 100 "*_ops" structures, most if not all of which are vtables of some sort. There are also several structs such as struct mdk_personality which are essentially vtables but do not have particularly helpful names. Among these nearly 200 vtable structures there is plenty of variability and so plenty of scope to look for interesting patterns. In particular we can look for common variations from the "pure vtable" pattern described above and determine how these variations contribute to our understanding of object use in Linux. NULL function pointersThe first observation is that some function pointers in some vtables are allowed to be NULL. Clearly trying to call such a function would be futile, so the code that calls into these methods generally contains an explicit test for the pointer being NULL. There are a few different reasons for these NULL pointers. Probably easiest to justify is the incremental development reason. Because of the way vtable structures are initialized, adding a new function pointer to the structure definition causes all existing table declarations to initialise that pointer to NULL. Thus it is possible to add a caller of the new method before any instance supports that method, and have it check for NULL and perform a default behavior. Then as incremental development continues those vtable instances which need it can get non-default methods. A recent example is commit 77af1b2641faf4 adding set_voltage_time_sel() to struct regulator_ops which acts on struct regulator_dev. Subsequent commit 42ab616afe8844 defines that method for a particular device. This is simply the most recent example of a very common theme. Another common reason is that certain methods are not particularly meaningful in certain cases so the calling code simply tests for NULL and returns an appropriate error when found. There are multiple examples of this in the virtual filesystem (VFS) layer. For instance, the create() function in inode_operations is only meaningful if the inode in question is a directory. So inode_operations structures for non-directories typically have NULL for the create() function (and many others) and the calling code in vfs_create() checks for NULL and returns -EACCES. A final reason that vtables sometimes contain NULL is that an element of functionality might be being transitioned from one interface to another. A good example of this is the ioctl() operation in file_operations. In 2.6.11, a new method, unlocked_ioctl() was added which was called without the big kernel lock held. In 2.6.36, when all drivers and filesystems had been converted to use unlocked_ioctl(), the original ioctl() was finally removed. During this transition a file system would typically define only one of two, leaving the other defaulting to NULL. A slightly more subtle example of this is read() and aio_read(), also in file_operations, and the corresponding write() and aio_write(). aio_read() was introduced to support asynchronous IO, and if it is provided the regular synchronous read() is not needed (it is effected using do_sync_read() which calls the aio_read() method). In this case there appears to be no intention of ever removing read() - it will remain for cases where async IO is not relevant such as special filesystems like procfs and sysfs. So it is still the case that only one of each pair need be defined by a filesystem, but it is not simply a transition, it is a long-term state. Though there seem to be several different reasons for a NULL function pointer, almost every case is an example of one simple pattern - that of providing a default implementation for the method. In the "incremental development" examples and the non-meaningful method case, this is fairly straightforward. e.g. the default for inode->create() is simply to return an error. In the interface transition case it is only slightly less obvious. The default for unlocked_ioctl() would be to take the kernel lock and then call the ioctl() method. The default for read() is exactly do_sync_read() and some filesystems such as ext3 actually provide this value explicitly rather than using "NULL" to indicate a default. With that in mind, a little reflection suggests that if the real goal is to provide a default, then maybe the best approach would be to explicitly give a default rather than using the circuitous route of using a default of NULL and interpreting it specially. While NULL is certainly the easiest value to provide as a default - as the C standard assures us that uninitialized members of a structure do get set to NULL - it is not very much harder to set a more meaningful default. I am indebted to LWN reader wahern for the observation that C99 allows fields in a structure to be initialized multiple times with only the final value taking effect and that this allows easy setting of default values such as by following the simple model:
#define FOO_DEFAULTS .bar = default_bar, .baz = default_baz
struct foo_operations my_foo = { FOO_DEFAULTS,
.bar = my_bar,
};
This will declare my_foo with a predefined default value for baz and a localized value for bar. Thus for the small cost of defining a few "default" functions and including a "_DEFAULTS" entry to each declaration, the default value for any field can easily be chosen when the field is first created, and automatically included in every use of the structure. Not only are meaningful defaults easy to implement, they can lead to a more efficient implementation. In those cases where the function pointer actually is NULL it is probably faster to test and branch rather than to make an indirect function call. However the NULL case is very often the exception rather than the rule, and optimizing for an exception is not normal practice. In the more common case when the function pointer is not NULL, the test for NULL is simply a waste of code space and a waste of execution time. If we disallow NULLs we can make all call sites a little bit smaller and simpler. In general, any testing performed by the caller before calling a method can be seen as an instance of the "mid-layer mistake" discussed in a previous article. It shows that the mid-layer is making assumptions about the behavior of the lower level driver rather than simply giving the driver freedom to behave in whatever way is most suitable. This may not always be an expensive mistake, but it is still best avoided where possible. Nevertheless there is a clear pattern in the Linux kernel that pointers in vtables can sometimes be NULLable, typically though not always to enable a transition, and the call sites should in these cases test for NULL before proceeding with the call. The observant reader will have noticed a hole in the above logic denouncing the use NULL pointers for defaults. In the case where the default is the common case and where performance is paramount, the reasoning does not hold and a NULL pointer could well be justified. Naturally the Linux kernel provides an example of such a case for our examination. One of the data structures used by the VFS for caching filesystem information is the "dentry". A "dentry" represents a name in the filesystem, and so each "dentry" has a parent, being the directory containing it, and an "inode" representing the named file. The dentry is separate from the inode because a single file can have multiple names (so an "inode" can have multiple "dentry"s). There is a dentry_operations vtable with a number of operations including, for example, "d_compare" which will compare two names and "d_hash" which will generate a hash for the name to guide the storage of the "dentry" in a hash table. Most filesystems do not need this flexibility. They treat names as uninterpreted strings of bytes so the default compare and hash functions are the common case. A few filesystems define these to handle case-insensitive names but that is not the norm. Further, filename lookup is a common operation in Linux and so optimizing it is a priority. Thus these two operations appear to be good candidates where a test for NULL and an inlined default operation might be appropriate. What we find though is that when such an optimization is warranted it is not by itself enough. The code that calls d_compare() and d_hash() (and a couple of other dentry operations) does not test these functions for NULL directly. Rather they require that a few flag bits (DCACHE_OP_HASH, DCACHE_OP_COMPARE) in the "dentry" are set up to indicate whether the common default should be used, or whether the function should be called. As the flag field is likely to be in cache anyway, and the dentry_operations structure will often be not needed at all, this avoids a memory fetch in a hot path. So we find that the one case where using a NULL function pointer to indicate a default could be justified, it is not actually used; instead, a different, more efficient, mechanism is used to indicate that the default method is requested. Members other than function pointersWhile most vtable-like structures in the kernel contain exclusively function pointers, there are a significant minority that have non-function-pointer fields. Many of these appear on the surface quite arbitrary and a few closer inspections suggest that some of them result of poor design or bit-rot and their removal would only improve the code. There is one exception to the "functions only" pattern that occurs repeatedly and provides real value, and so is worth exploring. This pattern is seen in its most general form in struct mdk_personality which provides operations for a particular software RAID level. In particular this structure contains an "owner", a "name", and a "list". The "owner" is the module that provides the implementation. The "name" is a simple identifier: some vtables have string names, some have numeric names, and it is often called something different like "version", "family", "drvname", or "level". But conceptually it is still a name. In the present example there are two names, a string and a numeric "level". The "list", while part of the same functionality, is less common. The mdk_personality structure has a struct list_head, as does struct ts_ops. struct file_system_type has a simple pointer to the next struct file_system_type. The underlying idea here is that for any particular implementation of an interface (or "final" definition of a class) to be usable, it must be registered in some way so that it can be found. Further, once it has been found it must be possible to ensure that the module holding the implementation is not removed while it is in use. There seem to be nearly as many styles of registration against an interface in Linux as there are interfaces to register against, so finding strong patterns there would be a difficult task. However it is fairly common for a "vtable" to be treated as the primary handle on a particular implementation of an interface and to have an "owner" pointer which can be used to get a reference on the module which provides the implementation. So the pattern we find here is that a structure of function pointers used as a "vtable" for object method dispatch should normally contain only function pointers. Exceptions require clear justification. A common exception allows a module pointer and possible other fields such as a name and a list pointer. These fields are used to support the registration protocol for the particular interface. When there is no list pointer it is very likely that the entire vtable will be treated as read-only. In this case the vtable will often be declared as a const structure and so could even be stored in read-only memory. Combining Methods for different objectsA final common deviation from the "pure vtable" pattern that we see in the Linux kernel occurs when the first argument to the function is not always the same object type. In a pure vtable which is referenced by a pointer in a particular data structure, the first argument of each function is exactly that data structure. What reason could there be for deviating from that pattern? It turns out that there are few, some more interesting than others. The simplest and least interesting explanation is that, for no apparent reason, the target data structure is listed elsewhere in the argument list. For example all functions in struct fb_ops take a struct fb_info. While in 18 cases that structure is the first argument, in five cases it is the last. There is nothing obviously wrong with this choice and it is unlikely to confuse developers. It is only a problem for data miners like your author who need to filter it out as an irrelevant pattern. A slight deviation on this pattern is seen in struct rfkill_ops where two functions take a struct rkfill but the third - set_block() - takes a void *data. Further investigation shows that this opaque data is exactly that which is stored in rfkill->data, so set_block() could easily be defined to take a struct rfkill and simply to follow the ->data link itself. This deviation is sufficiently non-obvious that it could conceivably confuse developers as well as data miners and so should be avoided. The next deviation in seen for example in platform_suspend_ops, oprofile_operations, security_operations and a few others. These take an odd assortment of arguments with no obvious pattern. However these are really very different sorts of vtable structures in that the object they belong to are singletons. There is only one active platform, only one profiler, only one security policy. Thus the "object" on which these operations act is part of the global state and so does not need to be included in the arguments of any functions. Having filtered these two patterns out as not being very interesting we are left with two that do serve to tell us something about object use in the kernel. quota_format_ops and export_operations are two different operations structures that operate on a variety of different data structures. In each case the apparent primary object (e.g. a struct super_block or a struct dentry) already has a vtable structure dedicated to it (such as super_operations or dentry_operations) and these new structures add new operations. In each case the new operations form a cohesive unit providing a related set of functionality - whether supporting disk quotas or NFS export. They don't all act on the same object simply because the functionality in question depends on a variety of objects. The best term from the language of object-oriented programming for this is probably the "mixin". Though the fit may not be perfect - depending on what your exact understanding of mixin is - the idea of bringing in a collection of functionality without using strict hierarchical inheritance is very close to the purpose of quota_format_ops and export_operations. Once we know to be on the lookout for mixins like these we can find quite a few more examples. The pattern to be alert for is not the one that led us here - an operations structure that operates on a variety of different objects - but rather the one we found where the functions in an "operations" structure operate on objects that already have their own "operations" structure. When an object has a large number of operations that are relevant and these operations naturally group into subsets, it makes a lot of sense to divide them into separate vtable-like structures. There are several examples of this in the networking code where for instance both tcp_congestion_ops and inet_connection_sock_af_ops operate (primarily) on a struct sock, which itself has already got a small set of dedicated operations. So the pattern of a "mixin" - at least as defined as a set of operations which apply to one or more objects without being the primary operations for those objects - is a pattern that is often found in the kernel and appears to be quite valuable in allowing better modularization of code. The last pattern which explains non-uniform function targets is probably the most interesting, particularly in its contrast to the obvious application of object-oriented programming style. Examples of this pattern abound with ata_port_operations, tty_operations, nfs_rpc_ops and atmdev_ops all appearing as useful examples. However we will focus primarily on some examples from the filesystem layer, particularly super_operations and inode_operations. There is a strong hierarchy of objects in the implementation of a filesystem where the filesystem - represented by a "super_block" - has a number of files (struct inode) which may have a number of names or links (struct dentry). Further each file might store data in the page cache (struct address_space) which comprises a number of individual pages (struct page). There is a sense in which all of these different objects belong to the filesystem as a whole. If a page needs to be loaded with data from a file, the filesystem knows how to do that, and it is probably the same mechanism for every page in every file. Where it isn't always the same, the filesystem knows that too. So we could conceivably store every operation on every one of these objects in the struct super_block, as it represents the filesystem and could know what to do in each case. In practice that extreme is not really helpful. It is quite likely that while there are similarities between the storage of a regular file and a directory, there are also important differences and being able to encode those differences in separate vtables can be helpful. Sometimes small symbolic links are stored directly in the inode while larger links are stored like the contents of a regular file. Having different readlink() operations for the two cases can make the code a lot more readable. While the extreme of every operation attached to the one central structure is not ideal, it is equally true that the opposite extreme is not ideal either. The struct page in Linux does not have a vtable pointer at all - in part because we want to keep the structure as small as possible because it is so populous. Rather the address_space_operations structure contains the operations that act on a page. Similarly the super_operations structure contains some operations that apply to inodes, and inode_operations contains some operations that apply to dentries. It is clearly possible to have operations structures attached to a parent of the target object - providing the target holds a reference to the parent, which it normally does - though it is not quite so clear that it is always beneficial. In the case of struct page which avoids having a vtable pointer altogether the benefit is clear. In the case of struct inode which has its own vtable pointer, the benefit of having some operations (such as destroy_inode() or write_inode()) attached to the super_block is less clear. As there are several vtable structures where any given function pointer could be stored, the actual choice is in many cases little more than historical accident. Certainly the proliferation of struct dentry operations in inode_operations seems to be largely due to the fact that some of them used to act directly on the inode, but changes in the VFS eventually required this to change. For example in 2.1.78-pre1, each of link(), readlink(), followlink() (and some others which are now defunct) were changed from taking a struct inode to take a struct dentry instead. This set the scene for "dentry" operations to be in inode_operations, so when setattr and getattr were added for 2.3.48, it probably seemed completely natural to include them in inode_operations despite the fact that they acted primarily on a dentry. Possibly we could simplify things by getting rid of dentry_operations altogether. Some operations that act on dentries are already in inode_operations and super_operations - why not move them all there? While dentries are not as populous as struct page there are still a lot of them and removing the "d_op" field could save 5% of the memory used by that structure (on x86-64). With two exceptions, every active filesystem only has a single dentry operations structure in effect. Some filesystem implementations like "vfat" define two - e.g. one with case-sensitive matching and one with case-insensitive matching - but there is only one active per super-block. So it would seem that the operations in dentry_operations could be moved to super_operations, or at least accessed through "s_d_op". The two exceptions are ceph and procfs. These filesystems use different d_revalidate() operations in different parts of the filesystem and - in the case of procfs - different d_release() operations. The necessary distinctions could easily be made in per-superblock versions of these operations. Do these cases justify the 5% space cost? Arguably not. Directly embedded function pointersFinally it is appropriate to reflect on the alternate pattern mentioned at the start, where function pointers are stored directly in the object rather than in a separate vtable structure. This pattern can be seen in struct request_queue which has nine function pointers, struct efi which has ten function pointers, and struct sock which has six function pointers. The cost of embedded pointers is obviously space. When vtables are used, there is only one copy of the vtable and multiple copies of an object (in most cases) so if more than one function pointer is needed, a vtable would save space. The cost of a vtable is an extra memory reference, though cache might reduce much of this cost in some cases. A vtable also has a cost of flexibility. When each object needs exactly the same set of operations a vtable is good, but if there is a need to individually tailor some of the operations for each object, then embedded function pointer can provide that flexibility. This is illustrated quite nicely by the comment with "zoom_video" in struct pcmcia_socket /* Zoom video behaviour is so chip specific its not worth adding this to _ops */ So where objects are not very populous, where the list of function pointers is small, and where multiple mixins are needed, embedded function pointers are used instead of a separate vtable. Method Dispatch SummaryIf we combine all the pattern elements that we have found in Linux we find that:
So in exploring the Linux Kernel code we have found that even though it is not written in an object-oriented language, it certainly contains objects, classes (represented as vtables), and even mixins. It also contains concepts not normally found in object-oriented languages such as delegating object methods to a "parent" object. Hopefully understanding these different patterns and the reasons for choosing between them can lead to more uniform application of the patterns across the kernel, and hence make it easier for a newcomer to understand which pattern is being followed. In the second part of our examination of object oriented patterns we will explore the various ways that data inheritance is achieved in the Linux kernel and discuss the strengths and weaknesses of each approach so as to see where each is most appropriate.
Patches and updates Kernel trees
Core kernel code
Development tools
Device drivers
Filesystems and block I/O
Memory management
Architecture-specific
Security-related
Miscellaneous
Page editor: Jonathan Corbet Distributions Illumos: the successor to the OpenSolaris communityIn the August 2010, storage vendor Nexenta Systems announced Illumos, as a project to free the OpenSolaris distribution. The ultimate goal of Illumos is twofold: to replace all proprietary code that is still in OpenSolaris by open source code, and to build an independent community around the former OpenSolaris code base. Now almost 10 months later, Nexenta Systems organized its Nexenta European User Conference in Amsterdam on May 20 and presented its plans for Illumos and the Nexenta products, so this is the perfect opportunity to take a look at where Illumos is heading. CodeThe Illumos code base forms or will form the foundation of a couple of distributions, including a storage appliance from Nexenta, a cloud platform from Joyent, and the general-purpose OpenIndiana (which is the spiritual successor of the OpenSolaris distribution). One of its initial goals was to replace the few remaining binary-only components of the OpenSolaris code base by open source counterparts. Today, Illumos still has some of these binary-only components, such as the NFS lock manager, the IKE daemon (Internet Key Exchange, part of IPsec), some support libraries, and a few drivers, but according to Garrett D'Amore (Director of Engineering at Nexenta Systems) the Illumos developers are busy addressing all those issues. Also, while last year compiling Illumos still depended on the proprietary Sun Studio compiler, Illumos now also builds using GCC, although it needs more testing. Illumos has already received code contributions from some hardware vendors, such as Areca and 4Front, and according to D'Amore there are additional code contributions in the pipeline, especially from vendors of host bus adapters and network interface cards. Moreover, Intel and LSI are technology partners of Nexenta, the main sponsor of Illumos, which means that Nexenta has access to specifications and engineering support of these companies. According to D'Amore, code contributions from both vendors will be coming. The most notable absentees here are AMD and Nvidia, which apparently don't have a friendly relationship with Illumos/Nexenta. However, in the comments of one of his blog posts, D'Amore explained that these relationships or non-relationships are not carved in stone: "I'm happy to be friendly with either NVDA or AMD or any other vendor that wants to reciprocate. Its just that at this point, only Intel of those three has stepped up for illumos or Nexenta." Growing painsRecently, there was some criticism about the way Illumos is being managed. Stamos Tolias has called for a fork because he is dissatisfied with the "narrow-minded, ZFS centric dictatorship and cathedral development model". Apparently he has funding and wants to go public with the fork in November, with project leadership coming from the University of Athens, financial support from an unspecified EU research fund, and EADS as a commercial partner. Tolias didn't respond to your author's questions for clarification. As some of the criticism by Tolias is directed personally at D'Amore, the latter responded to the allegations on his blog: First the claim is that I've got omnipotent control over illumos. This is absolutely false. While I created the project, and serve as technical lead, I've offered to step down if the developer-council and admin-council would like to me to do so. Notably my employer (Nexenta) has minority representation on both councils, and I've tried to keep the groups as neutral as possible. I said when I created the illumos project, and I still maintain, illumos is a community project, not a Nexenta one.
Further, D'Amore explains that Illumos is not under the absolute control of Nexenta: I've also handed over determination of the Advocate list (the list of people who get to approve and integrate submissions) to developer-council. So far Nexenta has 75% of the advocate slots, but this can change at the request of developer-council. Since about 75% of the contributions to the illumos code have come from my team at Nexenta, this should hardly be surprising. In fact, I've flatly refused to add any more Nexenta advocates, even though there are meritorious candidates, until we get broader representation here.
Currently, D'Amore is working with the Software Freedom Law Center to create a legal entity for Illumos, which would be a foundation completely independent of Nexenta. So while it may be true to some degree that Illumos development is not happening as openly and independently as it could be, Nexenta is doing its best to lay the groundwork for a real community-made Illumos. NexentaNexenta is building on the Illumos base, and the result is twofold: an open source GNU/Solaris operating system, Nexenta Core Platform (NCP) on the one hand (we looked at version 2 in 2009), and a commercial storage solution, NexentaStor, on the other hand. The latter is a ZFS-based storage solution that is typically purchased from systems integrators and VARs as a part of a total solution with certified hardware from vendors such as LSI, Quanta, Dell, Supermicro, and Intel. The key message from Nexenta is its so-called "Open Storage" approach: NexentaStor users don't face vendor lock-in for their data, because they can easily migrate away to any other operating system supporting (the same or a newer version of) ZFS, such as Solaris, FreeBSD, and so on. In contrast, the big storage vendors use a proprietary on-disk format, and their solutions are tightly coupled to their own hardware. With Nexenta, users can choose for themselves which hardware to buy. While the user space of previous Nexenta releases was based on Ubuntu, Nexenta 4 (which will be released at the end of 2011) will be based on Debian 6 ("Squeeze") instead. The Nexenta engineers made this decision because Debian is more portable and Ubuntu has too many patches that make use of specific Linux features which makes porting the user space to the Illumos kernel more difficult. While Nexenta 3 was still based on the OpenSolaris kernel, Nexenta 4 will be the first Illumos-based Nexenta product. So with the change from the OpenSolaris kernel to the Illumos kernel and from an Ubuntu user space to a Debian user space, Nexenta 4 will be a big switch. However, there's still a lot of "fear, uncertainty, and doubt" spread about ZFS and patents. Last year, storage vendor NetApp sent a threatening letter to its competitor Coraid, and as a result Coraid decided to stop reselling their ZFS based storage appliance (based on NexentaStor). There was also a mutual lawsuit between Oracle and NetApp, which had to do with NetApp's contention that ZFS violated certain of its patents on the one hand, and with NetApp's rights to use certain technologies that originated in Sun on the other hand. Oracle and NetApp settled their dispute, but the details of their agreement were never made public. Nexenta and any other companies using ZFS in their operating system surely don't have to be afraid of lawsuits by Oracle: the CDDL license that the ZFS code was made available under explicitly allows the use of any patents Oracle might have on the code. However, NetApp or any other party can still sue Nexenta for violations of their patents in the ZFS code, although since the beginning of this year Nexenta is part of the Open Invention Network (OIN). In his keynote speech at the Nexenta European User Conference, CEO Evan Powell made it clear that he considered this membership as a protection against potential patent infringement litigation by NetApp: "If NetApp wants to sue us, we can sue them with the core UNIX patents from Novell that are shared through the OIN." StormOSIn January 2011, Nexenta Systems also hired Andrew Stormont, the developer of StormOS, an Xfce-based derivative of Nexenta Core Platform. It's a desktop operating system for people who like the ZFS/apt combo that Nexenta offers, but don't want to install the desktop packages themselves (because Nexenta is not aimed at desktop users). Stormont is now working as a software engineer at Nexenta Systems, primarily packaging software for the upcoming Nexenta Core Platform 4, but he recently also submitted patches for Illumos. As part of his job at Nexenta Systems, Stormont gets one day a week to work on StormOS, and most of his work at Nexenta ends up benefiting StormOS too, but StormOS is still pretty much a one-man show since it started. The next release, StormOS Flash, will be based on NCP4 with some components from Debian 7 (Wheezy), such as newer X.Org and Xfce packages. One of the new features will be a graphical interface for Xfce to integrate ZFS snapshots in the file manager, like the Time Slider functionality in GNOME on OpenSolaris. A more experimental feature that Stormont is considering is FatELF to support 32 and 64-bit architectures in a single binary. The Illumos kernel is already capable of automatically booting up in either 32 or 64-bit mode based on the hardware it is running on, so it makes sense for the binaries sitting on top of the kernel to also automatically run in the correct mode. In an email interview with your author, Stormont explains which benefits FatELF could offer to StormOS: FatELF would remove the need for separate /bin and /bin/amd64, /lib/ and /lib/amd64 directories, and so on. It would also remove the need for the stinky isaexec command — which seems to be universally hated. It would also make packaging of 32/64 bit libraries easier — no need to hack 64 bit rules into Makefiles. And finally, it would make a portable install of StormOS possible, that could run in either 32/64 bit mode automatically. Most people cringe at the idea of FatELF, citing bloat as the main reason, but I think that is no longer an issue since most people have broadband and hard disks are large and cheap nowadays. I'm not sure if FatELF will ever make it into an official StormOS release, but I will definitely experiment with it.
Other companiesNexenta is not the only one backing Illumos. Joyent is another big contributor, and it hired some core Solaris people like Bryan Cantrill and Brendan Gregg. Joyent has published a Git repository with their patches to the Illumos tree. In his announcement of the tree, John Sonnenschein wrote: The illumos-joyent tree is a vehicle for Joyent to release patches to illumos quickly, in the same vein as the -mm Linux kernel tree. That is, it is not a fork nor an alternative development hub; both Joyent employees and community members are encouraged to bundle patches from illumos-joyent and fold them in to the mainline Illumos tree.
Some of these patches are fairly Joyent-specific, but others are useful more generally. Most of the patches are enhancements for zones functionality (operating-system level virtualization), as well as DTrace (the Solaris dynamic tracing framework) enhancements for better analytics in the cloud. Many of these changes will probably find their way into Illumos in the near future, although there's the practical issue that Illumos is using Mercurial while Joyent uses Git. Another code contributor is the database company Delphix, which hired Adam Leventhal who worked on RAID-Z (a redundancy scheme similar to RAID 5, using ZFS) and is also a co-inventor of DTrace. Delphix seems to prefer to keep its involvement more quiet, as they haven't announced anything with respect to their contributions, but a cursory look at their commits in the Illumos repository shows that they are definitely involved. So while Nexenta Systems is still the main contributor, there are others too. Google Summer of CodeAfter less than a year Illumos is already active as a mentoring organization for Google Summer of Code (GSoC) 2011. Two Illumos GSoC projects have been accepted: Harshit Jain will update Illumos to be able to boot with GRUB 2 and to support booting from a disk with a GUID partition table (GPT), which will be mentored by D'Amore, and Shashank Karkare will rewrite some of the Perl system components (like intrd and kstat) in C, under the supervision of Joyent's John Sonnenschein. The goal of the latter project is to remove Perl as a runtime dependency for Illumos. Furthermore, Nexenta also funded three students that didn't get into Google Summer of Code. For instance, Andrey Sokolov will work on supporting Linux binaries on Illumos without zones (branded zones are a way to run Linux binaries on Solaris in a virtual container), Bayard Bell will work on an IKEv2 daemon to replace IKE for Illumos, and Viktor Ikonnikov will write a GUI frontend for the service configuration tool svccfg. For future editions of GSoC or Illumos Students, Illumos has an extensive list of project ideas. ZFS working groupAnother interesting development under the Illumos umbrella, although it is somewhat under the radar, is the ZFS working group. This group was founded a few months ago by Adam Leventhal and Matthew Ahrens (who has worked on ZFS at Sun/Oracle) of Delphix and Garrett D'Amore. Ahrens is the leader of this group. The goal is to facilitate collaboration of the community around ZFS, by coordinating the work and exchanging ideas. In particular, the founders had concerns about fracturing ZFS into sub-communities of the various operating systems that are supporting the file system. The ZFS working group contains developers from Illumos/OpenIndiana, Mac OS X (most likely Don Brady developing ZFS support with his company Ten's Complement), FreeBSD (probably Pawel Dawidek who ported ZFS), Linux, and Solaris. According to D'Amore, Oracle has also a seat on the group. The workgroup's activities are not done in the open, and some of the members don't want their membership to be known, but it's the intention to publish results of the discussions regularly. The first result from the ZFS working group is the feature flags proposal. In the announcement, Matthew Ahrens wrote: The first product of the working group is the design for a ZFS on-disk versioning method that will allow for distributed development of ZFS on-disk format changes without further explicit coordination. This method eliminates the problem of two developers both allocating version number 31 to mean their own feature.
This "feature flags" versioning allows unknown versions to be identified, and in many cases the ZFS pool or filesystem can be accessed read-only even in the presence of unknown on-disk features. My proposal covers versioning of the SPA/zpool, ZPL/zfs, send stream, and allocation of compression and checksum identifiers (enum values).
As Ahrens said, this feature flags functionality is needed because ZFS development outside Oracle is now happening in a much more distributed way, compared to when only Oracle/Sun developed ZFS. After ZFS v28, Oracle has stopped publishing its source code, and the developers of Illumos, FreeBSD, and other ZFS-supporting operating systems don't want to wait until Oracle's next code drop after the release of Solaris 11, so development goes on. Thanks to the feature flags, these developers outside Oracle could add new compression algorithms or new RAID variants without breaking interoperability: in most cases the operating system should still be able to access a ZFS pool or file system in read-only mode if it has an unknown feature. Feature flags will be implemented this summer and integrated into Illumos. ConclusionIllumos is clearly going forward, although the way it is doing this is frustrating some people. The Illumos developers have more of a cathedral-like development model, while some of the more vocal members of the Illumos community would like more of a completely open bazaar-like development model like Linux is using. The call for a fork and some criticism about the closed nature of the ZFS working group are consequences of these sentiments. However, the critics forget that the OpenSolaris development model was much more closed. Although Sun released the bulk of the Solaris system code and called OpenSolaris "open source", there never was a real OpenSolaris community: it always depended too much on Sun and later Oracle, and both companies didn't make an effort to create an independent community around their "open source" project. Oracle's current development model for Solaris 11 is completely behind closed doors. So while Illumos is not yet the completely open Solaris code base and does not yet have the vibrant community that many OpenSolaris people have dreamed of, things are going in the right direction. It's not a Nexenta-only project anymore, there will be an independent Illumos foundation, and the project has attracted support of some big hardware vendors. Now it's just waiting on the Illumos-based Nexenta release and the first stable Illumos-based OpenIndiana release to get the ball rolling and regain the interest of all the people who left the OpenSolaris community during Oracle's radio silence.
Brief items Distribution quotes of the week
More rules will never save an organization. Sometimes you need rules,
but I think that for a group like Gentoo to work well we need to keep
them to a minimum. "Well, that's not written in black and white so I
won't cooperate until it is" is no reason for anybody to pause even a
moment before escalating. Unclear policies are a reason to assume
good intentions - not a reason to overlook obvious bad intentions.
You can't solve a people problem with a better algorithm.
-- Rich Freeman
A long time ago, far far away there was an Operating System named Red Hat Linux. In that Operating System there was an installer which had an option called "( ) Install Everything". Now "( ) Install Everything" was a wonderful tool for QA a release because one could get everything onto one system and find all the problems people could run into (oooh need more than 8GB to install stuff, oh look package XYZ conflicts with ABC, oh hey this package if installed will reboot your system.). It was also a bane of developers because people by their nature will install everything and then open bugs because one really only wanted Reboot-My-Box-Daily-0.1.rpm installed if you knew what you were doing. And because so many people would install everything and complain about things like "My goodness Red Hat Linux is bloated! It takes over 8 GB to install now.", it was decided to remove it.
-- Stephen
Smoogen
Linux Mint 11 releasedThe Linux Mint 11 ("Katya") release is available. It features improved multimedia support, an enhanced software manager, LibreOffice, and more; see the release notes and LWN's review of this release for more information.
Mageia 1Mageia 1 has been released. Mageia is a community driven fork of Mandriva that began in September 2010. This release contains tools that will be familiar to Mandriva users, and a migration guide is available to upgrade to this release from an existing Mandriva install. See the release notes for more information.
openSUSE renames OBS (openSUSE News)The openSUSE project has renamed openSUSE Build Service to the simpler and less confusing "Open Build Service", which has the nice property of retaining the OBS acronym. "In highlighting the benefits of OBS to the masses, the over-reaching assumption that the service is openSUSE-specific proved to be a deterrent. It clearly takes additional effort to convince a potential user that despite the name, openSUSE Build Service was not just for openSUSE. And the distribution-independent technological benefits became lost in the confusion. This effect is very apparent in face to face communication as youll almost immediately hear others saying "No, I'm a Fedora packager, this has nothing to do with me. Sorry." when they hear about OBS. This same effect [led] to less people reading articles or attending talks on the subject."
openSUSE 12.1 Milestone 1The first milestone on the road to openSUSE 12.1 is available for testing. "It is the first milestone, hence far from stable, but the images are now finally building, so we have a good starting point for further development."
Scientific Linux 6.1 Alpha 1 ReleaseScientific Linux has released the first alpha for Scientific Linux 6.1. "This release has all the new updated packages from The Upstream Vendors (TUV) Update 1. It will install, and it appears to work, but no major testing has been done."
Distribution News Debian GNU/Linux bits from the DPL: some oldiesDebian Project Leader Stefano Zacchiroli has an update on what he's been up to lately. "Since the last update of mine---released shortly after the recent elections---has been mostly about "DPL interaction tips & tricks", this one will cover the time span from mid April to now. Better get started ..."
Debian and Linux 3.0The Debian kernel maintainers look at the consequences of the change in kernel versioning. "The kernel team will not maintain linux-2.6 vs linux-3.0 packages. We will change the binary metapackages whose names include '2.6' into transitional packages, to be removed after 'wheezy', and we may rename the source packages linux-2.6 and linux-latest-2.6. There are likely to be many programs and build scripts that test for a kernel version prefix of '2.6' vs '2.4' and will behave incorrectly when they find '3.0'. Others require that there are at least 3 numeric components in the version string, which may cease to be true."
Fedora Fedora Scholarship 2011Ian Weller has announced that Ricky Elrod is the recipient of the 2011 Fedora Scholarship. In an interview Ricky talks about his contributions to Fedora. "A lot of my work so far within the infrastructure team has been internal things that end users don't directly see. Whether enhancing tools that we use to monitor services, upgrading things that we use to provide the best possible infrastructure for users and developers alike, and working with people to coordinate when things are happening that the infrastructure has to make any kind of change for."
Appointment to the Fedora Board and reminder of upcoming electionsJared K. Smith has announced that Guillermo Gomez has accepted a seat on the Fedora Board. "Guillermo has demonstrated his commitment to Fedora through his tireless efforts on a number of different fronts, including working on packaging, development, and ambassador programs. He has also shown his ability to mentor new contributors and help grow the Fedora community. I have no doubt that he'll continue these efforts while on the Board, and that he'll bring his perspective to help the Board as it makes decisions affecting the direction that Fedora takes. I would like to publicly thank Guillermo for his willingness to serve, and I hope the entire Fedora community will join me in welcoming him to the Board."
FUDCon APAC now open for bidsThere will be a FUDCon (Fedora Users and Developers Conference) in the Asia-Pacific region sometime around the end of this year. The bidding process will be open until June 23, 2011.
Retiring blogs.fedoraproject.org on 2011-07-01Kevin Fenzi outlines several reasons for the discontinuation of blogs.fedoraproject.org, including a lack of maintenance staff, the existence of other blogging services, and low usage.
Gentoo Linux Gentoo Council Nominations for 2011 to 2012Nominations for the Gentoo Council are open from June 3-17, 2011. "Nominated candidates must accept nominations before the nomination period closes to progress into the final ballot. Self nomination (and acceptance) is permitted."
Ubuntu family Ubuntu 6.06 LTS ("Dapper Drake") end of lifeAfter five years of updates, Ubuntu's first "long-term support" (LTS) release, Dapper Drake (6.06), has reached its end of life. "Ubuntu announced its 6.06 Server release 5 years ago, on June 1, 2006. For the LTS Server releases, Ubuntu committed to ongoing security and critical fixes for a period of 5 years. The maintenance period has now ended for Ubuntu 6.06 LTS Server. [...] Ubuntu 6.06 LTS was a major milestone for the Ubuntu project, being the first long-term release. Its retirement evokes memories of Ubuntu as a younger project, and reminds us of all that we've accomplished together in the five years since we released the "Dapper Drake"."
Ubuntu 9.10 (Karmic Koala) end-of-lifeKate Stewart has sent out a reminder that v9.10 (Karmic Koala) reached its end of life May 1, 2011. The supported upgrade path from Ubuntu 9.10 is via Ubuntu 10.04 LTS (Lucid Lynx).
Other distributions Linaro Kernel 2011.05-2.6.38 Snapshot ReleasedThe Linaro Kernel Working Group (KWG) has announced the availability of a kernel development snapshot: linux-linaro-11.05-2.6.38. "This snapshot is based on the 2.6.38.7 stable kernel with a number of changes developed by Linaro and integrated from upstream trees including the 2.6.39, OMAP, and Power Management trees to name a few."
Newsletters and articles of interest Distribution newsletters
How much GNU is there in GNU/Linux?Pedro Côrte-Real has posted the results of an investigation into the provenance of the code making up a Linux distribution. "Figure 1 shows the total LOC in Ubuntu natty split by the major projects that produce it. By this metric GNU software is about 8%. I didnt include GNOME in the GNU category because it seems to now be effectively run outside GNU but including that the total for GNU would be around 13%. I found two things to be really surprising in this chart. The first is that the kernel is actually comparable in size to all the GNU software. The second is that small projects actually dominate the total amount. It seems that at least for what Ubuntu packages, the origin of the software is highly dispersed."
Bacon: Six Months Of Rocking Ubuntu EventsJono Bacon has announced several upcoming events for Ubuntu contributors, including Ubuntu Developer Week, Ubuntu Cloud Days, Ubuntu Global Jam, and more. "Education and learning has always been an important part of the Ubuntu culture. It is important to us because we always want to present an environment in which everyone is welcome to participate, whatever your skills, location, or experience. We want to make it really easy to get involved in Ubuntu and be a part of the revolution. :-)"
Page editor: Rebecca Sobol Development Web-based development with Eclipse OrionWeb developers should have web-centric tools for development. That seems like a perfectly logical thesis, but in actual practice web developers are chained (figuratively) to their desktop with tools meant for standard development. The Eclipse Project is looking to change that with Orion, a web-based set of tools for developing web applications. It's still in very early stages, but is showing a lot of promise. The Eclipse project announced Orion in early January, and pushed out a hosted implementation in March. The initial contribution came from IBM's original Eclipse team as a "seed" to kick things off. The latest release, Orion 0.2 M7 was released at the beginning of May. Orion is completely separate from the Eclipse IDE that many developers are already familiar with. Orion is a set of loosely coupled components that are written in JavaScript (on the client side) and a server written in Java (using Jetty and Equinox). According to the Orion architecture overview, the vision for Orion is to feature browser-based tools that interact with data stored locally or to access data hosted in "the cloud" via REST APIs. The client is dual-licensed under the Eclipse Public License (EPL) and Eclipse Distribution License (EDL). The server is available only under the EPL. ![[Orion navigator]](/images/2011/orion-navigator-sm.png)
The proposed use cases for Orion are quite diverse. The scope for Orion is any kind of web development — from single-developer scenarios developing simple sites with PHP, JavaScript, HTML, and CSS, to collaborative development of JavaScript libraries, to development of frameworks like Orion itself that include server-side Java components and the JavaScript, CSS, and HTML client pieces. This isn't the first web-centric effort from the Eclipse Foundation. A web-based Eclipse 4.0 prototype was demonstrated at EclipseCon in 2008. That was abandoned, however, and Orion is a new codebase that contains very little from the current Eclipse codebase. Given that the world already has plenty of IDEs (like Eclipse) and development tools, you might wonder why it's necessary to create one that runs in the browser. Eclipse developer Boris Bokowski has a simple answer to that — Orion isn't meant to be an IDE. According to Bokowski, developers already use a lot of web-based software as part of development: Bugzilla, code review tools like Gerrit, reading documentation online, and code repositories with web-based tools like GitHub. But then they use desktop tools like a text editor or full-blown IDEs for actually coding and debugging — which isn't optimal. Bokowski describes it this way:
So all we are suggesting is that you move the remaining software
development activities to the browser, too. [...]
For this to make sense as a way to develop software, you'd want all those browser-based tools to work together. Ideally, you'd want something similar to the tight integration you typically get from a classic IDE, but without building another IDE. We want to build something that works well with existing tools out there on the web, and we cannot ask others to buy into a new programming model that we may come up with. Therefore, our approach has to be based on well-proven web architectures - using hyperlinks to navigate between different tools and resources, making data accessible in a RESTful way, and using web technologies like HTTP, JSON, OAuth, OpenID, and others. ![[Orion editor]](/images/2011/orion-editor-sm.png)
What's the advantage to that? Generally the same advantages that one derives from most web-based applications: Developers don't have to install an IDE, nor maintain one (assuming they're using a hosted instance provided by someone else). It fits well with web development, since the developer is already using the browser for testing and debugging. Being developed in JavaScript means that its target audience — web developers — should be able to extend and contribute to the project as well. Orion is also being designed to be very extensible, with a plugin architecture that's already working. However, Orion handles plugins a bit differently than you might be used to. Instead of installing a plugin on the server where Orion is hosted, Orion plugins are links to external sites that have the JavaScript for the plugin. This eliminates the need for developers to have server access to install plugins they wish to use. Instead they can simply link to (or create) plugins and point to them. ![[Orion plugins]](/images/2011/orion-plugins-sm.png)
The plugins, when installed, are persistent — that is, developers won't have to re-install plugins after pointing Orion to them the first time. One thing that seems to be missing, though, is any kind of vetting of the plugins. At the moment, Orion doesn't seem to have a plan for ensuring plugins are not malicious or providing a repository of known-good plugins. One hopes this will be added if the Orion community grows. The Orion developer principles recommend reusing existing tools rather than inventing new ones whenever possible. Case in point, Orion is working on Firebug integration, and having Firebug invoke Orion as an editor. Getting and using OrionDevelopers have a couple of options for using Orion. One is to sign up for the hosted version of Orion, called OrionHub. You might want to hurry, though — according to the beta announcement, the service is open to the first 5,000 developers who sign up. Additional accounts may, or may not, be made available as the Orion community expands — the announcement doesn't make any promises. Alternately, you can download and install Orion on your own machine. The project offers builds for Windows, Mac OS X, and (of course) Linux. It is a slightly more involved affair than simply signing up for OrionHub, but it does provide more control and is much better for those who would like to hack on Orion as well as make use of it. One note of caution for those running their own instance of Orion — right now there's no upgrade or update functionality in Orion. This means that, as a rapidly developing project, you'll probably want to update it often but you'll be on your own for backing up user accounts and data. There's a little guidance on the wiki about wiping server data for demos of Orion that you can use to figure out how to back up accounts and data, but it's all up to the user at this point. After spending a bit of time with Orion to see what it has to offer, I have concluded that it has some growing to do before it's going to be suitable for many users. And, in fact, the FAQ for Orion says as much. Orion isn't yet ready for "real work," though the Orion team is using it themselves. Orion only recently gained things like syntax highlighting for HTML and still lacks the content assist (code completion) for JavaScript. Some content assist is available for CSS, but it's a work in progress. Eclipse uses Textmate's grammars for highlighting. Those interested in hacking in highlighting for their own favorite language can find details on the wiki. Orion does have quite a few features for working with Git repositories, and some basic functionality for pushing changes from OrionHub into GitHub projects. But it's still evolving. You need to use another Git client to actually push or pull changes. Orion also lacks any support for other version control systems, so users of Subversion, Mercurial, or any other VCSes are out of luck for the time being. Orion going forwardThe next milestone for Orion is planned for June 3rd. This release should simplify server configuration, move to Dojo 1.6, and provide extension points for content assist, keybindings, and working with different editors. Orion is already attracting some interest from Mozilla, with an eye towards using the Orion editor in the Mozilla codebase. From the looks of the Orion development list the project is fairly active in general. For now, Orion is not a tool for serious development. It has the beginnings of a competent development environment, but it's not something that one would trust for real projects now, or likely in the next year. If you're hoping to use it to knock out your next web site, you had better have low expectations or be willing to work with relatively primitive tools. However, if the concept of a browser-based web development environment stirs your fancy, then the Orion project is open for contributions. It should be interesting to see how the project develops, and if (or when) the browser-based model overtakes desktop tools.
Brief items Quotes of the week
Code that assumes that ASCII is good enough for writing English
properly is stupid, shortsighted, illiterate, broken, evil, and
wrong. Off with their heads! If that seems too extreme, we can
compromise: henceforth they may type only with their big toe from
one foot (the rest still be ducktaped).
-- Tom
Christiansen
Unfortunately, there is a problem with Free Software developers,
firstly - they often don't wear suits, and (get this) some have
beards: which just shows you the kind of schmucks they are. But
worse - they have odd, meritocratic, collaborative decision making
processes, that don't come up with suitably corporate answers.
-- Michael
Meeks
Not surprisingly, a lot of software that claims to be 64 bits-ready
isn't. This touches all web navigators, most jit engines, and
probably lots more of software (our ports tree version of gnu-grep,
for instance).
-- Marc Espie
How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to "work", by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. This is yet another example of the patheticness that is modern software development. Instead of going headfront and fixing the actual problems, most systems cope out and just sweep the problem under the carpet, hoping no-one will notice.
PiTiVi 0.14 releasedVersion 0.14 of the PiTiVi video editor has been announced. New features include audio and video effects, previews in the import dialog, a "welcome dialog" to help new users get started, and more; see the release notes for more information.
Python 2.5.6 releasedPython 2.5.6 is out, for those who haven't moved on yet. "This is a source-only release that only includes security fixes. The last full bug-fix release of Python 2.5 was Python 2.5.4. Users are encouraged to upgrade to the latest release of Python 2.7 (which is 2.7.1 at this point). This release is most likely the final release of Python 2.5; under the current release policy, no security issues in Python 2.5 will be fixed after October, 2011."
Google open-sources WebRTCGoogle has announced (in early May) the release of the WebRTC code which, it hopes, will help to jump-start the creation of a new generation of communication applications. "Until now, real time communications required the use of proprietary signal processing technology that was mostly delivered through plug-ins and client downloads. With WebRTC, we are open sourcing the voice and video engine technologies from our acquisition of GIPS, giving developers access to state of the art signal processing technology, under a royalty free BSD style license. This will allow developers to create voice and video chat applications via simple HTML and JavaScript APIs."
Newsletters and articles Development newsletters from the last week
Duffy: Version-controlled, automagical backup and file sharing system with Sparkleshare and FedoraMáirín Duffy has put together a step-by-step tutorial on setting up Sparkleshare for remote file synchronization. "I've built a solution using Fedora and Sparkleshare - completely free and open source software - that over the past week has addressed all of these issues and has substantially improved the quality of my computing life. It backs my work files up to an internal corporate server and it backs my Fedora files up to a Fedora-maintained public server. I'm planning to configure it to back up some personal files to my Dreamhost account and some to my NAS at home."
Neary: Effective mentoring programsOver on his blog, Dave Neary investigates mentoring programs, like Google Summer of Code and others, to see what works and what doesn't. In particular, he looks at why so few of those who are mentored end up as project contributors, and what can be done to change that. "Mentored tasks should be small, bite-sized, and allow the apprentice to succeed or fail fast. This has a number of advantages: The apprentice who won't stick around, or who will accomplish nothing, has not wasted a lot of your mentor's time. The apprentice who will stay around gets a quick win, gets his name in the ChangeLog, and gains assurance in his ability to contribute. And the quick feedback loop is incredibly rewarding for the mentor, who sees his apprentice attack new tasks and increase his productivity in short order."
Page editor: Jonathan Corbet Announcements Brief items Ada Initiative "Seed 100" donation campaign opensThe Ada Initiative has started a fundraising campaign to find 100 folks willing to donate at two different levels: "Difference Engineer" for $512 or "Analytical Engineer" for $1024. Thank-you gifts are offered for each level. The Ada Initiative is a non-profit that seeks to increase the participation of women in open technology and culture. "The Ada Initiative is opening its first fundraising campaign, the Seed 100 campaign! The Seed 100 campaign will fund our remaining startup costs, in particular project development for projects like First Patch Week, and continued work on conference policy development, so that we can take them to future partners and larger donors."
FSF: Announcing our license recommendations guideBrett Smith of the Free Software Foundation has announced a license recommendation guide. "Today I'm happy to share something we've been working on for a little while: "How to choose a license for your own work" is a comprehensive set of license recommendations for new projects. This page explains what factors are important to consider when making licensing decisions, and suggests specific licenses for different scenarios. If you're starting a new project (whether it's software, documentation, or something else related) and unsure what license to use, you just need this one link to find our recommendations." (Bradley Kuhn has some comments on the recommendation site on his blog as well.)
Libertine Open Fonts Project releases version 5.0.0The Libertine Open Fonts Project has announced the release of version 5.0.0 of the open source font families Libertine and Biolinum. "Since 2003 the Libertine Open Fonts Project works on a versatile Unicode font family with an elegant, good-readable type face for daily and professional use. It is designed to give you an alternative for fonts like T*mes New Roman. We're creating /free/ software and publish our fonts under terms of the GPL and Open Font License (OFL)."
Linaro plan reviews (May 31st to June 8th)Linaro will be holding a series of conference calls to present the engineering plans for various work groups starting on May 31. Each one hour call will be held at 15:00 UTC with dial-in numbers available for most countries, and there will be an IRC channel for questions. "For each call we will provide a set of slides discussing features planned and the blueprints that specify and track them. The calls are open to the general public; anybody is welcome to dial in and listen." The first review will be the Power Management working group on Tuesday.
Oracle proposes donating OpenOffice.org to Apache Software FoundationOracle has proposed contributing the OpenOffice.org code to the Apache Software Foundation's (ASF's) incubator program. So far, there is no announcement per se from Oracle, but there is a collection of statements from people associated with Oracle, Apache, and, perhaps not surprisingly, IBM, including "'With today's proposal to contribute the OpenOffice.org code to The Apache Software Foundation's Incubator, Oracle continues to demonstrate its commitment to the developer and open source communities. Donating OpenOffice.org to Apache gives this popular consumer software a mature, open, and well established infrastructure to continue well into the future. The Apache Software Foundation's model makes it possible for commercial and individual volunteer contributors to collaborate on open source product development.' -- Luke Kowalski, vice president, Oracle Corporate Architecture Group."As might also be expected, The Document Foundation (which is behind the LibreOffice fork of OpenOffice.org) is a bit disappointed with the move: "The Document Foundation would welcome the reuniting of the OpenOffice.org and LibreOffice projects into a single community of equals in the wake of the departure of Oracle. The step Oracle has taken today was no doubt taken in good faith, but does not appear to directly achieve this goal. The Apache community, which we respect enormously, has very different expectations and norms — licensing, membership and more — to the existing OpenOffice.org and LibreOffice projects. We regret the missed opportunity but are committed to working with all active community members to devise the best possible future for LibreOffice and OpenOffice.org." In addition, IBM's ODF architect Rob Weir has issued a call for contributors to the ASF project if it gets accepted: "At Apache, you can't just walk in off the street, drop some code and call yourself an Apache project. There is a multi-step process for initiating, reviewing and approving a new project. We're at the first step, with the proposal submission, which Oracle made earlier today. This proposal will now be reviewed and voted on by the Apache Incubator Project Management Committee (PMC) over the next few days. If approved, the project then advances into incubation as a "Podling". Incubation at Apache is a probationary stage, where the project recruits new members, reviews the code to establish IP provenance, adapts the project to the Apache infrastructure, and so on."
Articles of interest Linux Foundation Monthly Newsletter: May 2011The May edition of the Linux Foundation Monthly Newsletter covers LinuxCon Japan, Linux T-shirt, Video Contest deadline, and several other topics.
HTC: no more locked-down phonesThe CEO of HTC has seemingly posted on Facebook that its phones will not be locked down from now on. "Today, I'm confirming we will no longer be locking the bootloaders on our devices. Thanks for your passion, support and patience."
Morevna Project progress, project problems...The Morevna Project seeks to create a full-length anime movie using open source software. The Morevna project blog has a status report after three years of development. "First of all, we weren't sure if the software we have chose is able to handle the tasks that we need. For Blender (for 3D works) everything was quite clear — that tool is stable enough and already proven with a series of Blender Foundation's open movie projects... But 3D in Morevna Project handles just a helper function — Morevna is mostly about 2D animation! And the all burden here comes to Synfig Studio, which is completely different case! Without any usage cases of such scale, questional stability, totally undeveloped character animation techniques... There were few examples of characters animation, but we wanted the complex motion. This most powerful open-source animation tool still had a lots of issues. We just weren't sure if this tool will be able to handle all the tasks (OK, well... in terms of faith we were sure, but have no idea how)." (Thanks to Paul Wise) LWN looked at the project in March 2010.
New Books The Architecture of Open Source ApplicationsThe Architecture of Open Source Applications is a new book with chapters on the design of a wide variety of programs, including Asterisk, bash, Eclipse, LLVM, Mercurial, Sendmail, Telepathy, and many more. It's available for purchase or downloadable under the terms of the CC Attribution 3.0 license; some readers have already taken advantage of that license to make an epub version available. Revenue from sales go to Amnesty International.
Calls for Presentations #PgWest 2011: Call for PapersPostgreSQL Conference West will be held September 27-30, 2011 in San Jose, California. The talk submission deadline is July 31.
Call for papers - PGConf.EU 2011PostgreSQL Conference Europe 2011 will be held on Ocober 18-21, 2011 in Amsterdam, The Netherlands. The submission deadline is August 21.
Upcoming Events PyCon Ireland 2011: Registration now openPyCon Ireland will be held October 8-9, 2011 in Dublin, Ireland. Early bird and student tickets are available and the call for talks is open.
Events: June 9, 2011 to August 8, 2011The following event listing is taken from the LWN.net Calendar.
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Whiskey presentation]](/images/conf/2011/lc-japan/whiskey-sm.jpg)
![[Linus Torvalds]](/images/conf/2011/lc-japan/LinusTorvalds-sm.jpg)