LWN Weekly Edition
Leading itemsSecurity
Kernel development
Distributions
Development
Announcements
->Multipage format
This page
Previous weekFollowing week
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for March 4, 2010SCALE 8x: Relational vs. non-relationalPostgreSQL hacker Josh Berkus set out to do some "mythbusting" about differences in database technologies in his talk at SCALE 8x. While there are plenty of differences between the various approaches taken by database systems, those are not really the ones that are being highlighted by the technical press. In particular, the so-called "NoSQL movement" makes for a great soundbite, but is "not very informative or accurate". Berkus went on to survey the current database landscape while giving advice on how to approach choosing a database for a particular application. This is a "more exciting time" to be a "database geek" than ever before, he said. Looking back seven years to 2003, he noted that there were essentially seven different free choices, all of which are SQL-based. In 2010, there are "dozens of new databases breeding like rabbits", with some 60 choices available. As an example of how quickly things are moving, Berkus noted that while he was in New Zealand at linux.conf.au, where a colleague was giving a related talk, two new databases were released. MythbustingBerkus likened the NoSQL term to a partition that is created by putting dolphins, clown fish, and 1958 Cadillacs on one side and octopuses, Toyota Priuses, and redwood trees on the other—labeled as the "NoFins" group. The non-relational databases that are lumped together as NoSQL have "radically different" organizations and use cases. But, that's not just true of the non-relational databases, it's also true for the various relational databases as well. Another myth that he pointed out was the "revolutionary" tag that gets associated with all of the new types of databases. Once again, that is a convenient soundbite that isn't accurate. He has not seen a new database algorithm since 2000, and all of the new crop of database systems are new implementations and combinations of earlier techniques. The new systems are not revolutionary, just evolutionary. As an example, he put up a slide with the following description of a database: "A database storing application-friendly formatted objects, each containing collections of attributes which can be searched through a document ID, or the creation of ad-hoc indexes as needed by the application." He noted that it applies equally well to one of his current favorites, CouchDB, which was created in 2007, and to the Pick database system—the original object of the description—which was created in 1965. Instead of a revolution, what we are seeing now is a "renaissance of non-relational databases". That description is far more accurate, Berkus said, and is a better way to view the change. It is a "big thing" that is going to "change the way that people use databases", so it is important to label it correctly. Another myth is that non-relational databases are "toys", which is something that is often pushed by people who work on relational systems. Berkus pointed out that many SCALE sponsors would disagree: Google using Bigtable, Facebook using Memcached, Amazon with Dynamo, and so on. The other side of that myth is that relational databases will become obsolete. Unsurprisingly, that myth is often promulgated by those who work on non-relational databases, and it is something that the relational community has heard before. Berkus pointed to a keynote speech in 2001 proclaiming that relational databases would be replaced with XML databases. He then asked if anyone even remembered or used XML databases; when even the crickets were silent, he pointed out that various relational and non-relational databases had hybridized with XML databases, incorporating the best features of XML databases into existing systems. He predicted that "over the next five years, we will see more hybridization" between different types of database technologies. "Relational databases are for when you need ACID transactions" was myth number five. Support for transactions is "completely orthogonal" to the relational vs. non-relational question. There are systems like Berkeley DB and Amazon Dynamo that provide robust transactions in non-relational databases, as well as MS Access and MySQL that provide SQL without transactions. The final myth that needs busting is the Lord of the Rings inspired "one ring theory of database use", Berkus said. There is "absolutely no reason" to choose one database for all of one's projects. He recommends choosing the database system that fits the needs of the application, or to use more than one, such as MySQL with Memcached or PostgreSQL with CouchDB. Another alternative is to use a hybrid, like MySQL NDB, which puts a distributed object database as a back-end to MySQL, or HadoopDB which puts PostgreSQL behind the Hadoop MapReduce implementation. So, what about relational vs. non-relational?Relational databases provide better transaction support than non-relational databases do, mostly because of the age and maturity of relational databases, Berkus said. Transaction support is something that many open source people don't know about because the most popular database (MySQL) doesn't implement it. Relational databases enforce data constraints and consistency because that is the basis of the relational model. There are other benefits of today's relational databases, he said, including complex reporting capabilities and vertical scaling to high-end hardware. He also noted that horizontal scaling was not that well-supported and that relational databases tend to have a high administrative overhead. On the question of SQL vs. Not-SQL, Berkus outlined the tradeoffs. SQL promotes portability, multiple application access, and has ways to manage database changes over time. There are many mature tools to work with SQL, but SQL is a full programming language that must be learned to take advantage of it. Not-SQL allows fast interfaces to the data, without impedance-matching layers, which in turn allows for faster development. Typically, there are no separate database administrators (DBAs) for Not-SQL databases, with programmers acting in that role. "It's always a tradeoff", Berkus said, but one place that a SQL-relational database makes the most sense is where you have "immortal data". If the data being stored has a life independent of the specific application and needs to be available to new applications down the road, SQL-relational is probably the right choice. How to chooseFor other situations, you need to define the "features you actually need to solve that particular problem" plus another list of features you'd like, "then go shopping". Chances are, he said, there is a database or combination of databases that fits your needs. He then went on to some specific application requirements, suggesting possible choices of database or databases to satisfy them.
The slides [PDF] from Berkus's talk have additional examples. The basic idea is that "different database systems do better at different tasks" and it is impossible for any database system to do everything well, "no matter what a vendor or project leader may claim". For those who are looking for open source solutions, he recommended the Open Source Database survey which Selena Deckelmann has put together. While it is, as yet, incomplete, it does list around a dozen lesser-known database systems. It is clear from the talk that it is an exciting time to be a database developer—or user for that matter. There are many different options to choose from, each with their own strengths and weaknesses, some of which can be combined in interesting ways. It is also very clear that there are many more axes to the database graph than just the overly simplified SQL vs. NoSQL axis that seems to dominate coverage of these up-and-coming database systems.
Apple's patent attackSoftware patents have long been the source of a great deal of concern in the free software community; patents are by far the biggest restraint on our ability to program our own computers. Those who worry about these things have expected that attacks might come from patent trolls, or from software companies with fading prospects. Apple's lawsuit against HTC shows that the real threat may come from a different direction.HTC is not normally thought of as a Linux company; it is a Taiwanese manufacturer which provides cellular phone handsets to a number of other companies. HTC has only recently begun promoting phones under its own name; as it happens, a number of those run Android. Since Android increasingly looks like the base for some of the strongest competition against Apple's products, this suit certainly has the look of an attack against Android and not just an action against one hardware manufacturer. Indeed, Android is named specifically in both components to the attack.
There are some 20 patents named in Apple's actions. Ten of them are named in the patent infringement suit filed in Delaware:
Additionally, Apple has filed with the US International Trade Commission with the purpose of blocking the import of HTC's products into the US. That filing names a different, generally older, and more fundamental set of patents:
A few of the patents are hardware-related and don't have much to do with Linux. Many of the rest, however, purport to cover fundamental programming techniques. It would appear that Apple wants to take Android out of the picture - or at least extract substantial rents for its continued existence. But many of these patents, if upheld, could have an influence far beyond Android. Needless to say, the validity of many of these patents is questionable. Proving a patent invalid is a lengthy, expensive, and highly risky process, though; it's not something that one can automatically expect a litigation defendant to jump into. So there is no saying how HTC will react, or what sort of assistance HTC will get from the rest of the industry. In summary: this may be the software patent battle that many of us have feared for a long time. An outright victory by Apple could well leave it "owning" much of the computing and mobile telephony industry - in the US, at least. One assumes that the rest of the industry is going to take note of what is happening here. Nokia is already involved in its own patent disputes with Apple, but this battle could spread well beyond Nokia and HTC. It will be in few companies' interest to let Apple prevail on these claims and entrench their validity. This battle is going to be an interesting one to watch.
SCALE 8x: Color management for everyoneOn Sunday at SCALE 8x, Inkscape developer Jon Cruz presented a talk entitled "Why Color Management matters to Open Source and to You," putting the need for color management into real-world terms for the average Linux user, outlining current development work on the subject at the application and toolkit levels, and giving example color-managed workflows for print and web production. Color management is sometimes unfairly characterized as a topic of interest only to print shops and video editors, but as Cruz explained at the top of his talk, anyone who shares digital content wants it to look correct, and everyone who uses more than one device knows how tricky that can be. "If you have eyes and a display, you need color management"Color management, broadly speaking, is the automatic transformation of image colors so as to provide a uniformly accurate representation across devices. This includes output-only devices such as televisions and printers, as well as CRT and LCD displays on which editing as well as final output is viewed. The first problem is that every device is capable of generating a different spectrum of colors — different hues, different ranges of white-to-black values, and different degrees of saturation. Collectively, the color capabilities of the device are its gamut, which can be represented by a three-dimensional volume in one of several mathematical color models (or "color spaces"). ![[Cruz juggling]](/images/2010/scale-cruz1-sm.jpg)
The second problem is that digital files store the color of each pixel as a numeric triple that may or may not represent coordinates in some specified color space. If the color space to which the file referenced is known, mapping each triple from its stored value into the gamut of the output device is a simple transformation, and the user can visually examine the full range of pixel data. Without that transformation, multiple colors outside the display device's gamut get mapped to the boundaries, causing artifacts and loss of detail, and the entire image can get mapped too dark or too light, misrepresenting the scene. Although it is clear that graphics professionals need color managed displays and printers, Cruz said, the explosion of user-generated digital content in recent years makes it a problem for everyone. Home users want to be able to edit video and share it online, knowing that what appears appropriately bright on-screen will not look washed-out or too dark on DVD or YouTube. They also want to drop off family photos at the corner drugstore kiosk and not be disappointed by a red or green cast to the skin-tones. Photo kiosks may be inexpensive per-print, he said, but online vendors like Apple and Google's Picasa are increasingly offering more elaborate services, such as hardbound books, with correspondingly higher prices. Consumers might shrug off paying a few cents for a bad-looking 4x6 print, but getting burned on an expensive book is considerably more aggravating. Just as importantly, Cruz added, business users need to care about the professionalism of their presentations, both for aesthetic reasons, and because a mis-colored partner logo could accidentally sour the opinion of the executive at the table who recently spent months determining the "perfect shade of puce" to represent the company image. Finally, he said, anyone who sells products online should know that the number one reason for returned consumer purchases is mismatched colors — if the product shots on the web site make the red shirts look orange, the seller is financially at risk for the cost of returns. In addition to these use cases, Cruz explained that users need color management support in their desktop applications to cope with the variety of different display devices they use over the course of a day. Multiple computers are commonplace, from desktops to laptops to netbooks to hand-held devices, and each have different display characteristics. Laptop screens have noticeably smaller gamuts than desktop LCDs, which are in turn smaller than CRTs, and different also from the displays of consumer HDTVs. Mobile devices, based on different graphics hardware, may not even support full 8-bit-per-channel color. Presenting a consistent display across these platforms cannot be left to chance. Status reportFortunately for Linux users, Cruz continued, color management support in Linux is in good shape, although more still needs to be done. Most creative graphics applications support color management already, thanks in large part to the collaborative efforts of the Create project at Freedesktop.org. These include Gimp, Krita, Inkscape, Scribus, Digikam, F-Spot, and Rawstudio, as well as several image viewing utilities. Enabling users to acquire good ICC profiles (tables measuring the device's attributes against points in a known color space, thus allowing for interpolation of color data) or to build their own is one of the key areas of current color work. Projects like Argyll and Oyranos handle tasks such as precisely measuring monitor color output through hardware colorimeters, creating profiles for printers, scanners, and cameras through color targets, and linking profiles for advanced usage. A simpler solution aimed at the home user is GNOME Color Manager (GCM); unlike the previous two examples GCM does not attempt to be a complete ICC profile management tool, but focuses on easily enabling users to correctly assign a profile to their monitor. Default profiles are usually available from the manufacturer, either through the web or on the "driver" CDs in the box, and for normal usage they are an excellent first step. Developers from these and several related projects collaborate on common goals in the OpenICC project. Developers interested in adding color management to their applications should start with LittleCMS, Cruz advised, noting that he personally added Inkscape's color management support in less than one week's time with LittleCMS. LittleCMS is a library that handles the mathematical transformations between color spaces automatically, quickly, and with very little overhead. ![[Jon Cruz]](/images/2010/scale-cruz2-sm.jpg)
Currently, however, one drawback of the Linux color management scene is that most color-aware applications work in isolation from one another, requiring the user to choose display, output, and working ICC profiles in each program — whether through LittleCMS or with in-house routines. Ongoing work to bring color management to a wider range of programs includes adding support to the Cairo vector graphics rendering library, attaching display profiles to X displays, and building color management into GTK+ itself. The latter, in particular, would enable "dumb" applications to automatically be rendered in color-corrected form on the monitor, while still allowing "smart" applications to manage their own color. This is important because graphics and video editing applications need to be able to switch between different profiles for tasks like soft-proofing (simulating a printer's output on-screen by rendering with a different ICC profile) or testing for out-of-gamut color. To the work!Finally, Cruz showed several example workflows for print and web graphics, first illustrating potential problem points when working in a non-color-managed environment, then explaining how using a color-aware setup would trap and eliminate the problem. For web graphics, the example scenario was a simple photo color-correction. Over-correcting the color balance on an improperly-managed monitor easily leads to site visitors seeing a wildly distorted image. In addition, Windows and Macs use different system gamuts, which leads to photos looking either too bright on Macs or too dark on Windows. With a managed workflow, users should target the sRGB color space, previewing the results with Windows, Mac OS X 10.4 and Mac OS X 10.5 profiles (due to changes introduced by Apple in 10.5), as well as mobile devices under different conditions. Because most web site audiences do not have color-corrected displays, he said, not everything is under the designer's control — but if the end user's monitor is broken and the artwork is broken, the problems multiply. For print graphics, the workflow is more complicated, starting with the fact that — despite the popularity of the term — there is no single, standard "CMYK" color space. All process-color spaces are device-dependent, including common four-ink CMYK printers, CcMmYK photo printers, Hexachrome, and others; there is not even an analogous color space to the "Web safe" sRGB standard. Process color's small gamut makes it very easy to produce poor output when not using color management to edit and proof. Fortunately, Inkscape and other SVG-capable editing tools can take advantage of the fact that SVG allows different color profiles to be attached to different objects in a drawing. A CMYK profile for the target printer can be used for most of the drawing, with a separate spot-color profile attached to specific objects that need careful attention, and corrective profiles for embedded RGB elements like raster graphics. A test run is always the best idea, Cruz said, but having proofing profiles available on the system saves both money and time. ConclusionColor management on Linux has come a long way in the last four years. The application support in the basic graphics suite is good, and for professionals tools like Argyll and Oyranos open the door to complete solutions; as Cruz observed in his talk, the colorimeter hardware that used to cost thousands of dollars and lack support on free operating systems is now cheap and well-supported. Still, the average desktop Linux distribution does not install in a color-managed state, which is unfortunate. Proper support for transforming pixels from one color space to another is straightforward math that, much like window translucency, smooth widget animation, and audio mixing, should happen without requiring the user to stop and think about it. It is promising that headway is being made on that front as well, with GCM and GTK+; perhaps in a few release cycles Linux will have full color management out-of-the-box.
Security Fedora 13 to debut a security "spin"Fedora already has a number of variations—called "spins"—to support different use cases: alternative desktops (KDE, LXDE, XFCE), gaming, hardware design, education, etc. Starting with Fedora 13, those will be joined by the Fedora Security Lab (FSL), which is meant to be a "safe test-environment for working on security-auditing, forensics and penetration-testing, coupled with all the Fedora-Security features and tools". The target audience is much the same as that of the BackTrack security distribution—security professionals along with those who want to learn about various security techniques. FSL is based on the LXDE desktop environment because of its small resource footprint, which will leave more memory available for running various security and forensic tools. The LXDE menu has been customized to present a categorized list of tools and applications available to a user. The distribution comes with a fairly extensive list of packages, as well as a wish list of additional packages that would be added to FSL once they are packaged for Fedora. The release itself will be an ISO image that can be used as a Live CD, which can then be installed on the hard disk. A more likely scenario is creating a bootable system on a USB stick using Fedora's liveusb-creator. That will allow the user to reserve some extra space on the USB stick for persistent storage. That storage can be used for installing additional packages or storing the output or configuration of various utilities so that they are available after each boot. Fedora's Joerg Simon is leading the FSL effort, which got final approval from the Fedora advisory board in mid-February. FSL provides a number of advantages for Fedora and its users—many of which are listed on the FSL page—but there is one item in particular that Simon seems to be excited about: using it as a platform to teach about security. Simon has slides [PDF] from a presentation he gave that proposed FSL as the basis for teaching classes based on the Open Source Security Testing Methodology Manual (OSSTMM). Simon is involved in both projects and sees benefits to both from a collaboration. FSL would provide a stable platform that teachers and students could rely upon and Fedora would benefit from the wider exposure those classes would bring. In addition to the various utilities and tools that are packaged with the spin, FSL also showcases the security features that are part of all Fedora spins. Things like SELinux, default firewall rules, PolicyKit, and various protections like stack smashing protection, buffer overflow protection, and so forth, are all available for students and others to examine and play with. Having a larger parent organization like Fedora—and to some extent Red Hat—may help FSL achieve a higher-profile than BackTrack or other security distributions have in the past. One can imagine that FSL will be the tool of choice for recovery of broken systems in the Fedora and RHEL worlds, as users will already be familiar with the underlying distribution. Working with other organizations that are targeting security education is another thing that may very well help foster FSL as a tool of choice for security professionals. While FSL is somewhat late to this particular party, and still has a number of important tools (Metasploit, OpenVAS, SiLK, etc.) on its wish list, it does have the infrastructure and user community of Fedora behind it. There is ample room for collaboration with BackTrack and other security-focused distributions—one hopes that can come about. By sharing information, configuration, tools, and techniques, in much the same way that free software development is done, better security distributions will result. That can only help bring about increased security for all free software.
Brief items Microsoft's other takedown actionThis ars technica article describes how Microsoft took down the control structure for the Waledac botnet. "By obtaining the restraining order, this command-and-control system was disrupted; with the domain names offline, the machines in the botnet were no longer able to locate their control servers, rendering them mostly harmless. The court action had to be taken in secret to avoid warning the botnet's operators; with sufficient warning, they might have been able to set up new domain names and new control systems, thereby circumventing Microsoft's efforts. The names have now been offline for three days, presumably sufficient to cause permanent disruption, and the injunction is now public."
New vulnerabilities apache: unknown vulnerability
apache-mod_security: denial of service
kernel: denial of service
kvm: privilege escalation
puppet: insecure tempfile creation
samba: denial of service
sudo: unintended privilege escalation
Page editor: Jake Edge Kernel development Brief items Kernel release statusThe 2.6.34 merge window is open so there is no development kernel release to mention at this time. See the separate article, below, for a summary of what has been merged for 2.6.34 so far.There have been no stable updates released over the last week, and none are currently in the review process.
Quotes of the week
So guys: feel free to rebase. But when you do, wait a week
afterwards. Don't "rebase and ask Linus to pull". That's just
_wrong_. It means that the tree you asked me to pull has gotten
zero testing.
-- Linus Torvalds
yikes, that macro should be killed with a stick before it becomes
self-aware and starts breeding.
-- Andrew Morton tries to save us all
Linux-2.6.33-libre releasedThe Free Software Foundation Latin America has sent out an announcement for its 2.6.33-libre kernel distribution. "Linux hasn't been Free Software since 1996, when Mr Torvalds accepted the first pieces of non-Free Software in the distributions of Linux he has published since 1991. Over these years, while this kernel grew by a factor of 14, the amount of non-Free firmware required by Linux drivers grew by an alarming factor of 83. We, Free Software users, need to join forces to reverse this trend, and part of the solution is Linux-libre, whose release 2.6.33-libre was recently published by FSFLA, bringing with it freedom, major improvements and plans for the future." Many words are expended on their motivations and methods, but they don't get around to saying where to get the package; interested users should look over here.
New features and configuration defaultsEvery merge window seems to exhibit a theme or two, usually along the lines of "how not to try to merge code." This time around, it seems to be configuration options; a few new features have shown up with their associated configuration options set to "yes" by default. That goes against established practice and tends to make Linus grumpy. He put it this way:
But if it's not an old feature that used to not have a config
option at all, and it doesn't cure cancer, you never EVER do
"default y". Because when I do "make oldconfig", and I see a "Y"
default, it makes me go "I'm not pulling that piece of sh*t".
The message seems clear: new features aimed at the mainline should not be configured in by default.
Divorcing namespaces from processesFor the last few years, the development community interested in implementing containers has been working to add a variety of namespaces to the kernel. Each namespace wraps around a specific global kernel resource (such as the network environment, the list of running processes, or the filesystem tree), allowing different containers to have different views of that resource. Namespaces are tightly tied to process trees; they are created with new processes through the use of special flags to the clone() system call. Once created, a namespace is only visible to the newly-created process and any children thereof, and it only lives as long as those processes do. That works for many situations, but there are others where it would be nice to have longer-lived namespaces which are more readily accessible.To that end, Eric Biederman has proposed the creation of a pair of new system calls. The first is the rather tersely named nsfd():
int nsfd(pid_t pid, unsigned long nstype);
This system call will find the namespace of the given nstype which is in effect for the process identified by pid; the return value will be a file descriptor which identifies - and holds a reference to - that namespace. The calling process must be able to use ptrace() on pid for the call to succeed; in the current patch, only network namespaces are supported. Simply holding the file descriptor open will cause the target namespace to continue to exist, even if all processes within it exit. The namespace can be made more visible by creating a bind mount on top of it with a command like:
mount --bind /proc/self/fd/N /somewhere
The other piece of the puzzle is setns():
int setns(unsigned long nstype, int fd);
This system call will make the namespace indicated by fd into the current namespace for the calling process. This solves the problem of being able to enter another container's namespace without the somewhat strange semantics of the once-proposed hijack() system call. These new system calls are in an early, proof-of-concept stage, so they are likely to evolve considerably between now and the targeted 2.6.35 merge.
Fishy businessMany pixels have been expended about the presence of the Android code in the mainline kernel, or, more precisely, the lack thereof. There are many reasons for Android's absence, including the Android team's prioritization of upcoming handset releases over upstreaming the code and some strong technical disagreements over some of the Android code. For a while, it seemed that there might be yet another obstacle: source files named after fish.Like most products, Android-based handsets go through a series of code names before they end up in the stores. Daniel Walker cited an example: an HTC handset which was named "Passion" by the manufacturer. When it got to Google for the Android work, they concluded that "Mahimahi" would be a good name for it. It's only when this device got to the final stages that it gained the "Nexus One" name. Apple's "dirty infringer" label came even later than that. Daniel asked: which name should be used when bringing this code into the mainline kernel? The Google developers who wrote the code used the "mahimahi" name, so the source tree is full of files with names like board-mahimahi-audio.c; they sit alongside files named after trout, halibut, and swordfish. Daniel feels these names might be confusing; for this reason, board-trout.c became board-dream.c when it moved into the mainline. After all, very few G1/ADP1 users think that they are carrying trout in their pockets. The problem, of course, is that this kind of renaming only makes life harder for people who are trying to move code between the mainline and Google's trees. Given the amount of impedance which already exists on this path, it doesn't seem like making things harder is called for. ARM maintainer Russell King came to that conclusion, decreeing:
There's still precious little to show in terms of progress on
moving this code towards the mainline tree - let's not put
additional barriers in the way.
Let's keep the current naming and arrange for informative comments in files about the other names, and use the common name in the Kconfig - that way it's obvious from the kernel configuration point of view what is needed to be selected for a given platform, and it avoids the problem of having effectively two code bases. That would appear to close the discussion; the board-level Android code can keep its fishy names. Of course, that doesn't help if the code doesn't head toward the mainline anyway. The good news is that people have not given up, and work is being done to help make that happen. With luck, installing a mainline kernel on a swordfish will eventually be a straightforward task for anybody.
Kernel development news 2.6.34 Merge window, part 1As of this writing, the 2.6.34 merge window is open, with 4480 non-merge changeset accepted so far. As usual, your long-suffering (i.e. slow learning) editor has read through all of them in order to produce this summary of the most interesting changes. Starting with user-visible changes:
Changes visible to kernel developers include:
The merge window is normally open for two weeks, but Linus has suggested that it might be a little shorter this time around. So, by the time next week's edition comes out, chances are that the window will be closed and the feature set for 2.6.34 will be complete. Tune in then for a summary of the second half of this merge window.
Huge pages part 3: Administration[Editor's note: this is the third part in Mel Gorman's series on the use of huge pages in Linux. For those who missed them, a look at part 1 and part 2 is recommended before diving into this installment.]In this chapter, the setup and the administration of huge pages within the system is addressed. Part 2 discussed the different interfaces between user and kernel space such as hugetlbfs and shared memory. For an application to use these interfaces, though, the system must first be properly configured. Use of hugetlbfs requires only that the filesystem must be mounted; shared memory needs additional tuning and huge pages must also be allocated. Huge pages can be statically allocated as part of a pool early in the lifetime of the system or the pool can be allowed to grow dynamically as required. Libhugetlbfs provides a hugeadm utility that removes much of the tedium involved in these tasks.
1 Identifying Available Page SizesSince kernel 2.6.27, Linux has supported more than one huge page size if the underlying hardware does. There will be one directory per page size supported in /sys/kernel/mm/hugepages and the "default" huge page size will be stored in the Hugepagesize field in /proc/meminfo. The default huge page size can be important. While hugetlbfs can specify the page size at mount time, the same option is not available for shared memory or MAP_HUGETLB. This can be important when using 1G pages on AMD or 16G pages on Power 5+ and later. The default huge page size can be set either with the last hugepagesz= option on the kernel command line (see below) or explicitly with default_hugepagesz=. Libhugetlbfs provides two means of identifying the huge page sizes. The first is using the pagesize utility with the -H switch printing the available huge page sizes and -a showing all page sizes. The programming equivalent are the gethugepagesizes() and getpagesizes() calls.
2 Huge Page PoolDue to the inability to swap huge pages, none are allocated by default, so a pool must be configured with either a static or a dynamic size. The static size is the number of huge pages that are pre-allocated and guaranteed to be available for use by applications. Where it is known in advance how many huge pages are required, the static size should be set. The size of the static pool may be set in a number of ways. First, it may be set at boot-time using the hugepages= kernel boot parameter. If there are multiple huge page sizes, the hugepagesz= parameter must be used and interleaved with hugepages= as described in Documentation/kernel-parameters. For example, Power 5+ and later support multiple page sizes including 64K and 16M; both could be configured with:
hugepagesz=64k hugepages=128 hugepagesz=16M hugepages=4
Second, the default huge page pool size can be set with the vm.nr_hugepages sysctl, which, again, tunes the default huge page pool. Third, it may be set via sysfs by finding the appropriate nr_hugepages virtual file below /sys/kernel/mm/hugepages. Knowing the exact huge page requirements in advance may not be possible. For example, the huge page requirements may be expected to vary throughout the lifetime of the system. In this case, the maximum number of additional huge pages that should be allocated is specified with the vm.nr_overcommit_hugepages. When a huge page pool does not have sufficient pages to satisfy a request for huge pages, an attempt to allocate up to nr_overcommit_hugepages is made. If an allocation fails, the result will be that mmap() will fail to avoid page fault failures as described in Huge Page Fault Behaviour in part 1. It is easiest to tune the pools with hugeadm. The --pool-pages-min argument specifies the minimum number of huge pages that are guaranteed to be available. The --pool-pages-max argument specifies the maximum number of huge pages that will exist in the system, whether statically or dynamically allocated. The page size can be specified or it can be simply DEFAULT. The amount to allocate can be specified as either a number of huge pages or a size requirement. In the following example, run on an x86 machine, the 4M huge page pool is being tuned. As 4M also happens to be the default huge page size, it could also have been specified as DEFAULT:32M and DEFAULT:64M respectively.
$ hugeadm --pool-pages-min 4M:32M
$ hugeadm --pool-pages-max 4M:64M
$ hugeadm --pool-list
Size Minimum Current Maximum Default
4194304 8 8 16 *
To confirm the huge page pools are tuned to the satisfaction of requirements, hugeadm --pool-list will report on the minimum, maximum and current usage of huge pages of each size supported by the system.
3 Mounting HugeTLBFSTo access the special filesystem described in HugeTLBFS in part 2, it must first be mounted. What may be less obvious is that this is required to benefit from the use of the allocation API, or to automatically back segments with huge pages (as also described in part 2). The default huge page size is used for the mount if the pagesize= is not used. The following mounts two filesystem instances with different page sizes as supported on Power 5+.
$ mount -t hugetlbfs /mnt/hugetlbfs-default $ mount -t hugetlbfs /mnt/hugetlbfs-64k -o pagesize=64K Ordinarily it would be the responsibility of the administrator to set the permissions on this filesystem appropriately. hugeadm provides a range of different options for creating mount points with different permissions. The list of options are as follows and are self-explanatory.
It is up to the discretion of the administrator whether to call hugeadm from a system initialization script or to create appropriate fstab entries. If it is unclear what mount points already exist, use --list-all-mounts to list all current hugetlbfs mounts and the options used.
3.1 QuotasA little-used feature of hugetlbfs is quota support which limits the number of huge pages that a filesystem instance can use even if more huge pages are available in the system. The expected use case would be to limit the number of huge pages available to a user or group. While it is not currently supported by hugeadm, the quota can be set with the size= option at mount time.
4 Enabling Shared Memory UseThere are two tunables that are relevant to the use of huge pages with shared memory. The first is the sysctl kernel.shmmax kernel parameter configured permanently in /etc/sysctl.conf or temporarily in /proc/sys/kernel/shmmax. The second is the sysctl vm.hugetlb_shm_group which stores which group ID (GID) is allowed to create shared memory segments. For example, lets say a JVM was to use shared memory with huge pages and ran as the user JVM with UID 1500 and GID 3000, then the value of this tunable should be 3000. Again, hugeadm is able to tune both of these parameters with the switches --set-recommended-shmmax and --set-shm-group. As the recommended value is calculated based on the size of the static and dynamic huge page pools, this should be called after the pools have been configured.
5 Huge Page Allocation Success RatesIf the huge page pool is statically allocated at boot-time, then this section will not be relevant as the huge pages are guaranteed to exist. In the event the system needs to dynamically allocate huge pages throughout its lifetime, then external fragmentation may be a problem. "External fragmentation" in this context refers to the inability of the system to allocate a huge page even if enough memory is free overall because the free memory is not physically contiguous. There are two means by which external fragmentation can be controlled, greatly increasing the success rate of huge page allocations. The first means is by tuning vm.min_free_kbytes to a higher value which helps the kernel's fragmentation-avoidance mechanism. The exact value depends on the type of system, the number of NUMA nodes and the huge page size, but hugeadm can calculate and set it with the --set-recommended-min_free_kbytes switch. If necessary, the effectiveness of this step can be measured by using the trace_mm_page_alloc_extfrag tracepoint and ftrace although how to do it is beyond the scope of this article. While the static huge page pool is guaranteed to be available as it has already been allocated, tuning min_free_kbytes improves the success rate when dynamically growing the huge page pool beyond its minimum size. The static pool sets the lower bound but there is no guaranteed upper bound on the number of huge pages that are available. For example, an administrator might request a minimum pool of 1G and a maximum pool 8G but fragmentation may mean that the real upper bound is 4G. If a guaranteed upper bound is required, a memory partition can be created using either the kernelcore= or movablecore= switch at boot time. These switches create a “Movable” zone that can be seen in /proc/zoneinfo or /proc/buddyinfo. Only pages that the kernel can migrate or reclaim exist in this zone. By default, huge pages are not allocated from this zone but it can be enabled by setting either vm.hugepages_treat_as_movable or using the hugeadm --enable-zone-movable switch.
6 SummaryIn this chapter, four sets of system tunables were described. These relate to the allocation of huge pages, use of hugetlbfs filesystems, the use of shared memory, and simplifying the allocation of huge pages when dynamic pool sizing is in use. Once the administrator has made a choice, it should be implemented as part of a system initialization script. In the next chapter, it will be shown how some common benchmarks can be easily converted to use huge pages.
SCALE 8x: Ubuntu kernel development processCanonical's kernel manager, Pete Graner, spoke at UbuCon—held just prior to SCALE 8x—on the "Ubuntu Kernel Development Process". In the talk, he looked at how Ubuntu decides what goes into the kernel and how that kernel gets built and tested. It provided an interesting look inside the process that results in a new kernel getting released for each new Ubuntu version, which comes along every six months. Graner manages a "pretty big" group at Canonical, of 25 people split into two sub-groups, one focused on the kernel itself and the other on drivers. For each release, the kernel team chooses a "kernel release lead" (KRL) who is responsible for ensuring that the kernel is ready for the release and its users. The KRL rotates among team members with Andy Whitcroft handling Lucid Lynx (10.04) and Leann Ogasawara slated as KRL for the following ("M" or 10.10) release. The six-month development cycle is "very challenging", Graner said. The team needs to be very careful about which drivers—in-tree, out-of-tree, and staging—are enabled. The team regularly takes some drivers from the staging tree, and fixes them up a bit, before enabling them in the Ubuntu tree so that users "get better hardware coverage". Once the kernel for a release has been frozen, a new branch is created for the next release. For example, the Lucid kernel will be frozen in a few weeks, at which point a branch will be made for the 10.10 release. That branch will get the latest "bleeding edge" kernel from Linus Torvalds's tree (presumably 2.6.34-rc1), and the team will start putting the additional patches onto that branch. The patches that are rolled into the tree include things from linux-next (e.g. suspend/resume fixes), any patches that Debian has added to its kernel, then the Ubuntu-specific patchset. Any of those that have been merged into the mainline can be dropped from the list, but it is a "very time-consuming effort" to go through the git tree to figure all of that out. With each new tag from Torvalds's tree, they do a git rebase on their tree—as it is not a shared development tree—"see what conflicts, and deal with those". The focus and direction for the Ubuntu kernel, like all Ubuntu features, comes out of the Ubuntu Developer Summit (UDS), which is held shortly after each release to set goals and make plans for the following release. Before UDS, the kernel team selects some broad topics and creates blueprints on the wiki to describe those topics. In the past, they have focused on things like suspend/resume, WiFi networking, and audio; "a big one going forward is power management", he said. The specifications for these features are "broad-brush high-level" descriptions (e.g. John has a laptop and wants to get 10 hours of battery life). The descriptions are fleshed out into various use cases, which results in a plan of action. All of the discussion, decisions, plans, and so on are captured on the UDS wiki One of the longer kernel sessions at UDS looks at each kernel configuration option (i.e. the kernel .config file) to determine which should be enabled. New options are looked at closely to decide whether that feature is needed, but the existing choices are scrutinized as well. In addition, Graner said that the team looks at the patches and drivers that were added to the last kernel to see which of those should be continued in the next release. He pointed to Aufs as a problematic feature because it always breaks with each new kernel release and can take up to three weeks to get it working. They have talked about dropping it, because Torvalds won't merge it into the mainline, but the live CDs need it. The kernel team has to balance the Ubuntu community needs as well as Canonical's business needs, for things like Ubuntu One for example, and come up with a set of kernel features that will satisfy both. The discussions about what will get in at UDS can get intense at times, Graner said, "Lucid was pretty tame, but Karmic was kind of heated". Lucid will ship with the 2.6.32 kernel which makes sense for a long-term support (LTS) release. 2.6.32 will be supported as a stable tree release for the next several years and will be shipped with the next RHEL and SLES. That means it will get better testing coverage which will lead to a "very stable kernel for Lucid". Each stable tree update will be pulled into the Ubuntu kernel tree, but LTS updates to the kernel will only be pushed out quarterly unless there is a "high" or "medium" security fix. For new kernel feature development, new mainline kernel releases and release candidates are pulled in by the team as well. Graner gave two examples of new development that is going on in the Ubuntu kernel trees: adding devicetree support for the ARM architecture, which will reduce the complexity of supporting multiple ARM kernels, and the AppArmor security module that is being targeted for the 2.6.34 kernel. Once the kernel version has been frozen for a release, the management of that kernel is much more strictly controlled. The only patches that get applied are those that have a bug associated with them. Stable kernel patches are "cherry-picked" based on user or security problems. There is a full-time kernel bug triager that tries to determine if a bug reporter has included enough information to have any hope of finding the problem—otherwise it gets dropped. One way to ensure a bug gets fixed, though, is to "show the upstream patch that fixes the problem"; if that happens, it will get pulled into the kernel, Graner said. There are general freezes for each alpha, beta, and the final release, but the kernel must already be in the archive by the time of those freezes. Each time the kernel itself freezes, it "takes almost a full week to build all of the architectures" that are supported by Ubuntu. There are more architectures supported by Ubuntu than any other distribution "that I am aware of", he said. Each build is done in a virtualized environment with a specific toolchain that can be recreated whenever an update needs to be built. All of that means the kernel needs to freeze well in advance of the general release freeze, typically about a month before. Once the kernel is ready, it is tested in Montreal in a lab with 500 or 600 machines. The QA team runs the kernels against all that hardware, which is also a time-consuming process. Previously, the kernels would be tossed over the wall for users to test, but "now Canonical is trying to do better" by dedicating more resources to testing and QA. Managing kernel releases for a distribution is big task, and the details of that process are not generally very well-known. Graner's talk helped to change that, which should allow others to become more involved in the process. Understanding how it all works will help those outside of the team do a better job of working with the team, which should result in better kernels for Ubuntu users.
Patches and updates Kernel trees
Core kernel code
Development tools
Device drivers
Documentation
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Security-related
Virtualization and containers
Miscellaneous
Page editor: Jonathan Corbet Distributions News and Editorials The Ubuntu One music store and free software for profitOne of the features expected with the upcoming Ubuntu 10.04 release is the Ubuntu One Music Store (UOMS). The UOMS is a mechanism by which Ubuntu users can purchase songs in the MP3 format, with some of the revenue going to support Canonical. These songs are evidently compressed at a relatively high bit rate and lack any sort of DRM or watermarks. Support for the UOMS has been integrated into the Rhythmbox music player, with support for other players expected in the future. Discussion of this new feature has been relatively subdued thus far, but developers elsewhere are beginning to take notice and ask some questions about the extent to which the UOMS should be supported.Recently, Amarok hacker Jeff Mitchell went to the openSUSE community to ask them how they felt about the UOMS. In particular, he would like to know how openSUSE might react if Canonical were to push its Rhythmbox changes back upstream - which has not yet happened, as of this writing. Would openSUSE be willing to ship a Rhythmbox plugin which existed for the purpose of funding another distributor? How, asked Jeff, do we feel about free software which is designed to make money for others? To an extent, this question has been answered for years: both Rhythmbox and Amarok include support for Magnatune's music store, and distributors have shipped that support. This plugin generates income - a significant amount, evidently - for Magnatune, which kicks a portion back to Rhythmbox and Amarok. So simply operating a for-profit music store is not, itself, reason for concern or for exclusion from free music player applications. The Ubuntu music store appears to be looked at differently, though, for a couple of reasons, one of which may hold more water than the other. Jeff described the rules which music stores like Magnatune must meet for inclusion in Amarok:
So far our policy for music stores has been pretty strict: they
must allow full-length previews, they must allow tracks that have
been purchased to be redownloaded at any time, and they must allow
tracks to be purchased in a free format (which could be in addition
to a non-free format).
It is not clear what sort of preview capability will be included in the UOMS. It would appear, from Ubuntu's documentation, that tracks can be downloaded up to three times, so redownloads are indeed possible "at any time," but up to a limit. Where things will really fall down, though, is the requirement for free formats; the Ubuntu store looks to be MP3-only (the occasional track in Windows media format is unlikely to make anybody feel any better). So the simple act of playing tracks from the UOMS on an Ubuntu system will require the installation of codecs which have potential patent problems or which are not free software. That requirement is not, needless to say, encouraging the wider use of free audio formats. Perhaps this is a place where Canonical could have tried to push things in the right direction by insisting on the right to sell tracks in free (and preferably lossless) formats. Perhaps Canonical did try and failed; if so, that's not something which has been communicated to the rest of the world. The other complaint, again as expressed by Jeff, is this:
Canonical however is a for-profit company. Other distributions
shipping this plugin means that you're helping Canonical make their
money for them, and I haven't heard of any method of Canonical
sharing profit with other distributions.
In other words, does it make sense for one distribution to ship code which exists for the purpose of earning money for somebody else? Again, the precedent is fairly clear: the Firefox browser has been an reliable money-making tool for the Mozilla project, and Mozilla Corporation is a for-profit entity (though the Mozilla Foundation is not). Many drivers contributed to the kernel are put there by for-profit corporations which clearly hope to see that code spur sales of their products. Gstreamer has an array of commercial offerings designed to plug into it. And so on. Free software may be free-as-in-beer, but the profit motive is often not that far away. It is tempting to say that the real complaint here is that, if this support were to be shipped outside of Ubuntu, the beneficiary would be Canonical in particular. The truth of the matter, though, is that a music store designed to benefit any other distribution-owning corporation would likely raise eyebrows as well. But it is not clear that this is right; there is nothing inherently wrong with generating money for companies which are making free software. Free software licenses are not allowed to discriminate between different fields of use. Freedom means that users can use the code to do something its developers might find unpleasant - or worse. That does not mean, though, that distributors have to ship software aimed at any purpose. In the past, programs like hot babe and gnaughty have run into opposition at distributors. So, if distributors were to decide that selling MP3 files to users violates their standards of decency, there would be precedents for keeping the code out. On the other hand, explicitly patching out a music player plugin to prevent users from spending money with another distributor might be seen as petty, at best. So far, the situation is hypothetical; Canonical has not yet tried to push this code upstream, and nobody is expecting other distributors to fish this patch out of the Ubuntu source packages. It would not be surprising if this kind if situation were to arise at some point, though; indeed, it would be surprising if it doesn't. So it makes sense to have this discussion now; that way, the people involved may have some idea of what they want to do when a real decision must be made.
New Releases Community Fedora Remix 12.3Valent Turkovic has announced the release of Community Fedora Remix 12.3, available on live DVD/USB.
LFS-6.6 is releasedLinux From Scratch 6.6 has been released. "This release includes numerous changes to LFS-6.5 (including updates to Linux-2.6.32.8, GCC-4.4.3, Glibc-2.11.1) and security fixes. It also includes editorial work on the explanatory material throughout the book, improving both the clarity and accuracy of the text."
SystemRescueCd 1.4 released (The H)The H takes a look at the 1.4.0 release of SystemRescueCD. "The latest release uses the 2.6.32.9 Linux kernel and features the new options to boot from NFS or NBD, which lets users boot SystemRescueCd from a network if, for example, a computer doesn't have a CD drive. The developers note that although previous versions of the SystemRescueCd could also boot from a network, version 1.4.0 mounts the root file system through the network instead of copying the whole root file system image to the local system's memory. This allows computers with only 256 MB of memory to boot the 400 MB+ image from the network."
Lucid Alpha 3 releasedThe alpha 3 release of Ubuntu's Lucid Lynx has been released. "Pre-releases of Lucid are *not* encouraged for anyone needing a stable system or anyone who is not comfortable running into occasional, even frequent breakage. They are, however, recommended for Ubuntu developers and those who want to help in testing, reporting, and fixing bugs. Alpha 3 is the third in a series of milestone CD images that will be released throughout the Lucid development cycle. The Alpha images are known to be reasonably free of showstopper CD build or installer bugs, while representing a very recent snapshot of Lucid."
Fixstars Releases Yellow Dog Enterprise Linux for CUDAFixstars has released Yellow Dog Enterprise Linux for CUDA, "the first enterprise Linux OS optimized for GPU computing. YDEL for CUDA offers end users, developers and integrators a faster, more reliable, and less complex GPU computing experience."
Distribution News Fedora Fedora 13 Alpha slip by one weekFedora slipped the release of F13 Alpha by one week. Further milestone dates are expected to remain the same however.
Fedora Board Meeting Recap 2010-02-25Click below for a recap of the February 25, 2010 meeting of the Fedora Advisory Board. Topics include using Fedora Talk for board meetings, Improved metrics, Strategic Working Group outputs, and board member removal policy.
Board SWG Meeting 2010-03-01 RecapClick below for a recap of the March 1, 2010 meeting of the Advisory Board Strategic Working Group. Topics include spins and the default offering.
Mandriva Linux Mandriva Joins ARM Connected CommunityMandriva has announced that it is a new member in the ARM Connected Community, "The ARM Connected Community is a global network of companies aligned to provide a complete solution, from design to manufacture and end use, for products based on the ARM architecture. ARM offers a variety of resources to Community members, including promotional programs and peer-networking opportunities that enable a variety of ARM Partners to come together to provide end-to-end customer solutions. Visitors to the ARM Connected Community have the ability to contact members directly through the website."
Noteworthy Mandriva Cooker changes 15 February - 28 February 2010Frederik Himpe summarizes some changes in Mandriva Cooker. "The kernel is now updated to 2.6.33 final. As usual, KernelNewbies has a complete overview of the changes in this new kernel. Some noteworthy changes include: the new Nouveau driver for NVidia graphics cards is now included in the kernel and is now used by default on Mandriva instead of the NV driver."
Ubuntu family Minutes from the Ubuntu Technical Board meetingClick below for the minutes from the February 23, 2010 meeting of the Ubuntu Technical Board. Topics include security of package-sets, package set for CLI/Mono packages, and Ubuntu IRC Council Access level.
Other distributions OpenSolaris future assured by Oracle (The H)The H reports that Oracle will continue to support OpenSolaris. "At the OpenSolaris Annual Meeting, held on IRC, Oracle executive Dan Roberts has assured the community about the future of the open source version of Solaris. The statements, available as a log of the meeting, have led Peter Tribble, who had expressed concerns on the lack of communication, to conclude "rumours of its [OpenSolaris] death have been greatly exaggerated"."
North Korea develops Red Star OS based on LinuxNorth Korea has developed its own Linux variant, Red Star OS. "It's hard to substantiate most claims made about North Korea's IT industry, but details of the new operating system were made public by a Russian blogger (http://ashen-rus.livejournal.com/4300.html), who was able to buy a copy of the program off the street."
Distribution Newsletters CentOS Pulse #1001The CentOS Pulse newsletter for March 2, 2010 is out. "In this issue we have a very interesting interview on the usage of CentOS at University College London, a report on FOSDEM 2010 (where nearly all of the main CentOS personnel showed up) and, of course, the usual categories like community, jokes and updates."
DistroWatch Weekly, Issue 343The DistroWatch Weekly for March 1, 2010 is out. "For many users, the combination of Slackware Linux and the Xfce desktop is the perfect blend of stability and speed, whatever the age of their hardware. But if Slackware itself is too much hard work, why not try one of its derivatives with a friendlier approach to the desktop and with out-of-the-box support for popular hardware and multimedia codecs? Bernard Hoffmann, an experienced Slackware user, has taken three Slackware-based Xfce distributions (Zenwalk Linux, Salix OS and GoblinX) for a test drive to see which one would be a best fit for a blazing fast and powerful home desktop. In the news section, Oracle confirms the continued development of OpenSolaris, Fedora delays the upcoming alpha release of version 13, Mandriva switches to nouveau with the latest kernel update in "Cooker", and Linux Mint prepares for an imminent release of its LXDE edition. Also in this issue, a link to a good summary of bleeding-edge repositories for Kubuntu and a brief talk about zombie processes. Finally, we are pleased to announce that the recipient of the February 2010 DistroWatch.com donation is the Squid project. Happy reading!"
Fedora Weekly News 215The Fedora Weekly News for February 28, 2010 is out. "This issue kicks off with an announcement last week of one week slippage for Fedora 13 Alpha, as well as a call for Fedora 13 slogan suggestions, which will be finalized on 3/2. In news from the Fedora Planet, a report from the GNOME London UX Hackfest, a summary of the Fedora 13 Talking Points, and the return of Chromium to Fedora 12. In a new beat, "Fedora in the News", a recent article from LinuxPlanet on recent positive changes to Rawhide, Fedora's development version. In Quality Assurance team news, coverage of the recent Test Day on language pack plugin for yum, details on this week's Test Day, detailed coverage of the QA weekly meetings, and an update on Fedora 13 Alpha validation testing and delay. In Translation team news, fixes to Hivex and kf translations submission issues, announcement of an upcoming release of Transifex v 0.8 rc1, and new members for the Fedora Localization Project for Russian, Spanish, Italian, and Bengali! The Art/Design team brings us news of a couple Fedora 13 website banner designs, work on a LiveCD icon, and a call for help with testing the Fedora 13 Alpha backgrounds. This issue finishes off with a quiet week of security patches for Fedora 11, 12 and 13. Enjoy!"
openSUSE Weekly News/112This issues of the openSUSE Weekly News covers Honoring openSUSE Wiki Reviewing Contributions, * Michal Hrusecky: Public openSUSE 11.3 virtual machine, * Jared Ottley: Alfresco PDF Toolkit, * How to make Monitor refresh 120htz, and * Guillaume DE BURE (gdebure): A call for testers KMyMoney.
Ubuntu Weekly Newsletter #182The Ubuntu Weekly Newsletter for February 27, 2010 is out. "In this issue we cover: Lucid Alpha 3 Released, Rocking The Opportunistic Desktop, Can you hear the Music, New Ubuntu Members: Americas Board Meeting, Ubuntu Libya LoCo at the Technology & Science Fair, Help localization testing with the ISO tracker, Translating software descriptions with Nightmonkey, Attention Encrypted Home Users, Server Bug Zapping - Call for Participation, Ubuntu Women has a new IRC Channel, Full Circle Magazine #34, and much, much more!"
Interviews Matt Asay answers your questions (Slashdot)Slashdot has posted a lengthy interview with Matt Asay regarding his new role at Canonical. "I like to think of our guiding principle as 'make money because of the Ubuntu community, not from it.' At the scale where we operate, all sorts of financial opportunities become possible, opportunities that don't require us to hold back Ubuntu bits to goad people into purchasing. As we roll new services out, I hope you'll let us know how we're doing, and ensure we never sacrifice usability for financial gain." (Thanks to Paul Wise).
Page editor: Rebecca Sobol Development An introduction to the new development pageAs was announced to readers last week, long-time Development Page editor Forrest Cook has moved on to a new set of challenges. As a result, LWN is now faced with a new challenge of its own: maintaining quality content with fewer hands at the keyboard. To respond to this challenge, we are making some changes aimed at making the production of LWN more sustainable while maintaining (or improving) content quality.At recent events, your editor asked many readers what part of the LWN Weekly Edition would be missed least if it went away. The answers were surprisingly consistent; it seems that relatively few people plow through the long lists of software releases which have long appeared on this page. So that's what is going to go; this week inaugurates a new, thinner Development Page. The most important aspects of this page, we hope, will remain. It will still be led by our original content. We will still watch the stream of software release announcements as we did before; the difference is that only a small subset of them will be selected for mention on this page. Announcements will show up here if they are a major release of an important package, or if they highlight an application that we think our readers would be interested in, or if somebody just thinks it's worth posting. The value of LWN, we believe, has always been in selective judgment and conciseness, rather than in scooping up and posting everything. We hope that a more focused Development Page will increase that value. As this page evolves, we will certainly welcome any comments you may have, either posted as comments or sent directly to lwn@lwn.net.
A look at Simple ScanLots of people have complained that XSane is too complicated for many users, but little progress has been made towards creating a user-friendly and stable replacement for the SANE GUI. Until now. Simple Scan is a GTK-based front-end for SANE primarily developed by Robert Ancell and intended to replace XSane. Simple Scan will be landing on desktops in the upcoming Ubuntu Lucid (10.04) release, so now's a good time to take a look at the new kid on the scanning block. ![[Photo scan]](/images/2010/sscan-photos-sm.png)
Packages for Ubuntu are available via Ancell's PPA, the most recent version as of this writing was 0.9.5. Source is available for users on other distributions, and should build on most current distributions. To test Simple Scan, I scanned in several color photos, a handful of old black and white photos, line art, and a printed text document. The test system consisted of a dual Xeon 3.20GHz with 8GB of RAM, running Ubuntu 9.10 and using an Epson Perfection 1260 scanner. The scanner is a bit long in the tooth, and certainly not the fastest available, but has served well over the years and works well with Linux. ![[Prefereneces]](/images/2010/sscan-prefs-sm.png)
Simple Scan lives up to its name. The interface is uncluttered and offers only a few options. If no changes are made, Simple Scan will scan in photos at 300 DPI, or text documents at 150DPI. Photos and text are the only presets available. The DPI can be changed via the Preferences dialog. In fact, that's nearly all that can be changed, along with the scan source if more than one scanner is attached to the system. Once preferences are saved, you can choose to scan in a single page, or all pages if you happen to have a scanner with a document feeder. Unfortunately, the Epson is a flatbed scanner and I wasn't able to test the feeder feature. ![[XSane]](/images/2010/sscan-xsane-sm.png)
Users familiar with other scanning applications will probably be used to doing a preview scan, followed by cropping a section of the document to get a full scan. Simple Scan does a one-shot process and simply scans in the entire area. After this, the user can crop the picture if desired. This is much easier if one wants to scan in something that takes up the entire tray, but can cause a scan to take much longer in practice if you're working at a high DPI and only wish to capture a small portion of it. If you're scanning in, say, several old family photos it makes more sense to just scan an entire tray and do the cropping in The GIMP or another application. Simple Scan's performance leaves a bit to be desired when working at larger resolutions. Scanning a color photo in at 1200DPI nearly brought Simple Scan to its knees. It didn't crash, but the interface became laggy and slow to respond. Resizing the Simple Scan window would take 10 to 20 seconds. Even scanning in some black and white photos at 150DPI caused Simple Scan to become slow to respond. ![[Text scan]](/images/2010/sscan-text-sm.png)
Simple Scan makes it easy to scan in a document and send it as an email. Once a document is scanned in, just select Email from the File menu and Simple Scan will open a new email with the scan as an attachment. At least that's what will happen if you're using Evolution as the default mailer on GNOME. If you're using Thunderbird or another mailer, this doesn't work so well. Simple Scan will initiate a new email, but without the attachment. When selecting email, Simple Scan will always default to PDF. At the moment there appears to be no way to change this. That might be desirable for forms, but not so much for pictures. Editing within Simple Scan is limited to cropping and rotation. When saving scans, users are limited to JPEG, PNG, and PDFs. Simple Scan is really a no-frills tool that just does the most basic scanning operations. Some might wonder why a new application was developed from scratch, rather than improving GNOME Scan. According to the comments on Ancell's blog following the introduction of Simple Scan, GNOME Scan suffered stability issues and did not work well as a stand-alone scanning application. For those unfamiliar with GNOME Scan, the project has been in the works for some time, and is not only meant to be a standalone scanning application, but also is meant to allow other GNOME applications to acquire images from a scanner. All of the features for 1.0 are present in the 0.9.5 release of Simple Scan, and what remains are bugfixes and so on. According to the 0.9.0 announcement Ancell is interested in working on color management, OCR, integration with GNOME Scan and integration with photo management applications like F-Spot after the 1.0 release. Naturally, Simple Scan doesn't hold a candle to XSane's bag of tricks, nor is it meant to. If a user wishes to do color correction, optical character recognition (OCR), scan in slide negatives, or any number of other more complex operations, then XSane is still a better choice. But, if all you need is a fast scan of a form or quick and dirty scan of a color document or photo, then Simple Scan is shaping up to be a good choice.
Brief items Darcs 2.4 releasedVersion 2.4 of the Darcs revision control system is out. "The darcs team is proud to announce the immediate availability of darcs 2.4. darcs 2.4 contains many improvements and bugfixes compared to darcs 2.3.1. Highlights are the faster operation of record, revert and related commands, and the experimental interactive hunk editing." More information can be found in the release announcement.
Mandelin: Starting JägerMonkeyMozilla hacker David Mandelin writes about the JägerMonkey project, which is developing a new just-in-time JavaScript compiler for Firefox. "We decided to import the assembler from Apples open-source Nitro JavaScript JIT. (Thanks, WebKit devs!) We know its simple and fast from looking at it before (I did measurements that showed it was very fast at compiling regular expressions), its open-source, and its well-designed C++, so it was a great fit. Julian Seward modified it to run with our build system and support libraries. Its in our tree with the appropriate licensing, and were already using it to get that 18% speedup I mentioned before."
Mozilla Developer Preview (Gecko 1.9.3a2)Developers of applications using the Gecko renderer might be interested in this alpha release, which showcases the new "out-of-process plugins" mechanism. Running complex plugins in their own address space should result in improvements in both robustness and security.
New SOAP mailing list on python.orgThere is a new mailing list for Python developers wanting to discuss SOAP-related topics. "The goal of the list is to gather all discussions related to SOAP libraries and tools on Python, so that could include soaplib, SOAPpy, suds, IronPython using .NET SOAP libraries, using Java SOAP libraries from Jython, etc."
"Task Pooper" could revolutionize GNOME desktop (ars technica)Ryan Paul covers a week long GNOME hackfest. "The GNOME Task Pooper concept, which is intended to bring first-class task management to the desktop, has a content drop-zone that organizes itself temporally. It will automatically move expired content into an archive so that immediately relevant action items are easily accessible and not obscured by clutter. Beyond the initial 3.0 release, the document says that the Pooper could eventually be augmented so that users can drag entire windows and workspaces into it for later use."
Thunderbird 3.0.2 and 3.0.3The Thunderbird 3.0.2 release fixes the usual set of scary security issues; the developers "strongly recommend" that all users upgrade. The 3.0.3 update, instead, just fixes "an issue with mail folders" introduced in 3.0.2.
Newsletters and articles Caml Weekly NewsThe March 2 edition covers graphic visualization of function dependencies, hivex bindings, passing C pointers, and more.
PostgreSQL Weekly News - February 28 2010This issue mentions the PostgreSQL 9.0alpha4 release, PGCon 2010 registration, the Karoo Project, and a long list of applied patches.
Page editor: Jonathan Corbet Announcements Non-Commercial announcements EFF: Unintended Consequences - Twelve Years Under the DMCAThe Electronic Frontier Foundation "celebrates" twelve years of the DMCA with a report listing of the problems which have resulted from that law, and from its anti-circumvention rules in particular. "EFF's report details the numerous harms stemming from the DMCA's ban on circumventing DRM, including Apple's attempts to lock down the iPhone and force users into its App Store. Also new in this year's report is the account of hobbyists threatened by Texas Instruments for blogging about potential modifications to the company's programmable graphing calculators as well as the story behind the legal attacks on Real DVD and other products that create innovative new ways for consumers to enjoy DVD content they have legitimately purchased."
IIPA aims to put Indonesia on watch list over FOSSThe International Intellectual Property Alliance has issued a special report which puts Indonesia on a priority watch list for, among other things, mandating the use of open-source software in its government. Page 3 of this document [PDF] explains: "While IIPA has no issue with one of the stated goals of the circular, namely, reducing software copyright violation, the Indonesian governments policy as indicated in the circular letter instead simply weakens the software industry and undermines its long-term competitiveness by creating an artificial preference for companies offering open source software and related services, even as it denies many legitimate companies access to the government market. Rather than fostering a system that will allow users to benefit from the best solution available in the market, irrespective of the development model, it encourages a mindset that does not give due consideration to the value to intellectual creations..." (Thanks to Priyadi Iman Nurcahyo).
FOSS law journal is open to receive submissionsThe International Free and Open Source Software Law Review (IFOSS L. Rev. or IFOSSLR) is open for submissions. IFOSSLR is a collaborative legal publication aimed at increasing knowledge and understanding among lawyers about free and open source software. "The topics covered by the publication include copyright, licence implementation, licence interpretation, patents applicable to software and business methods, standards applicable to software, case law, statutory changes, license enforcement, competition law applicable to software, economics analysis, business models and due diligence."
Commercial announcements Novell: Linux finally breaks even (Channel Register)Channel Register takes a look at Novell's financial results. "As part of its discussion of its financial results for the first quarter of fiscal 2010 ended in January, Dana Russell, chief financial officer at operating system and systems software maker Novell, said that the SUSE Linux business was at break-even, what he called "a significant milestone.""
Elliott Associates Offers to Buy NovellThe New York Times reports that hedge fund Elliott Associates has made an offer to buy Novell. "Elliott said it would pay $5.75 a share in cash for Novell, a price that is 21 percent higher than Novell's closing stock price on Tuesday. Wall Street's initial response to the bid, announced after the stock market closed, was to anticipate the possibility of a higher offer. Novell's shares jumped $1.32, or nearly 28 percent, to $6.07 in after-hours trading." (Thanks to Jeff Schroeder)See also: Elliott's press release about the offer. "Over the past several years, the Company has attempted to diversify away from its legacy division with a series of acquisitions and changes in strategic focus that have largely been unsuccessful. As a result, we believe the Company's stock has meaningfully underperformed all relevant indices and peers. With over 33 years of experience in investing in public and private companies and an extensive track record of successfully structuring and executing acquisitions in the technology space, we believe that Elliott is uniquely situated to deliver maximum value to the Company's stockholders on an expedited basis." That suggests some rather significant changes should this deal be accepted.
LiMo Foundation Seeks Alliance With WAC (InformationWeek)InformationWeek reports that the LiMo Foundation is seeking a partnership with the Wholesale Applications Community. "In an open letter sent Tuesday, LiMo Foundation executive director Morgan Gillis said the mobile Linux platform group offers its "full support, our committed participation, and our immediate practical assistance" to WAC. Formed last month by 24 operators at the World Mobile Congress, WAC is an effort to build an open platform to deliver mobile phone apps. WAC's members, now numbering 27 mobile industry firms, serve some 3 billion mobile phone users."
Legal Announcements Apple files a patent suit against HTCApple has announced the filing of a lawsuit against HTC alleging the infringement of 20 of Apple's patents. "'We can sit by and watch competitors steal our patented inventions, or we can do something about it. We've decided to do something about it,' said Steve Jobs, Apple's CEO. 'We think competition is healthy, but competitors should create their own original technology, not steal ours.'" The press release does not say whether HTC's Android phones are the ones being targeted here.
Articles of interest Archos 5 Internet Tablet with Android Review (Anything but ipod)Not all Android devices are phones: here's an extensive review of the Archos 5 tablet (part 1), (part 2) on the Anything but ipod site. "The Archos 5 Internet Tablet with Android is a very touchy subject for some people as its part awesome, part scrap. On one hand, you have excellent hardware (with the exception of the resistive touch screen, which should have been capacitive) and awesome support for video and a giant market of apps to put on it, but on the other hand you have a device thats so unstable I legiimately wish it had a big giant red reset button on the back instead of a tiny reset hole."
Microsoft Takes Down Whistleblower Site (Wired)Wired reports that Microsoft has pulled Cryptome off the net. "Microsoft dropped a DMCA notice alleging copyright infringement on Cryptomes proprietor John Young on Tuesday after he posted a Microsoft surveillance compliance document that the company gives to law enforcement agents seeking information on Microsoft users. Young filed a counterclaim on Wednesday arguing he had a fair use to publishing the document, a full day before the Thursday deadline set by his hosting provider, Network Solutions." Wired is also hosting the document in question at the moment.
Novell flirts with Citrix (Channel Register)Channel Register discusses a potential partnership between Novell and Citrix. "There has been some chatter about Citrix Systems - the corporate entity behind the open source Xen hypervisor and the commercialized XenServer product - hooking up with commercial Linux distributor Novell to work out some sort of deal to collaborate on Xen in a more meaningful way than they currently do. While the two parties are dancing a little bit closer, Novell is not going to adopt XenServer as its main hypervisor, as some have expected and others, like El Reg, have encouraged."
Hands-on: Ubuntu goes social, gains Me Menu in 10.04 alpha 3 (ars technica)Ryan Paul looks at the Me Menu in Ubuntu's Lucid Lynx. "The Me Menu, which Canonical unveiled in December, provides a unified interface for managing your presence on instant messaging and social networking services. A text box that is embedded in the menu allows users to publish status messages to all of their accounts. The menu also provides easy access to the standard account and identity configuration tools."
How Sun's need to control the code cost them the company (ZDNet)Jeremy Allison looks back at the demise of Sun Microsystems on ZDNet. "The Solaris operating system, the Java language and virtual machine, the OpenOffice office suite - all of the really large software projects that Sun released - had strings attached that stopped any real external community from forming around the code. Usually it was the demand that any code contributions be contributed directly to Sun for their own use in proprietary products that was the major failing of all the Sun 'community' projects. Poor licensing choices, demands for ownership of all contributors work, ignoring contributors outside of Sun, all of these can be blamed for Suns inability to maintain active coding communities around their Open Source code, but in the end it comes down to the desire to maintain control and ownership of the code at all costs. People are smart enough to understand when theyre being taken advantage of, especially programmers."
Resources CE Linux Forum Newsletter: February 2010This issue of the CE Linux Forum Newsletter covers ELC 2010 sessions announced and Registration is Open, 32nd Japan Technical Jamboree, CELF Hardware Donations, and CELF sponsors LWN.net.
Calls for Presentations PyCon 2011 - Call for Tutorial VolunteersPlans for PyCon 2011 in Atlanta have already begun. "The main conference will once again be proceeded by two days of tutorials. There was quite a bit of feedback from students and teachers this year that we want to incorporate in next years classes. In order to do this, more people need to get involved; why not you?"
Upcoming Events Ubuntu 10.10 Developer Summit AnnouncedUbuntu community manager Jono Bacon has announced that the Ubuntu Developer Summit for Ubuntu 10.10 is taking place May 10 - 14, 2010 at Dolce La Hulpe Hotel and Resort in Brussels, Belgium. "The Ubuntu Developer Summit one of the most important events in the Ubuntu calendar and at it we discuss, debate and design the next version of Ubuntu. We bring together the entire Canonical development team and sponsor a large number of community members across the wide range of areas in which people contribute to Ubuntu. This includes packaging, translations, documentation, testing, LoCo teams and more. UDS is an incredible experience, filled with smart and enthusiastic people, fast paced and exhausting, but incredibly gratifying to be part of the process that builds the next Ubuntu."
FSFE proclaims Document Freedom Day 2010The Free Software Foundation Europe has announced Document Freedom Day 2010 will be held on March 31. "On Document Freedom Day, we will raise awareness for Open Document Formats and Open Standards by organizing activities all over the world together with partner organizations and volunteers. During the whole month of March, we will spread the word on open document formats and Open Standards."
Day Against DRM: May 4The Free Software Foundation has announced that May 4, 2010 will be this year's International Day Against Digital Restrictions Management (DRM). "The Day Against DRM will unite a wide range of projects, public interest organizations, web sites and individuals in an effort to raise public awareness to the danger of technology that restricts users' access to movies, music, literature and software; indeed, all forms of digital data. Many DRM schemes monitor a user's activities and report what they see to the corporations that impose the DRM."
Events: March 11, 2010 to May 10, 2010The following event listing is taken from the LWN.net Calendar.
If your event does not appear here, please tell us about it. Web sites LearnUbuntu.com.auA new website, LearnUbuntu.com.au, is available. The site offers an introduction to Ubuntu and various training options. "If you are considering Ubuntu for your home or office computing requirements, please consider our training and installation packages. Face to face, classroom-style or email based, Jon [Jermey] can provide training to get you up and running faster and much more smoothly."
Audio and Video programs FOSDEM 10 videos now availableVideo recordings from the Distribution Developer Rooms at FOSDEM 10 are available. "All but two talks were recorded and are available in Ogg Theora+Vorbis format, in low-bandwidth (~300 kbit/s) and high-bandwidth (~1.5 Mbit/s) versions. These recordings should also be available later on the FOSDEM YouTube channel."
Page editor: Rebecca Sobol
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||