LWN.net Weekly Edition for September 29, 2011

Papering over a binary blob

GTA04 bills itself as the next-generation OpenMoko phone. The end product is intended to be a board that can fit inside an existing OpenMoko handset case but which provides a number of features the OpenMoko phone never had: 3G/HSPA connectivity, USB 2.0 on-the-go support, BlueTooth, an FM transceiver, various sensors (barometer, gyroscope, ...), and the obligatory flashlight device. Also provided will be a Debian port to make the whole thing work. The hardware design is reasonably far along, with boards from an early adopter program about to make their way to developers. There is, though, one nagging problem that the project would still like to solve.

The GTA04 uses a Marvell 8686 "Libertas" chip for its WiFi connectivity; that is the same chip used in the first generation OLPC XO laptop. That chip requires a significant firmware blob to be loaded before it can function. One of the earliest bugs filed at OLPC called for the replacement of this blob with free software, but nobody stepped up and got the job done. So, five years later, the GTA04 project, whose goal is the creation of a 100% free handset, is stuck shipping that same firmware blob.

Some projects would shrug their collective shoulders, treat the blob as part of the hardware, and move on to the long list of other problems in need of solution. Others might bite the bullet, get hardware documentation from somewhere, and write a replacement blob. Yet others might decide that Marvell's hardware is incompatible with their goals and find a different WiFi chip for their board. GTA04, though, has done none of the above.

What the GTA04 developers would like to do is documented on this project page:

In other words, the project proposes to add another microcontroller to its board for the sole purpose of shoving the firmware blob into the Marvell WiFi controller. By so doing, they turn the blob into "effectively circuitry" and, seemingly, no longer feel quite so dirty about it. The Free Software Foundation agrees with this goal, having apparently set it as a condition for their endorsement of the device:

One might object that, if the firmware blob contains malicious code, that code will still exist even if it does not pass through the CPU as data first. The GTA04 is meant to be a free device; the operating software will be under the user's control. So one would expect that the firmware blob will not be spontaneously replaced by something nastier. If the firmware blob were somehow able to maliciously "upgrade" itself against the will of the Linux system running the phone, it would be able to do any of a number of other unpleasant things regardless of which processor loads it into the WiFi controller. Meanwhile, if Marvell comes out with a new blob offering better performance or fixing bugs, users will no longer have the option of installing it on their phones.

It is, in other words, hard to see the benefits that are being bought through this exercise.

In fact, it would seem that the GTA04 project is wanting to saddle itself with a more complex design, higher hardware costs, less flexibility in the future and increased power usage to create a system that runs the exact same binary blob on the same controller. That's a high price to pay for the comfort that comes from having swept the blob under the rug where it can't be seen. This whole thing might be humorous if it weren't for the little fact that it would really be nice to see this project succeed; more freedom in this area is sorely needed. What they are trying to do is challenging enough without the addition of artificial obstacles. It will be a sad day if the pursuit of a truly free handset is impeded by an exercise in papering over an inconvenient binary blob.

(Thanks to Paul Wise for the heads-up).

Comments (59 posted)

Integrated color management with colord

Desktop Linux users used to be easily divided into two camps when it came to color management: there were the graphics geeks — who painstakingly profiled their scanners, printers, and displays and manually set up color management in each of their creative applications — and there was everyone else, who saw color management as an esoteric hobby with no practical value. The color geeks are quietly taking over, however, thanks to recent work that automates color management at the system and session level.

The root of the color management problem is that no two devices have exactly the same color reproduction characteristics: monitors and printers vary wildly in the tonal range and gamut that they can reproduce; similarly cameras and scanners vary wildly in what they can pick up. If you spend all week staring at a single display device it is easy to forget this, but anyone who has set their netbook on the desk beside a desktop LCD display recognizes immediately how different they can be. As a practical matter, when a user makes a printout and finds it too dark, or orders a piece of clothing online and is surprised at its color when it arrives, lack of color management is the problem.

But color management is essentially a solved problem that has yet to be implemented system-wide on Linux. Every display or input device can be profiled — that is, its color characteristics measured and saved in a standardized format like an ICC color profile. With profiles in hand, applications need only to perform a transformation on RGB data to map it from one profile (say, a camera's) to another (a display's). Most of the time, a mapping with a perceptual rendering intent is used, which means the final image will appear the same to the human eye on the display as it did through the camera viewfinder.

The old way

Several free software packages exist for applications to perform these transformations. By far the most popular is LittleCMS, an MIT-licensed library used by GIMP, Krita, ImageMagick, Inkscape, Scribus, and many other graphics applications. Another well-known alternative is Argyll, which is licensed under the AGPL. Argyll consists of a collection of command-line tools, however, rather than a shared library designed for use by application programs.

LittleCMS support simply gives each application the ability to perform profile-to-profile transformations. Users are still required to open the preferences dialogs in each application and specify the .icc file for each profile used in their hardware setup. As a result, "color management" came to be a feature that could be enabled or disabled on a per-application basis, since only a small percentage of users cared enough to have profiles for their displays and printers. A far better approach would be for the system to keep track of the relevant profiles (since displays, scanners, and printers tend to stay put for long periods of time), and have applications automatically retrieve them as needed — no user intervention required. That is the approach taken by both Windows and Mac OS X.

The technicolor dawn

This is where Richard Hughes's colord and Gnome Color Manager (GCM) come into play. Colord is a framework for automating the storage and retrieval of color profiles. It includes a system daemon that maintains an SQLite database to track which profiles map to which devices; it also provides a D-Bus interface for other programs to query (or in the case of GCM, add and change) profiles.

It can retrieve scanner and printer device IDs through Udev and present a simple interface with which users can select their preferred profile for each device. Multiple profiles per device are supported, to allow separate profiles for different paper stocks, ink options, flatbed and transparency adapters, and other parameters. An application can ask for the profile for a specific device using dot-separated qualifiers, such as RGB.Glossy.600dpi. If colord does not have a stored profile for the requested qualifiers, it can fall back one parameter at a time to find the best match, e.g., to RGB.Glossy.* or RGB.*.600dpi, and eventually back to RGB.*.* and the default * for a device ID.

The other half of the framework is GCM, which is a session-level process implemented as a GNOME settings daemon plug-in. GCM talks to the X.org session to retrieve the display device information through XRandR and, where possible, to set display settings such as video card gamma tables (VCGTs). The session-level process can also read ICC profile files stored in the user's home directory (as opposed to the system-wide ICC profile directories /var/share/color and /var/lib/color, which are watched by the colord daemon).

Hughes said in a recent talk that the system/session split, though it sounds confusing at first, is required for several reasons. First, the system daemon needs to talk to other system-level processes such as CUPS (for printers) and SANE (for scanners), and needs to work even when there is no active user session (consider shared printers, printers that print directly from memory cards, and GDM login screens, for example). But the session-level process also needs to be separated from privileged operations for other reasons. SELinux will not allow a system daemon to load files (in this case, ICC profiles) from a user's home directory because of security concerns, and colord would not even be able to read them if the home directory is encrypted. The separation also makes it possible for KDE, LXDE, or other environments to write their own colord-compatible session management code independent of GCM.

The application's viewpoint

The framework provided by colord and GCM is very high-level; applications must still do the color transformations using LittleCMS or another library. But by relying on colord to provide the correct profile information for a particular hardware device, there is less for the application developer to worry about.

GCM provides an interface for the user to manage his or her profiles, which alleviates the problem of manually setting the same preferences in every application. But applications do need to be updated to work with colord. At the moment, the list of colord-compatible programs is short: Hughes told Alexandre Prokoudine at Libre Graphics World that it includes CUPS, Foomatic, Simple Scan, and Compiz-CMS. The GTK+ toolkit is also compatible he said, and KDE support is on the way.

General-purpose GTK+ or KDE applications can decide to entrust colord with their entire color management workflow, but creative applications may still want to provide additional options. For example, users may want to simulate an output device with a different profile to "soft proof" before printing or rendering images. Colord can still help, because it can maintain profiles for "virtual devices" in addition to the local hardware, but of course presenting that functionality to the user requires pulling back the curtain away from the "it just works" approach favored by simple applications.

Perhaps understandably, support for colord among the already-color-aware Linux creative suite appears to be growing slowly. Krita has integration plans, but it may be a while before the other applications that already implement color management adopt colord, on the grounds that the users who already care have also already configured their software.

On the other hand, Hughes has been busy submitting patches to other projects to enable colord support. He added the colord support now found in CUPS' gstoraster filter (which converts PostScript and PDF content to raster data) and foomatic (which can directly generate printer-language files) and has reportedly been patching other tools as well. Colord debuted in 2010, and was available in GNOME 3.0, but is now a hard dependency for the imminently-due GNOME 3.2, so for the first time, application developers can count on its availability.

Where profiles come from

As a practical matter, the big lingering question for end users is where the all-important device profiles come from. After all, it makes no difference how easily the applications can retrieve the default profile for a display or printer if there is no profile available. Hughes has designed colord to support profiles from a variety of sources, and always takes the "the user is right" approach: a manually-selected profile is assumed to be better than a generic one.

The best possible profiles are those that are created with proper measuring device: a tristimulus colorimeter for displays, a high-quality target for scanners or digital cameras, or a spectrophotometer for a printer. But all of these methods involve some outlay of cash. The price of colorimeters has fallen sharply in recent years, but even the cheapest devices run close to US $100, which can sound excessive for a hardware device only used on isolated occasions. There are non-profit producers of quality IT 8.7 targets, such as Wolf Faust, but they are still not free. Spectrophotometers, which measure reflected light, remain expensive, although there are paid services to which users can mail a test printout and get a high-quality ICC profile in return. But the one-time cost of buying such a profile rises when you consider the need to have each ink and paper combination measured separately.

Still, GCM does support creating a display profile with a USB colorimeter, importing camera or scanner images of calibration targets for scanner or camera profiles, and scanning printouts to generate printer profiles. GCM uses the low-level profiling tools from Argyll for much of this process, but Hughes has also been adding native drivers for common colorimeters like the Pantone Huey.

The next rung down on the quality ladder is manufacturer-provided profiles. These vary depending on device type and quality control. An expensive film scanner might ship with an accurate profile on the accompanying CD, while an inexpensive laptop display is completely hit or miss. As Hughes explained,

Even where manufacturer-provided profiles exist, however, Linux distributions generally do not have the rights to ship them, so they are inaccessible to an "it just works" framework.

As an alternative, Hughes said, GCM can probe XRandR devices for Extended display identification data (EDID), and auto-generate a basic display profile based on it.

Here again, Hughes warned that the quality of the device in question correlates to the quality of the EDID data one should expect. Manufacturers who regularly swap in LCD panels from Korea, China, and Taiwan without indicating a difference in the model number are likely to burn in generic EDID data that covers the average characteristics of the product line.

Unfortunately, how useful the provided EDID data is can also depend on extraneous factors like the video driver. For example, the NVIDIA binary drivers only support XRandR 1.2, which in turn does not include support for sending separate VCGTs to multiple monitors. Thus regardless of the quality of the data, at least one display will be forced to use gamma tables that are only accurate for a different monitor. It is also possible to run into older monitors that do not provide EDID information at all, or to have KVM switches or splitters lose it en route to the video card.

Kai-Uwe Behrmann of the Oyranos project, a competing color management framework, recently undertook an effort to build an open, user-contributed database of device profiles. Called taxi, the system was developed in the 2011 Google Summer of Code by student Sebastian Oliva. There is a web front-end running at icc.opensuse.org, which returns both raw .icc files and JSON-formatted metadata about the source device. It is an interesting approach, though one with its own set of problems. Display characteristics change over time, for example, as the backlight ages, and there is no way to verify the conditions under which another person's profile was created.

In the final analysis there is no automated substitute for a good profile created with quality test materials; a fallback profile will always be better than nothing, but unpredictable. Perhaps the best thing the community at large can do if it wishes to spread good, non-generic profiling is to make sure that the distributions and desktop environments pack a colorimeter and IT 8.7 target into the "booth box" when heading to community Linux shows. It certainly would not hurt.

The future

The colord daemon and GCM session manager provide the framework on which future applications can build end-to-end, "it just works" color management for the Linux desktop environment. That future is a big step up from the recent past, where each color-aware application needed to pester the user individually for a collection of profile settings, but it is still not color nirvana. For that one needs to look ahead to full-screen color management.

Hughes outlined the situation in his interview at Libre Graphics World. A full-screen color managed system would need a color-aware compositing manager, but would also allow applications to mark specific regions of the screen to "opt out" from the transformation — UI widgets are assumed to have sRGB pixels, for example, but a photograph would have a camera profile instead. But by pushing the transformation into the compositor, it becomes possible to offload the calculations to the GPU, potentially speeding up workflow dramatically.

The compiz-cms plug-in is an attempt to do this, although an incomplete one: because the applications are not able to mark specific regions with different transformation needs, the plug-in assumes all images are sRGB. For Compiz users, however, it is an intriguing glimpse at the possibilities.

Behrmann has developed a draft specification for communicating color regions, formerly called net-color but recently renamed X Color Management (currently in the draft stage for version 0.3). Hughes has concerns about the specification for use with colord, including its network transparency and its ease of implementation for application developers, but so far there is not a competing proposal. As he told Libre Graphics World, however, he is hoping to put together a plan for GNOME 3.4 that will incorporate colord support into Mutter, the GNOME compositing manager, but it is likely to involve changes to GTK+ and other parts of the stack as well.

Color-management is not something that can be completely automated, but desktop Linux is on track toward the next best thing, that the choice of sane defaults is automated. Colord effectively moves color-awareness out of the application space and down to the session level. For most people, on modern video hardware, an automatically-detected display profile will provide a "good enough" color balanced experience. The list of applications taking advantage of it is short, but growing, but the color geeks need not frown — as was the case before, those who are interested in hand-tweaking their color matching can still dig into the details and customize the experience.

Comments (25 posted)

Control over our hardware

UEFI secure boot

LWN first wrote about the UEFI "secure boot" feature last June. At that point, the potential for trouble was clear, but it was also mostly theoretical. More recently, it has been revealed that Microsoft intends to require the enabling of secure boot for any system running the client version of Windows 8. That makes the problem rather more immediate and concrete.

The secure boot technology is not without its value. If an attacker is able to corrupt the system's firmware, bootloader, or kernel image, no amount of good practice or security code will be able to remedy the situation; that system will be owned by the attacker. Secure boot makes such attacks much harder to carry out; the system will detect the corrupted code and refuse to run it. An automated teller machine should almost certainly have this kind of feature enabled, for example. Many LWN readers would likely find that the amount of time they have to put into family technical support would drop considerably if certain family members had their systems protected in this way.

Secure boot requires trust in whatever agency applies its signature to the code. A better name for the feature might be "restricted boot," since it restricts the system to booting code that has been signed by a trusted key. The idea is sound enough, except for one little problem: who decides which keys are trusted? Hardware vendors seeking Microsoft certification will create a secure boot implementation that trusts Microsoft's keys. They need not trust any others - not even from other hardware vendors selling Windows-compatible hardware.

Secure boot would not be a big problem if users were guaranteed the right to install their own keys or to disable the feature altogether. The owner of a specific computer may well want to restrict the system to booting kernels signed by Red Hat, SUSE, or OpenBSD. They might also want to say that Windows is not a trusted system - but only as long as the driver firmware needed to boot is signed by somebody other than Microsoft. The owners may want to build their own kernels signed with their own keys. Or they may decide that secure boot is a pain that they would rather do without. With this freedom, secure boot could be a beneficial feature indeed.

But nobody is guaranteeing that freedom. The ability to disable secure boot, at least, may come standard on traditional "desktop PC" systems, but the role of those systems in the market is declining. Microsoft very much wants to push Windows into tablets, handsets, refrigerators, and other new systems. Such machines do not have a stellar record with regard to enabling owner control even now; it does not seem likely that Microsoft's certification requirements will improve that situation. Just as things seemed to be getting better in that area, we may be about to see them get worse again.

That said, loss of control over our systems is not a foregone conclusion. Microsoft will have to be very careful about monopoly concerns in the areas where it is dominant. In the areas where Microsoft has failed to gain dominance, there is no guarantee that it ever will. And, even then, users have been clear enough about their desire for access to their own systems to gain the attention of some big handset manufacturers. Lockdown via secure boot is not a foregone conclusion; in fact, it looks like a battle we should be able to win. But we must certainly keep our eyes on the situation.

Trusted botnets

The pointer to this paper by Alan Dunn et al [PDF] came via Alan Cox. These investigators have figured out a way to use the trusted platform module (TPM) found in most systems to hide malware from anybody trying to investigate it. In essence, the TPM can be used to create a trusted botnet capable of resisting attempts to determine what the hostile code is actually doing.

The TPM provides a number of cryptographic functions along with a set of "platform configuration registers" (PCRs) that can be used to make guarantees about the state of the system. As long as the boot path is trusted, the TPM can sign a message containing PCR values proving that a specific set of software is running on the system. Fears that this "remote attestation" capability would be used to lock down systems from afar have not generally come true - so far. The TPM can also perform encryption and decryption of data, optionally tied to specific PCR values.

One other TPM-supported feature is "late launch," a mechanism by which code can be executed in an uninterruptable and unobservable manner. Late launch is used to enable mechanisms like Intel TXT; it is another way of ensuring that only "trusted" code can run on a system.

The attack described in the paper requires gaining control of the TPM, an act which may or may not be easy (even after the system itself has been compromised) depending on how the TPM is being used. Once that has been done, the compromised software will be able to attest to a remote controlling node that it is in full control of the system. That node can then send down encrypted code to be run in the late launch mode. This code is limited in what it can do - it cannot call into the host operating system for anything, for example - but it can make important policy decisions controlling how the malware will operate.

Understanding - and defeating - malware often depends on the ability to observe it in action and reverse engineer its decision making. If it proves impossible to observe malware in operation or to run it in a virtualized mode, that malware will be harder to stop. The attack is not easy, but experience has shown that the world does not lack for capable, motivated, and well-funded attackers who might just take up the challenge. That would not bode well for the future security of the net as a whole.

Needless to say, the protection of botnets seems counter to the objectives that led to the creation of the TPM in the first place. It has always been clear that technology imposed in the name of "security" has the potential to cost us control over our own systems. Now it seems that technology could even hand control over to overtly hostile organizations. That does not seem like a more secure situation, somehow.

Comments (18 posted)

Page editor: Jonathan Corbet

Security

SSSD: System Security Services Daemon

Linux client-side configuration for centralized identity and authentication stores with caching and offline support has traditionally required configuration of several independent components; the end result and operational efficiency has not always been optimal. Offline support means that previously logged-in users' password hashes and identity information are stored locally so that all operations requiring authentication or UID to username mapping can be processed locally.

Let us consider the components in a typical case where LDAP and Kerberos are used for identity and authentication and there are mobile users who roam around with their laptops between different networks. First, nss_ldap has to be configured to retrieve the user identity information from LDAP. Then pam_krb5 needs to be configured to allow for user authentication. Alas, neither of these components supports caching or offline mode. So nscd needs to be configured to cache user information. And then finally pam_ccreds is needed for caching authentication credentials while offline. Most readers would probably agree that this scheme isn't the most efficient and robust solution so there may be some room for improvement.

What SSSD does

SSSD provides several features but the most important is to provide access to identity and authentication resources through a common framework that can provide caching and offline support to the system. For offline support SSSD keeps the credentials in a local cache. When a user logs in to an organization's network with their centrally managed account on their laptop, the user information and credentials are automatically stored in the SSSD cache.

Secondly, it supports queries to multiple servers. Thus, one can query a number of different user databases. Third, the daemon has its own NSS and PAM interfaces for use by client systems. From a performance point of view, this offers advantages. Instead of needing to set up a connection for each and every application that queries the NSS LDAP database, only a single socket from SSSD to the LDAP server is required. And all these features can be configured in a single configuration file.

For users, authentication and Kerberos tickets will then work in a straightforward way: when logging in while online (i.e., a connection to the central user account service can be made), a user enters their username and password and, once verified, a Kerberos ticket for the user is automatically acquired. A successful online login also refreshes the user's cache entry without any manual steps.

When logging in while offline, authentication is done against the cached information. When SSSD observes that the system is online again (e.g., after the user has established a VPN connection), it can acquire a Kerberos ticket for the user in the background without any additional effort by the user. Kerberos tickets can be also be automatically renewed based on the SSSD configuration. If an organization has implemented single sign-on (SSO) using Kerberos then SSSD helps to provide very smooth but secure user experience.

In practical terms, SSSD has one central configuration file, /etc/sssd/sssd.conf, which contains all the configuration options needed for one or several domains, possibly with different retention policies for each domain. NSS and PAM are configured to use the SSSD modules, libnss_sss.so and pam_sss.so, respectively, and the sssd service needs to be enabled. Distributions like Fedora and RHEL have also integrated SSSD as part of their authconfig tool used to configure user information sources removing the need for manually editing NSS or PAM configuration files (also providing basic configuration for sssd.conf).

It should be noted that, in addition to sssd.conf, /etc/krb5.conf needs to be configured when using Kerberos for authentication. That is also required for applications and utilities using the Kerberos libraries directly. The manual page sssd.conf(5) provides a comprehensive overview of the available configuration options and Fedora SSSD Guide offers a complete walk-through for setting up SSSD.

In addition to identity and authentication methods like LDAP and Kerberos, SSSD also includes support for netgroups and proxied authentication (for example to be used with NIS, since a native NIS backend is not yet available, although it is in the roadmap). These might be helpful features for organizations during a transition when moving from NIS to LDAP/Kerberos. Another interesting feature is host-based access control (HBAC) using FreeIPA. HBAC rules can be used to control which users or groups can access a specific host.

Past and future

SSSD can trace its origins to the FreeIPA project. The SSSD project, originally codenamed "Bluebox" for reasons lost to history, was envisioned as the FreeIPA's primary client component. As SSSD began to take shape, it was realized that many of the enhancements that were being developed to support FreeIPA would also be valuable for users of other LDAP and Kerberos environments. Thus the long-term vision for SSSD was revised and it became a project in its own right, related to FreeIPA, but distinct. Since its introduction in Fedora 11, SSSD's user and developer community has grown rapidly. It is now available for all major distributions (Fedora, Ubuntu, RHEL, openSUSE, and others) and there are already some large enterprises which have already deployed it globally as part of their Linux installations.

Several notable new features are in the roadmap. Work is going on to use sudo's plugin interface in SSSD to make it easier to maintain centralized sudo rules that also function while offline. Another planned addition is automounter integration which would allow SSSD to retrieve LDAP served automount maps for autofs. Enhanced Active Directory integration and D-Bus based interfaces for extended user information and data are also coming. There are other interesting features planned for SSSD — additional suggestions and participation from the community is warmly welcomed.

The use of the SSSD offers many benefits, especially for administrators and mobile users. Instead of having multiple accounts, users can simply use a single account. Kerberos tickets can be automatically acquired and renewed, which makes the use of "kerberized" services seamless but secure. Offline mode can also be useful in data centers to help bridge the gap caused by a temporary failure of the LDAP or Kerberos servers. Compared with older solutions, SSSD offers far more flexible management and simplified administration for client-side identity and authentication needs.

Comments (13 posted)

Brief items

Quote of the week

Comments (5 posted)

Garrett: Supporting UEFI secure boot on Linux: the details

Comments (7 posted)

Garrett: UEFI secure booting (part 2)

Comments (61 posted)

MySQL.com Hacked to Serve Malware (PC World)

Comments (17 posted)

Mozilla and Tor on the TLS attack

Comments (9 posted)

New vulnerabilities

apt: altered package installation

Comments (1 posted)

cherokee: multiple vulnerabilities

Comments (none posted)

flash-player: multiple vulnerabilities

Comments (none posted)

foomatic: insecure temporary files

Comments (none posted)

NetworkManager: privilege escalation

Comments (none posted)

pango: arbitrary code execution

Comments (none posted)

qt: code execution

Comments (none posted)

quassel: denial of service

Comments (none posted)

Page editor: Jonathan Corbet

Kernel development

Brief items

Kernel release status

3.1-rc7 came out on September 21, several microseconds after the LWN weekly edition was published. Linus said:

Among other things, that implies that the next merge window may overlap with the Kernel Summit, which could involve some craziness of its own.

Stable updates: no stable updates have been released in the last week.

Comments (none posted)

Quotes of the week

Here is an example of a PGA (Pin Grid Array) chip seen from underneath:

        A   B   C   D   E   F   G   H
      +---+
   8  | o | o   o   o   o   o   o   o
      |   |
   7  | o | o   o   o   o   o   o   o
      |   |
   6  | o | o   o   o   o   o   o   o
      +---+---+
   5  | o | o | o   o   o   o   o   o
      +---+---+               +---+
   4    o   o   o   o   o   o | o | o
                              |   |
   3    o   o   o   o   o   o | o | o
                              |   |
   2    o   o   o   o   o   o | o | o
      +-------+-------+-------+---+---+
   1  | o   o | o   o | o   o | o | o |
      +-------+-------+-------+---+---+

This is not tetris. The game to think of is chess.

Comments (1 posted)

A kernel.org status update

Full Story (comments: 75)

Kernel development news

The forest on the move

Quite a few trees have found new homes in the mean time. Here is an updated version of the list of relocated trees:

ACPI	https://github.com/lenb/linux.git
ALSA	git://github.com/tiwai/sound.git
ALSA driver	git://github.com/tiwai/alsa-driver-build.git
ALSA SOC	git://opensource.wolfsonmicro.com/linux-2.6-asoc.git
amd64 EDAC	git://amd64.org/linux/bp.git
APM	git://twin.jikos.cz/jikos/apm
arm-soc	git://git.linaro.org/people/arnd/arm-soc.git
DRM	git://people.freedesktop.org/~airlied/linux
fbdev	https://github.com/schandinat/linux-2.6
HID	git://twin.jikos.cz/jikos/hid
hwspinlock	git://github.com/ohadbc/hwspinlock-next.git
infiniband	https://github.com/rolandd/infiniband
input	https://github.com/dtor/input
ipvs	git://github.com/horms/ipvs.git
kbuild	http://repo.or.cz/w/linux-kbuild.git
kvm	git://github.com/avikivity/kvm.git
libata	git://github.com/jgarzik/libata-dev.git
linux-next	git://github.com/sfrothwell/linux-next.git
mainline	git://github.com/torvalds/linux.git
mmc	git://dev.laptop.org/users/cjb/mmc mmc-next
networking	git://github.com/davem330/net
pm	git://github.com/rjwysocki/linux-pm.git
regmap	git://opensource.wolfsonmicro.com/regmap.git
SCSI	git://bedivere.hansenpartnership.com/git/scsi-rc-fixes-2.6.git git://bedivere.hansenpartnership.com/git/scsi-misc-2.6.git
slab	git://github.com/penberg/linux.git
tip	git://tesla.tglx.de/git/linux-2.6-tip
tmem	git://oss.oracle.com/git/djm/tmem.git
trivial	git://twin.jikos.cz/jikos/trivial
utrace	git://github.com/utrace/linux.git
v9fs	git://github.com/ericvh/linux.git
wireless	git://git.infradead.org/users/linville/wireless.git git://git.infradead.org/users/linville/wireless-next.git git://git.infradead.org/users/linville/wireless-testing.git
xen	git://oss.oracle.com/git/kwilk/xen.git

That is a substantial list of moved trees, but, as linux-next maintainer Stephen Rothwell noted on September 27, that leaves 89 trees which still only exist on kernel.org. Those trees will not have seen any updates since kernel.org went off the net. Some of them will certainly be trees that are currently idle or close to it; not every tree feeding into linux-next carries patches for every development cycle. But others presumably exist for a reason; if kernel.org does not come back soon, they will need to find a different home.

One significant tree that has not moved is the stable release tree; the last stable updates came out on August 29.

With luck, kernel.org will come back soon and the above list will become moot. But kernel.org, when it returns, may look somewhat different. It has already been announced that there will be no shell access to the machines hosting the git trees. There may be other security measures put into place as well, some possibly requiring changes in how developers operate. Making changes of that nature in the time left before the next merge window could be hard to do. The 3.2 development cycle, in other words, might be a bit more interesting and less smooth than usual.

Comments (7 posted)

A look at the 3.1 development cycle

The most active developers in the 3.1 cycle were:

Most active 3.1 developers

By changesets Takashi Iwai 140 1.7% Mark Brown 137 1.6% Mauro Carvalho Chehab 127 1.5% Roland Vossen 108 1.3% Russell King 106 1.3% Al Viro 105 1.2% Arend van Spriel 105 1.2% Joe Perches 93 1.1% Rafał Miłecki 87 1.0% Alan Cox 85 1.0% Axel Lin 80 0.9% Christoph Hellwig 78 0.9% Jon Medhurst 75 0.9% Ben Skeggs 68 0.8% Neil Brown 68 0.8% Wey-Yi Guy 66 0.8% Kuninori Morimoto 65 0.8% David S. Miller 63 0.7% Shawn Guo 61 0.7% Jonathan Cameron 59 0.7%

By changed lines Greg Kroah-Hartman 121512 14.8% Ralph Metzler 26043 3.2% Takashi Iwai 24919 3.0% Vladislav Zolotarov 24109 2.9% Nicholas Bellinger 22825 2.8% Roland Vossen 20472 2.5% Alan Cox 20429 2.5% Oliver Endriss 19472 2.4% matt mooney 16804 2.0% Krishna Gudipati 15920 1.9% Arend van Spriel 15659 1.9% Chaoming Li 15319 1.9% Dominik Brodowski 15251 1.9% Mauro Carvalho Chehab 12974 1.6% Jonas Bonn 11112 1.4% Mark Brown 10820 1.3% Kamil Debski 9311 1.1% Andy Grover 6753 0.8% Yaniv Rosner 6526 0.8% Joe Perches 6502 0.8%

Media drivers would appear to dominate the listings on the "by changesets" side. Takashi Iwai continues to be incredibly productive in the area of audio drivers; Mark Brown, too, works mostly in the audio area. Mauro Carvalho Chehab is the Video4Linux2 maintainer; all of his patches fall within that tree this time around. Roland Vossen, instead, contributed a large number of changes to the Broadcom wireless network driver. Russell King not only serves as the top-level ARM maintainer; he also made a number of changes in that tree this time.

Greg Kroah-Hartman has, once again, been the top changer of lines in the kernel. Once again, the bulk of his work is in the staging tree; this time, though, he got there by deleting a number of drivers that either were not going to make it into the mainline or were on their way out. Ralph Metzler only contributed five patches, but three of them added new drivers to the Video4Linux2 tree. Takashi Iwai shows at the top of both columns for his sound driver work, Vladislav Zolotarov contributed a single patch with a bunch of new Broadcom firmware, and Nicholas Bellinger continues to enhance the SCSI target code.

Of the 182 employers identified as contributing to the 3.1 kernel, the most active were:

Most active 3.0 employers

By changesets (None) 1111 13.1% Red Hat 882 10.4% (Unknown) 749 8.8% Intel 616 7.3% Broadcom 428 5.1% Novell 380 4.5% IBM 301 3.6% Texas Instruments 276 3.3% (Consultant) 223 2.6% Freescale 182 2.2% Linaro 170 2.0% Samsung 162 1.9% Google 150 1.8% Wolfson Microelectronics 142 1.7% Fujitsu 131 1.5% Renesas Electronics 100 1.2% Oracle 82 1.0% MiTAC 80 0.9% Nokia 79 0.9% (Academia) 73 0.9%

By lines changed Novell 162583 19.8% (None) 90119 11.0% Broadcom 76810 9.4% Red Hat 58262 7.1% Intel 43505 5.3% (Unknown) 27109 3.3% Metzler Brothers Systementwicklung GbR 23681 2.9% Samsung 23238 2.8% Rising Tide Systems 23090 2.8% IBM 22231 2.7% Texas Instruments 21130 2.6% Freescale 17270 2.1% Brocade 16587 2.0% Realsil Microelectronics 15868 1.9% Wolfson Microelectronics 14004 1.7% (Consultant) 13710 1.7% South Pole AB 12087 1.5% Linaro 11129 1.4% Oracle 9390 1.1% Nokia 7450 0.9%

Broadcom's extensive work to move its wireless driver out of staging caused it to move to a higher than usual position on both lists. Also notable is the continued slow climb by companies like Texas Instruments and Samsung; Nokia, instead, appears to be about to fall out of the top 20. The handling of Linaro deserves an explanation: contributions by Linaro assignees is normally credited back to their home companies. Nonetheless, Linaro makes an appearance on its own here as the result of the work of an increasing number of engineers employed by the organization itself.

Finally, here is a plot showing the number of changesets merged for each stabilization release (those after -rc1) for the last few development cycles:

The dark blue line represents the 3.1 development cycle; as might be expected, the number of changesets merged drops significantly after 3.1-rc4, which is when the kernel.org outage started. Both 3.1-rc5 and 3.1-rc6 were smaller than usual releases, but 3.1-rc7 has made up for some of the slowdown. It would appear that the subsystem maintainers affected by the outage have mostly managed to find new places to host their trees. The kernel development show manages to go on, even with the loss of its primary repository hosting site.

Comments (none posted)

A new approach to opportunistic suspend

Suspend blockers are a way for either the kernel or user space to tell the system that it is not a good time to suspend; the use of suspend blockers from user space is a privileged operation. To work properly, suspend blockers must be supported by any device that can wake up the system. Drivers for such devices will, when a wakeup event occurs, acquire a suspend blocker and wake any user-space process waiting on the event; once that process reads the event, the suspend blocker will be released. The key is that said user space process, if it is sufficiently privileged, can acquire a suspend blocker of its own before reading the event that woke it up. The overlap between the acquisition of the user-space blocker and the release of the kernel-space blocker allows for reliable system wakeups without the risk of suspending before a wakeup event has been processed.

One of the things John didn't like about this mechanism was the implicit suspend blocker API between user space and wakeup-capable devices. So he has come up with something a bit different, even if the core idea is quite similar.

The whole point of suspend blockers is to allow "important" processes to keep the device awake even if it would otherwise choose to suspend itself. So, John asked, why not just mark the important processes as such? His patch adds a new scheduler option:

    sched_setscheduler(0, SCHED_STAYAWAKE, &param);

Any time that a process has been marked in this way, the kernel simply will not suspend the system. That is true even if the given process blocks; otherwise, a disk I/O operation or page fault would be enough to put the system into an untimely sleep.

Of course, life is not quite so simple; there are times when it is desirable to suspend the system even with "important" processes on it. One of those is whenever the important process blocks on a device that is capable of generating wakeup events. Enabling suspend at such times requires tweaking the relevant drivers; in essence, a line like:

    sched_deboost_task_active_count(current);

must be added to remove the blocking process's "important" status just before putting it to sleep. When that device wakes the blocked process, its special status will be returned to it.

The only remaining problem is, once again, keeping the system from re-suspending before the important process gets its status back. That is handled by placing calls to __pm_stay_awake() and __pm_relax() around the code that wakes blocked processes. John also had to change the suspend code to make __pm_stay_awake() a bit less advisory than it is in current kernels. With that in place, the device will not suspend before any important processes have had the chance to reclaim their status.

As of this writing, the only feedback on the patch set has come from scheduler maintainer Peter Zijlstra. Suffice to say he didn't like it. According to Peter, opportunistic suspend is an attempt to paper over a problem that should be solved in user space; unimportant processes, he says, simply should not be running in the first place. That view, in turn, is unlikely to be popular in the Android camp. So, even though John's approach simplifies the wakelock idea, it does not seem likely to put an end to the debate over that approach to power management.

Comments (21 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.1-rc8 . (September 28, 2011)

• Linus Torvalds: Linux 3.1-rc7 . (September 22, 2011)

Core kernel code

• Deepthi Dharwar: [RFC PATCH V6 0/4] cpuidle: Global registration of idle states with per-cpu statistics . (September 22, 2011)

• Glauber Costa: per-cgroup /proc/stat statistics . (September 25, 2011)

• Con Kolivas: BFS cpu scheduler and skip list implementation . (September 25, 2011)

• Rafael J. Wysocki: PM: Runtime PM and device PM QoS refinements . (September 25, 2011)

• Gilad Ben-Yossef: Reduce cross CPU IPI interference . (September 25, 2011)

• Venkatesh Pallipadi: Introduce greedy hrtimer walk on idle . (September 26, 2011)

• John Stultz: [RFC] Proposal for optimistic suspend idea. . (September 27, 2011)

Development tools

• Stephane Eranian: perf: make perf.data more self-descriptive (v5) . (September 26, 2011)

Device drivers

• Mike Turquette: Add a generic struct clk . (September 23, 2011)

• Joerg Roedel: IOMMU: Make iommu_ops per-bus_type . (September 25, 2011)

• Heikki Krogerus: First round in OTG rework . (September 26, 2011)

• Mark Brown: clk: Add initial WM831x clock driver . (September 26, 2011)

• Grant Likely: drivercore: Add driver probe deferral mechanism . (September 26, 2011)

• Joerg Roedel: PCI: Add support for PASID capability . (September 27, 2011)

• David Daney: netdev/of/phy: MDIO bus multiplexer support. . (September 28, 2011)

• Jassi Brar: DMAEngine: Define interleaved transfer request api . (September 28, 2011)

• Linus Walleij: drivers: create a pin control subsystem v8 . (September 28, 2011)

Documentation

• Michael Kerrisk: man-pages-3.34 is released . (September 23, 2011)

Filesystems and block I/O

• David Wagner: UBI: new module ubiblk: block layer on top of UBI . (September 22, 2011)

• Darrick J. Wong: jbd2 metadata checksumming . (September 25, 2011)

Memory management

• Michel Lespinasse: V2: idle page tracking / working set estimation . (September 28, 2011)

Architecture-specific

• Don Zickus: x86, nmi: new NMI handling routines . (September 25, 2011)

• keguang.zhang@gmail.com: MIPS: Add basic support for Loongson1B (UPDATED) . (September 25, 2011)

• Ohad Ben-Cohen: OMAP groundwork for IOMMU-based DMA API . (September 25, 2011)

• Benoit Cousson: OMAP: Add initial support for DT on OMAP3 & OMAP4 . (September 26, 2011)

• Santosh Shilimkar: OMAP4: PM: suspend, CPU-hotplug and CPUilde support . (September 26, 2011)

• Deepthy Ravi: OMAP3EVM: Add support for MT9T111 sensor . (September 27, 2011)

• Jamie Iles: Initial support for Picochip picoXcell . (September 27, 2011)

• Mark Salter: C6X: New architecture . (September 28, 2011)

Miscellaneous

• Simon Glass: Add accurate boot timing to a Linux system . (September 25, 2011)

Page editor: Jonathan Corbet

Distributions

CentOS and updates

Many users and organizations have turned to CentOS as a replacement for Red Hat Enterprise Linux (RHEL). While the quality of the CentOS distribution as a RHEL substitute seems unquestioned, the speed of releases and updates is another matter entirely. The project is taking some steps to address this with continuous release repositories, but it's not entirely clear yet how those will work out.

The continuous release (CRs) repositories are relatively new to CentOS. The CRs include RPMs that will be included in the next CentOS point release. For instance, a repository for 5.6 will contain the RPMs for 5.7 until 5.7 is released. The CRs have to be added manually, they're not added by default — even though CentOS project leader Karanbir Singh strongly recommends using the CRs for security and bugfix releases.

Users will want to exercise caution, though. Though CRs are the sole option to users wanting updates before ISOs are released, the CR description warns that CR packages have been "less comprehensively reviewed" at the stage when they enter the CR repository.

To get CRs, admins will need to install the "centos-release-cr" package. Announcements about the updates that land in the CRs are sent to a separate mailing list. While users are waiting for the next CentOS release, updates go into the CR repository. When the next release is available, the repository is "emptied" and starts receiving updates for the next release.

CentOS released the CR repository for 5.6 on August 15, with a note from Singh saying "we hope to have the 5.7 Release ready in the next 7 to 10 days." The 5.7 release was actually announced on September 13, just under two months after RHEL 5.7 was released. The CR for 6.0 was promised "within the next 48 hours" in the release announcement for 6.0 by Singh. That was on July 10, when CentOS 6.0 was released eight months after RHEL 6.0. But CentOS 6.0 CR was finally announced on September 27, well more than two months after the 48-hour promise.

The delay and apparent absence of updates has been of concern for many users of CentOS. Consider, for instance, the update to fix the recent DoS for Apache. The advisory was released on August 24, and Red Hat issued an update on August 31. If you look on the CentOS update list, it doesn't appear that any update was released for CentOS 5.x or 6.x. However, when asked on the CentOS discussion list, Singh replied that it was already pushed and should be on the mirrors - in the continuous release repository, where few users had seen it.

Looking over the CentOS users list, it appears the concept and existence of CRs has not made its way to quite a few CentOS users. Even after the CR repositories for 5.x were announced, plenty of users were still asking about updates for that release on the CentOS list. Note that few CentOS users ever actually visit the lists. Many users get CentOS from hosting providers, or just hear that it's a good RHEL replacement and go on their way. They don't get deeply involved in the community and sign up for mailing lists, so it's not surprising that many users fail to notice a new and not well-advertised feature.

In other words, while there is no way to really know what percentage of CentOS users have taken the extra step to use the CR repository, one has to assume that it is quite low.

Meanwhile, if you're thinking of complaining to the CentOS list about the speed of updates and releases, be forewarned that complaints will likely be met with advice to purchase a RHEL subscription. Indeed, that is one option that users have if they're dissatisfied with CentOS. While the CR repository looks to be a better solution than simply waiting for point releases to get updates, there's still no promise as to how long it will take for updates to reach CentOS users - unsurprising, since those users have not paid for any such promises. Some of the updates have been almost immediate, such as the nspr update released one day after the upstream release. Others, such as this RHEL Qt update from September 21, have not yet been seen in CentOS as of this writing one week later.

Users might also consider Scientific Linux or another RHEL clone, if they can't afford or do not wish to pay subscription fees for RHEL. While Scientific Linux is not as committed to being an exact clone of RHEL, it comes very close. And the project provides a steady stream of updates that follow RHEL, seemingly in a more timely manner.

Having the continuous release repositories is a positive step for CentOS, but it still seems to fall short of providing the kind of timely updates that many production users are likely to want. Providing timely security updates is hard - even, seemingly, when another company is doing the work of actually fixing the code involved. Users who are concerned about updates will want to consider what the CentOS community appears to be able to do and come to their own conclusions as to whether the addition of CR improves things enough to meet their needs.

Comments (3 posted)

Brief items

Distribution quote of the week

Comments (none posted)

CentOS 6.0 continuous release available

Full Story (comments: none)

Ubuntu 11.10 Beta 2 (Oneiric Ocelot) Released.

Full Story (comments: none)

Distribution News

Debian GNU/Linux

Bits from dpkg developers - dpkg 1.16.1

Full Story (comments: none)

delegation for the Debian NM Front Desk

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 424 (September 26)
• Fedora Weekly News Issue 286 (September 21)
• Maemo Weekly News (September 26)
• openSUSE Weekly News Issue 194 (September 24)
• Ubuntu Weekly Newsletter, Issue 234 (September 25)
• Vyatta Monthly Newsletter (September)

Comments (2 posted)

Page editor: Rebecca Sobol

Development

An introduction to creating GCC plugins

The free and open source community is largely based on extensibility, sharing, and modularity, so projects will often provide an API that allows users to add functionality. One way to accomplish that is via plugins to the base project, which allows a user to customize the project to their needs and easily share the result with others who might also see utility in such a modification. Such a framework was recently implemented in GCC.

As of version 4.5 of GCC, a user can create a plugin that provides extra features to the compiler, which they can further pass around as a shareable module. GCC plugins provide developers with a rich subset of the GCC API to allow them to extend GCC as they see fit. Whether it is writing an additional optimization pass, transforming code, or analyzing information, plugins can be quite useful.

In this article we will learn how to develop a GCC plugin by creating one that can be used for spell checking read-only strings in an executable. What is the reason for doing so, one might ask? Well, it's a lesson on plugins, and this example provides a nice way to investigate this aspect of GCC.

Creating a plugin

A plugin is just a shared object file, following an API defined by GCC. To get started cooking up a plugin we will need to obtain a version of GCC, 4.5 or later. After your GCC is built, you are ready to get your hack on. Create a development directory "mkdir myplugin" and "cd myplugin" From there we will create the source file which will be our plugin: myplugin.c.

Plugin preparation

The first step in creating a plugin is to fill out some of the structures that the plugin API provides. These allow GCC to properly call the plugin, provide the GCC user some "help" information, and verify that the plugin is able to run on the version of the compiler that is trying to use the plugin. Before I introduce the plugin API, take a peek at it for yourself, the easiest way to figure out where the API header is located is to run the GCC that you are targeting your plugin for:

    gcc -print-file-name=plugin   

The files in there are the GCC API accessible to your plugin. If, however, the command simply prints the word "plugin", that means those files are not installed on your system. Of interest right now is the boiler-plate stuff, located in gcc-plugin.h. The following structures are provided by the API, and their fields are documented in gcc-plugin.h.

• struct plugin_name_args:
  • Contains information GCC deduces about the plugin that is being called.
  • We do not fill this structure out, rather GCC passes us an instance of it when our plugin's initialization routine is called. We can also get arguments for the plugin that were passed via the command line, such as: -fplugin-arg-myplugin-foo=bar. All plugin arguments are prefixed with the "basename" of the plugin, which is the name of the plugin shared object file that results after compilation but without the .so extension.

• struct plugin_info:
  • Version and help string for your plugin, what gcc -v will display.
  • Must be registered upon initialization of your plugin
  • Optional

• struct plugin_gcc_version:
  • Ensures that your plugin will operate on the appropriate version of GCC.
  • Gets passed as input to the plugin via initialization. We can use this to verify version information.
  • Optional

Now to get started. Using your favorite text editor, create a file called myplugin.c and include the headers for the basic GCC API.

    #include <gcc-plugin.h>  

The next thing we must define is a variable called plugin_is_GPL_compatible. When plugins are being loaded, the symbols in the resulting object file are examined, and before the plugin will actually be usable, GCC makes sure that there is a symbol called plugin_is_GPL_compatible. If this symbol exists, then your plugin is deemed to have the proper license.

    int plugin_is_GPL_compatible = 1;  

Next, let's define some metadata about the plugin; that is done by instantiating an instance of struct plugin_info in case the user asks for help (via gcc -v and/or gcc -v --help) with the plugin loaded. This is not required but can be helpful. Also, to make this effective, during initialization GCC must be handed this structure:

    static struct plugin_info myplugin_info =
    {
        .version = "42",
        .help = "Not yet...",
    };

Next, we will define an instance of plugin_gcc_version so that, during initialization, we can verify that our plugin and GCC can dance together nicely.

    static struct plugin_gcc_version myplugin_ver =
    {
        .basever = "4.6",
    };

Note that every field of this structure (basever, datestamp, devphase, revision, and configuration_arguments), all of which are strings, must match that of the version of GCC you are using when you let GCC determine if the version matches, e.g. via a call at initialization to plugin_default_version_check(). You can use your own version check method instead, which is ideal, as I doubt that your configuration options would exactly match those of your plugin's users. In short, do your own version checking unless you are bundling your plugin as part of a GCC release where the configuration options and datestamps, etc. will match.

If you are curious, the 'plugin-version.h' has the data for the basever, datestamp, devphase, revision, used when you built GCC (or when your distribution package maintainer built it). For our example, we will just validate against the base version string of GCC and will handle the version check ourselves. For our purposes we only care about any GCC version 4.6 and are ignoring the micro version number, e.g. the '0' in 4.6.0.

Plugin initialization

By now we have provided a bunch of info about the plugin, but there is one more piece of information that is required by GCC: the routine that actually initializes the plugin and registers callbacks to occur during the compilation of a program. This is another symbol the plugin framework will pluck out of the file, so it must be named plugin_init(). Our plugin is going to operate as a GIMPLE pass, allowing us to analyze (and transform if we wish) GCC's middle-end representation of the program being compiled. There are a couple of other options available. IPA_PASS is an interprocedural pass where the developer parses the control flow graph of the program and optimizes code that is dependent across functions, such as function inlining. The RTL_PASS is the last pass the compiler performs. This type of pass operates on a RTL (register transfer layout) representation of the program. RTL is what maps the gcc GIMPLE code into the register of a machine (machine description).

In order to do that, we need to define the proper struct and pass it information to the plugin framework, which will then call our callback handler once for each function in the source file. There are a number of other things that can be accomplished via plugins. These things are called "events" and are enumerated values in the plugin.def file. The .def files in GCC represent enumerations and are translated to source code when GCC is being compiled. These are then included into GCC's source for plugin.c:

    #include "plugin.def"

This makes using ctags kind of tricky, because it does not inspect .def files, nor does it know how to parse them. On the other hand, cscope works well if tell it to look for .def files.

Below is the data structure we use when our plugin is asked to initialize itself.

    /* See tree-pass.h for a list and descriptions for the fields of this struct */
    static struct gimple_opt_pass myplugin_pass = 
    {
        .pass.type = GIMPLE_PASS,
        .pass.name = "myplugin", /* For use in the dump file */
    
        /* Predicate (boolean) function that gets executed before your pass.  If the
         * return value is 'true' your pass gets executed, otherwise, the pass is
         * skipped.
         */
        .pass.gate = myplugin_gate,  /* always returns true, see full code */
        .pass.execute = myplugin_exec, /* Your pass handler/callback */
    };

Please note that there are many more options we can specify for our pass, but to keep things simple, we are just going to have a basic pass with a gate and execution callback. myplugin_exec() is the function that is called back given the conditions specified in the optimization pass we define below. Here are the pieces for all three parts of the optimization pass (the source code for the full plugin can be found here):

Now that we have a structure defining how we want our pass to act, let's finally write the required plugin_init() function:

    /* Return 0 on success or error code on failure */
    int plugin_init(struct plugin_name_args   *info,  /* Argument information */
                    struct plugin_gcc_version *ver)   /* Version info of GCC  */
    {
        /* 
         * Used to tell the plugin-framework about where we want to be called in the
         * set of all passes.  This is located in tree-pass.h
         */
        struct register_pass_info pass;
        printf("Plugin initialized...\n");
        
        /*
	 * We could call: plugin_default_version_check() to validate our plugin, but
         * we will skip that.  Instead, as mentioned it can be more useful if we
         * validate the version information ourselves
         */
         if (strncmp(ver->basever, myplugin_ver.basever, strlen("4.6")))
           return -1; /* Incorrect version of GCC */
    
        /*
	 * Setup the info to register with GCC telling when we want to be called and
         * to what GCC should call, when it's time to be called.
         */
        pass.pass = &myplugin_pass.pass;
    
        /*
	 * Get called after GCC has produced the SSA representation of the program.
         * After the first SSA pass.
         */
        pass.reference_pass_name = "ssa";
        pass.ref_pass_instance_number = 1;
        pass.pos_op = PASS_POS_INSERT_AFTER;
    
        /* Tell GCC we want to be called after the first SSA pass */
        register_callback("myplugin", PLUGIN_PASS_MANAGER_SETUP, NULL, &pass);
    
        /*
	 * Tell GCC some information about us... just for use in --help and
         * --version
         */
        register_callback("myplugin", PLUGIN_INFO, NULL, &myplugin_info);
       
        /* Successful initialization */ 
        return 0;
    }  

The above ties everything together. The first thing we need our plugin to do is to visit each statement in the source code that is being compiled and identify the read-only data. As an SSA pass, the compiler will hand our plugin GIMPLE, GCC's intermediate language, a set of basic blocks per function. Since all frontends (C, C++, Go, Fortran, etc) produce GIMPLE, gcc can effectively optimize just the GIMPLE code, rendering all optimizers capable of working on any language that GCC can parse. Likewise, when a developer writes a pass for the GIMPLE gcc intermediate language, it becomes language agnostic and can be applied to any language GCC parses. Using a three-address code, GIMPLE represents each statement in the input language as a statement consisting of two operands and a result value. In the case of an assignment statement with an addition operator, we have something like:

    lhs = op1 + op2

Where lhs, op1, and op2 are the three address codes that make up the assignment statement.

Read-only identification

There might be an easier way of finding read-only data in a program, however our approach here allows one to understand how to traverse basic blocks and statements in the program. We are more concerned with learning than functionality for this tutorial.

When a function gets passed to our plugin we need to do something with it, mainly analyze it and identify read-only data. The following is our callback we registered previously myplugin_exec():

   
    static unsigned myplugin_exec(void)
    {
       unsigned i;
       const_tree str, op;
       basic_block bb;
       gimple stmt;
       gimple_stmt_iterator gsi;

       FOR_EACH_BB(bb)
         for (gsi=gsi_start_bb(bb); !gsi_end_p(gsi); gsi_next(&gsi))
         {
             stmt = gsi_stmt(gsi);
             for (i=0; i<gimple_num_ops(stmt); ++i)
               if ((op = gimple_op(stmt, i)) && (str = is_str_cst(op)))
                 spell_check(stmt, str);
         }

       return 0;
    }

FOR_EACH_BB operates on a global variable in GCC which represents the current function being processed, cfun. The gsi is a GIMPLE statement iterator. In short, this loop traverses each basic block in cfun and visits each statement in that function. We look at each statement in the basic block via the gsi interface. Finally we look at each operand that makes up the GIMPLE statement. The is_str_cst() is a predicate function that we will define below. This function determines if the operand we plucked from the statement represents a string constant. If it does we will spell check that string.

In GCC, every construct in the source code is essentially a tree. A tree node could represent constants, variables, functions, etc. Often these tree instances are wrapped multiple times. For instance the tree node might be encapsulated in an SSA_NAME instance for the SSA representation. Under that might be a POINTER_TYPE and under that an INTEGER_CST for a constant integer. Note that these node types are defined in tree.def. Helper functions in tree.h can help to identify and peel off these layers, such as with TREE_CODE() and TREE_OPERAND() respectively. Also, TREE_TYPE() is useful for peeling off the layers. The debug_* functions can help better understand this layering, particularly debug_tree(). The following routine is what we will start off with for identifying STRING_CST:

    static const_tree is_str_cst(const_tree node)
    {
       const_tree str = node;

       /* Filter out types we are ignoring */
       if (TREE_CODE(str) == VAR_DECL)
       {
           if (!(str = DECL_INITIAL(node))) /* nop expr  */
             return NULL_TREE;
           else if (TREE_CODE(str) == INTEGER_CST) /* Ignore single chars */
             return NULL_TREE;

           str = TREE_OPERAND(str, 0); /* addr expr */
       }
       else if (TREE_CODE(str) == ADDR_EXPR)
           str = TREE_OPERAND(str, 0);

       /* We only deal with readonly stuff */
       if (!TYPE_READONLY(str) && (TREE_CODE(str) != ARRAY_REF))
           return NULL_TREE;
      
       if (TREE_CODE(str) != STRING_CST) 
           str = TREE_OPERAND(str, 0);

       if (TREE_CODE(str) != STRING_CST)
           return NULL_TREE;
       else
           return str;
    }

As you can see, we detect the type of node to be processed. If we are initially presented with a declaration node (which one might get if the node were a parameter instance) we look at the declaration of it via DECL_INITIAL(). If we get an address, we peel off that variant and look at what it is an address expression of. If the node isn't read-only we assume that it's probably not a hard-coded string so we just ignore it. There is probably a better way of handling this, such as just placing the node in a loop around operand checks, but I'll leave other methods of peeling the types as an exercise for the reader.

Conclusion

Comments (8 posted)

Brief items

Quotes of the week

Comments (none posted)

Banshee 2.2 released

Full Story (comments: 55)

New versions of Firefox are available

Full Story (comments: 6)

GNOME 3.2 released

Full Story (comments: 76)

GTK+ 3.2.0

Full Story (comments: none)

Mozilla ponders long(er)-term support releases

Full Story (comments: 23)

PacketFence 3.0 released

Full Story (comments: 2)

PulseAudio 1.0 released

Full Story (comments: 4)

PyTables 2.3 released

Full Story (comments: none)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (September 27)
• PostgreSQL Weekly News (September 25)

Comments (none posted)

Richard Hughes on color management in Linux and GNOME (Libre Graphics World)

Comments (23 posted)

Poettering: systemd for Administrators, Part X

Comments (3 posted)

Poettering: systemd for Administrators, Part XI

Comments (26 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

MeeGo + LiMo = Tizen

Comments (59 posted)

Articles of interest

FSFE: Fellowship Interview with Stefan Kangas

Comments (none posted)

Upcoming Events

Linux Users' Group of Davis, October 17 meeting

Full Story (comments: none)

Events: October 6, 2011 to December 5, 2011

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol