User: Password:
Subscribe / Log in / New account

Two ABI troubles

Please consider subscribing to LWN

Subscriptions are the lifeblood of If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

By Jonathan Corbet
October 5, 2010
It has long been accepted by kernel developers that the user-space ABI cannot be broken in most situations. But what happens if the current ABI is a mistake, or if blocking changes risks stopping kernel development altogether? Both of those possibilities have been raised in recent discussions.

The capi driver provides a control interface for ISDN adapters - some of which, apparently, are still in use somewhere out there. If the devices.txt file is to be believed, the control device for CAPI applications should be /dev/capi20, while the first actual application shows up as /dev/capi20.00. That is not what the applications apparently want to see, though, so Marc-Andre Dahlhaus posted a patch moving the application devices under their own directory. In other words, the first CAPI application would show up as /dev/capi/0. The patch also modified the devices.txt file to match the new naming.

Alan Cox rejected the patch, saying:

devices.txt is the specification, and its ABI.

It is fixed and the kernel behaviour is to follow it. Those who didn't follow it, or who didn't propose a change back when it was specified in the first place have only themselves to blame. It isn't changing, and the ISDN code should follow the spec.

Maintaining the ABI is normally the right thing, but there are a couple of problems with the reasoning here. First is that, apparently, few (if any) distributions follow the rules described in devices.txt; the real ABI, in practice, may be different. Second: the kernel doesn't follow devices.txt either: current practice is to create /dev/capi as the control device, and /dev/capi0 as the first application device. The capifs virtual filesystem covered over some of this, but capifs is on its way out of the kernel.

In the short term, the fix appears to redefine the current behavior as a typo, tweaking things just enough that udev is able to create the right file names. The devices.txt file will not be touched for now. If regressions turn up, though, it may become necessary to support alternative names for these devices for well into the future.

Tracepoints, again

Jean Pihet recently posted a set of tracepoint changes for power-related events. The patch added some new tracepoints, added information to others, and added some documentation as well. Even more recently, Thomas Renninger came forward with a different set of power tracepoint changes, meant to clean things up and make the tracepoints more applicable to ARM systems. In both cases, Arjan van de Ven opposed the patches, claiming that they are an ABI break.

The ABI in question does have users - tools like powertop and pytimechart in particular. It seems that Intel also has "internal tools" which would be affected by this change. As Arjan put it: "the thing with ABIs is that you don't know how many users you have." When things are expressed this way, it looks like a standard case of a user-space ABI which must be preserved, but not all developers see it that way.

Peter Zijlstra argues that tools using tracepoints need to be more flexible:

These tools should be smart enough to look up the tracepoint name, fail it its not available, read the tracepoint format, again fail if not compatible. I really object to treating tracepoints as ABI and being tied to any implementation details due to that.

Steven Rostedt worries about the effects of a tracepoint ABI on kernel development:

Once we start saying that a tracepoint is a fixed abi, we just stopped innovation of the kernel. Tracepoints are too intrusive to guarantee their stability. Tools that need to get information from a tracepoint should either be bound to a given kernel, or have a easy way to update the tool (config file or script) that can cope with a change.

The issue of ABI status for tracepoints has come up in the past, but it has never really been resolved. In other situations, Linus has said that any kernel interface which is taken up by applications becomes part of the ABI whether that status was intended or not. From this point of view, it is not a matter of "saying" that there is an ABI here or not; applications are using the tracepoints, so the damage has already been done. Given that user-space developers are being pushed to use tracepoints in various situations, it makes sense to offer those developers a stable interface.

On the other hand, it is very much true that these tracepoints hook deeply into the kernel. If they truly cannot be changed, then either (1) changes in the kernel itself will be severely restricted, or (2) we will start to accumulate backward-compatibility tracepoints which are increasingly unrelated to anything that the kernel is actually doing. Neither of these outcomes is conducive to the rapid evolution of the kernel in the coming years.

If nothing else, if tracepoints are deemed to be part of the user-space ABI, there will be strong resistance to the addition of any more of them to large parts of the kernel.

Some alternatives have been discussed; the old idea of marking specific tracepoints as being stable came back again. Frank Eigler suggested the creation of a compatibility module which could attach to tracepoints which have been changed, remapping the trace data into the older format for user space. There has also been talk of creating a mapping layer in user space. But none of these ideas have actually been put into the mainline kernel.

This issue is clearly not going to go away; it can only get worse as more application developers start to make use of the tracepoints which are being added to the kernel. It seems like an obvious topic to discuss at the 2010 Kernel Summit, scheduled for the beginning of November. What the outcome of that discussion might be is hard to predict, but, with luck, it will at least provide some sort of clarity on this issue.

(Log in to post comments)

Two ABI troubles

Posted Oct 11, 2010 19:38 UTC (Mon) by NAR (subscriber, #1313) [Link]

There are about 4 kernel releases every year. Even the fastest moving distributions have no more than 2 releases per year - on average every other kernel doesn't even end up in a distribution used by many people. Why bother with backwards compatibility for kernel that are only used by gurus? They know what they are doing, don't they?

I think that the tracepoints should be fixed for those kernels that are expected to go into systems that will actually work for a long time. If these tools are low level (e.g. like udev), they have to be updated with each new distribution release anyway, so they can be updated to handle the new tracepoints. I mean the Linux distribution model (in practice) severly limits the compatible versions of applications anyway, so it would be not such a big extra burden.

I'm way too sleepy to write a better thought out comment, but you might understand my point.

Two ABI troubles

Posted Oct 14, 2010 12:05 UTC (Thu) by Duncan (guest, #6647) [Link]

> Even the fastest moving distributions
> have no more than 2 releases per year


The fastest moving distributions are likely to be rolling distributions, simply because six-month static release cycles is actually way too slow for a "fast moving" distribution. Consider KDE, for instance, which has monthly minor releases, and I've worked on projects that during sprints often have weekly releases. There's simply no way a six-month-cycle even /can/ keep up with that. But rolling releases can. =:^) (Tho even they tend not to keep up so well with sub-monthly releases, except possibly in lightly- or un-tested optional repos.)

For such rolling release distributions, depending on whether you count each possible combination of packages as a release (thus, a new "release" every time a package update is posted, typically several times a day, so say 1000+ releases per year), or whether you count each (typically automated) initial installation snapshot (weekly, for distributions like Gentoo, tho there are times when the automated snapshots break for a few weeks at a time, so say something /close/ to 50 releases a year counting this way), release frequency WILL vary, but by most definitions, such distributions release MUCH more frequently than the ~3 month kernel cycle, thus having several to many "releases" per kernel cycle.

Just sayin'. If you're going to talk about fast moving distributions, don't by definition disqualify the ones that /are/ fast moving, which tend to be rolling distributions, because that's really the only ones that can keep up with the model. In comparison to such distributions, your six-month "fast" distributions are more like glacial timescale, while distributions like Debian are more like geologic. No offense intended, just something to think about: The definitional bias just demonstrated rather reminds me of products that talk about supporting multiple OSs, and then only offer various MS Windows versions. =:^(

Two ABI troubles

Posted Oct 15, 2010 22:15 UTC (Fri) by oak (guest, #2786) [Link]

Normally APIs & ABIs are managed and they have their own life-times. More stable APIs/ABIs have longer life-time (with guaranteed minimum) and they need to be deprecated longer (with guaranteed minimum) before being replaced/removed.

So state the life-time expectancy upfront. This ABI is there for a year, then it will change (and if you're lucky, not).

The user-space ABI can also be stated as a future proof "Attaching to a given tracepoint WILL fail. If it succeeds, you get following data: ...".
This way they can be also in modules that may or may not be present depending on the kernel configuration.

If data can also change, type of data needs to be part of the tracepoint identifier.

Copyright © 2010, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds