2.6 and the user-space ABI [LWN.net]

The 2.6.22-rc1 kernel is out, and the reports of regressions are beginning to trickle in. A couple of those involve user-space binary interface changes: one in the video4linux2 interface and one in the i2c code (which affects hardware monitoring utilities). The V4L2 regression involves a change made to a structure passed to and from user space; chances are good that it will be reverted before the final 2.6.22 kernel comes out. For the time being, the i2c problem is "fixed" by upgrading to version 2.10.3 of the lm_sensors package.

Linus isn't happy about the forced lm_sensors upgrade; he has asked for a way to avoid that requirement. In response, i2c maintainer Jean Delvare raised some misgivings about the stable ABI policy:

While I'm all for keeping things relatively stable and not asking the user to constantly upgrade user-space, I believe that we just can't promise to never break user-level interfaces while keeping the development pace we have right now. We can promise to grant people significant delay before we drop compatibility options, but "forever" doesn't scale.

Those comments notwithstanding, Linux has managed to maintain user-space ABI compatibility quite nicely for many years. There are certainly exceptions, but they are few enough and far enough between that each one stands out. But, as Christoph Hellwig points out, the situation is not perfect:

Except for very rare case (modules support comes to mine) syscall compatibility works perfectly. But that's because syscalls are a very visible ABI and people don't break them by accident. They also don't decide they have a cool new scheme all syscalls need to follow now.

Now compare that to sysfs..

The user-space ABI now goes well beyond system calls. The huge sysfs interface (4800 files on your editor's desktop) is a big piece of user space's view of the system, and it is a piece which is difficult to avoid breaking. Directories in sysfs correspond directly to data structures within the kernel; changes to those structures will often have consequences in sysfs. So kernel developers may think that they are operating far away from the user-space interface, but end up breaking it anyway. Netlink, /proc, and ioctl() also make up part of the ABI, and they, too, can be easy to break. The V4L2 regression is the result of an attempt to extend one ioctl() call breaking others.

The new development model can also make it harder to maintain compatibility. Four or five major releases per year, each with a full load of new major features, adds up to a lot of code changes. There is also no clear point where whatever changes do prove to be unavoidable can be made without surprising users. If the kernel developers were to disappear for a year or two and return with a 3.0 release, nobody would be surprised if it required a small amount of adaptation on the user-space side. But a 2.6.22 release - which contains needed fixes and new drivers along with new features - is not expected to break working systems.

Arguments for returning to the older development model are hard to find, though. Despite occasional glitches, things are generally working far better than they did before 2.6 came out. The pace of development is unlikely to slow. So the problem of occasional ABI regressions is likely to remain with us. As is often the case, the best way to avoid such problems - after a high degree of attention by the developers - is extensive testing. User-space ABI changes caught during the development cycle will almost certainly not survive into the final release, but it is hard to fix problems that nobody knows about. As is also often the case, automating this testing is hard; nobody can put together all of the hardware and software combinations that the kernel will face. So the worthy cause of maintaining a stable user-space interface is likely to require a fair amount of human attention for the foreseeable future.

(Log in to post comments)

2.6 and the user-space ABI

Posted May 17, 2007 7:27 UTC (Thu) by tjasper (subscriber, #4310) [Link]

So is there a case now for alternate releases which do allow some user space breakage every other release, and maintain stability with no allowed breakage in userspace in the intervening releases? So the current model of releasing is maintained, but more churn in userspace-visible interfaces is do-able. Would that allow for more flexibility? Users/distributors would know which are the more stable kernels and give a full second release cycle to stabilise before upgrading.

Does the kernel need to break these things occasionally to reduce cruft/overhead/clever workarounds/whatever simply to avoid breakage?

Trevor

2.6 and the user-space ABI

Posted May 17, 2007 9:29 UTC (Thu) by sdalley (subscriber, #18550) [Link]

What we surely need is a userspace kernel-lib that is part of the kernel tree and released as a matter of routine with each kernel update. The current udev and alsa libraries could go in there.

Then also, whenever a sysfs interface had to change, would be the time to step back and ask "what would an *application* need to see of this service?" (lm_sensors or whatever) and add the new higher-level hindsight-improved interface to kernel-ulib, or whatever it ends up being called. Move the sysfs interface for that service from "provisional" to "deprecated".

We could then buy ourselves flexibility without sacrificing quality.

a userspace kernel-lib that is part of the kernel tree

Posted May 18, 2007 8:21 UTC (Fri) by xoddam (subscriber, #2322) [Link]

Yes yes yes yes yes.

Hear hear.

I agree with the above post.

What (s)he said.

2.6 and the user-space ABI

Posted May 18, 2007 12:55 UTC (Fri) by michaeljt (subscriber, #39183) [Link]

An intermediary solution would be to strongly suggest that people who need to access kernel ABIs from user space applications should look for existing libraries to do this and only do it through those libraries. If no suitable libraries were found, the functionality could be split off into a new library which could be published separately from the application, so that other applications could use that. That would limit the number of places where ABI changes would need to be fixed. Someone could even keep a database of known libraries which accessed the kernel ABI directly. Note that even syscalls are almost always accessed through libc.

2.6 and the user-space ABI

Posted May 21, 2007 3:19 UTC (Mon) by pimlott (guest, #1535) [Link]

Wait--that would be admitting that the filesystem isn't the single best interface for everything (the rationale for sysfs in the first place)! Heresy!

2.6 and the user-space ABI

Posted May 26, 2007 15:08 UTC (Sat) by anton (guest, #25547) [Link]

Yes. However, the number of incompatible changes could be reduced even more by out-of-kernel-tree stuff never accessing sysfs directly, but always through the kernel-lib. If such an application want something that is available through sysfs, but not the kernel-lib, then one could add an appropriate ABI to the kernel-lib, and that ABI is probably easier to keep stable than sysfs.

User-space ABI / Insulating the user

Posted May 17, 2007 16:18 UTC (Thu) by filker0 (guest, #31278) [Link]

Long before Linux was born, I worked on a small embedded event-driven multi-tasking kernel for an advanced development project on a multi-processor M68k platform. Churn in the ABI became a problem as the hardware and kernel maturatured as the other subsystem teams would move from one subproject to another.

We addressed this problem by adding a thin layer of sand and drawing a line in it. All code written to use an ABI that was written to a newer "stable" base-level would make a system call (we used traps) that identified the version of the ABI it expected. All of the old tasks that did not use a newer ABI would not make the call, and would be assumed to be of the oldest ABI tracked. If the ABI associated with a task was older than the current ABI, an emulation layer that logged the obsolete usage, but still serviced the task by emulating the ABI in question (we used a jump table for this, and had one for each "supported" ABI built in to the kernel). For development, this worked pretty well. Once all of the required functionality was implemented, we used the logs to locate all of the places where the old ABI was used and fix them, then we turned this feature off for the later testing and production builds. We actually left the select ABI system call in place (a no-op for production builds) in each of the tasks so that this could be used in future upgrade cycles.

This cannot be applied to Linux user-space as easily as it was to a small-ish project with under 15 engineers working on it, and I see no way for sysfs changes to be shielded this way (though changes in data structures within sysfs might, with a *lot* of work, be automatically translated in some cases). If a new system call was added to the ABI that registered some base-level compatibility (could be subdivided down to the specific feature or driver level), a table in the TCB could be used to determine how to handle the call. A set of hooks to log deprecated ABI usage could be added so that developers could isolate the ABI changes and get the maintainers of packages that depend on the older ABI to fix them.

I'm not suggesting that this should be done, just that there is a way to design for forward migration in the ABI. There are performance issues with this approach, and it may be harder to do with the context switch mechanism in Linux than it was with the trap instruction based context switch that we were using.