"Unstable kernel APIs" vs. the embedded reality :-(

Seriously, it's a non-trivial thing to do, lots of people and companies have tried to do so over the decades in operating system development, and no one has been able to do so for a general-purpose operating system that supported more than just a handful of hardware devices/types.

Remember if your in-kernel api is not changing over time, your operating system is dead :)

People forget that change happens because it has to, given the environment or requirements change over time, it's not done just because it is fun to do so. Well, not usually...

Anyway, take a look at how the enterprise distro kernels do this type of thing, they define a subset of all in-kernel apis and work hard to keep that stable over time. That works good for their target market, and is also something that embedded people should copy.

But, to get back to the main topic of this article, I fail to see how a stable api would solve any of the issues Neil had here, what ones do you think it would have helped resolve?

No API is really stable - we add things to the user->kernel API all the time. You might have noticed that a new event notification API was proposed recently. If we add things, then the API isn't really "Stable", it is "Changing".
Internally, we tend to manage this by providing shims to support the legacy API. For example, file change notification has a single internal implementation with three shims: dnotify, inotify, fanotify. You see this sort of pattern of various shims over an internal implementation a lot in the kernel.
This works well for the user->kernel interface because there are lots of applications making calls into one kernel, so the one kernel can easily support multiple APIs. For the kernel->driver interface the situation is a bit different. One kernel is calling lots of drivers.
To manage that, the kernel would need to detect what API the driver expects, and adjust its calling style accordingly. We do do that to some extent. The stand-out example is my mind is "unlocked_ioctl" in 'struct file_operations'. There used to be a 'ioctl' interface. The BKL would be held while that was called. Part of removing the BKL was to handle ioctls without it. So unlocked_ioctl was added and for a time we supported drivers with the new interface and drivers with the legacy interface, by detecting which of the two functions was not NULL. This is a thing that happens quite a bit, but it certainly isn't universal. And we do get rid of the legacy support because we find it adds no value, and does entail a cost.
It would have made my job easier if every time an internal API changed, the name changed (like ioctl -> unlocked_ioctl) so that old drivers wouldn't compile. When an old driver doesn't compile it is easy to find out why, find the commit that introduced the change, read the explanation, and fix the code. I might have done while while forward-porting the mt7621 drivers, but it would have been so "normal" that I wouldn't bother to remember.
The problem is when the API changes and the code still compiles. I think we could do better here, but I'm not sure that we are all that bad. In some cases it might be that the driver was misusing the API in some way. After all, we don't test against a generic API, we test against a particular kernel. If the kernel starts using an API in a new way, a driver might break even though the API didn't change. The only real solution to making drivers work with new kernels is regular testing.

If legacy driver-API support was really important to someone I suspect they could create a shim which exports exactly that API. Then they can keep the shim up-to-date as the kernel changes. Probably the most difficult part is creating a good API definition (lwn readers know that we suck at designing new APIs) and ensuring drivers adhere to the intent of the API. I think that is a lot of work, but maybe it would be worth it for someone.
We in the kernel community think it isn't worth it. We would rather keep all drivers in the common tree, and update all API users when we update an API. And we try to make this easy. If you have out-of-tree code, then *please* put it in drivers/staging. The barrier to entry is low and we will do our best to keep the API up-to-date, and may well clean up the code and make it maintainable as well. Free service!
I actually wonder if the swconfig Ethernet driver I mentioned should go into drivers/staging. Davem doesn't want it, but maybe gregkh would take it. Then we can fix it up in the open and convert to switchdev. Then it might be ready for davem.
I think my next sub-project with the gnubee will be to try to get all of the drivers into staging before looking to clean them up. See if it actually works :-)

A much better solution would be to simply not buy hardware that does not have in-tree support. I realize that is easier said than done with this type of hardware but is, AFAIK, possible.

I even hope to someday follow my own advice WRT to GPUs. Sigh.