> Many large-scale open source projects have found elegant solutions to similar problems
Most plug-in APIs (like browser plug-ins) have a small 'surface area'. This is not true with kernel drivers. There is a massive number of internal APIs/datastructures. Trying to keep back-compatibility would slow development.
For example, if the kernel would be faster if we eliminated the "jiffies" variable, should we delay that change because someone somewhere _may_ have a proprietary driver that _might_ use it?
> They let several APIs coexist for reasonable timeframes
Either you refactor the API or you don't. If your code is required to be back-compatible for some time, then you can't _really_ refactor your code. You can only "create a new API, shuffle things around a little bit." You are in a straight jacket and prevented from making radical simplifying changes.
> forcing people to migrate to new APIs after 1 or 2 years.
That implies forcing people to keep their radical patches for 1 or 2 years.
> Im not asking for refactorings to wait for 3 years.
Well, you just said 2. Even 1 year is too long in my book.
> I fail to see in what way the kernel would need to be different.
If an internal API is deemed to be defective, you are saying it would have to be supported for years. Today, it is simply deleted and rewritten, so the defective API can no longer cause problems. Much simpler, no maintenance burden. (For examples, see read the excellent articles here on how the RCU API has changed thru the years.)
You're asking the kernel developers to take on a *lot* more maintenance burden. But what do they get in return?