> I dont mean that we should stop changing these APIs. But we should do so over larger timeframes and with a clear roadmap.
The 2.4 kernel series tried to do what you propose: Many large, invasive patches were put off until later. Developers hand to carry their patches around for years. Then, when 'later' came, there was much pain trying to stabilize many massive changes at once.
So during 2.6, they decided to optimize for kernel developers, not anonymous 3rd parties who aren't maintaining the kernel. Patches were integrated when they were ready, instead of some artificial deadline. The USB API was rewritten to make drivers easier to write. The network API was rewritten to allow switching interrupts to polling during high loads (saving a lot of CPU when it's needed most.) A suspend/resume API was globally added to every driver. New APIs are created to better abstract memory management (especially under memory pressure). etc, etc, etc.
Huge swaths of the kernel are rewritten constantly, including internal "driver APIs".
> rewrite large stacks without any care for the consequences on out-of-tree drivers.
Be careful what you wish for. This is what economists call an externality.
The kernel developers want to refactor kernel code globally (without worrying about making all data structures and APIs back-compatible, which would take a lot of extra work.)
The 3rd parties (with outside code) want a stable API so they can do less work of trying to keep up with the changes in the kernel. The 3rd parties wouldn't pay any of the costs, since they (generally) aren't core kernel maintainers. The 3rd parties would literally choose to slow kernel development in order to make their lives easier.
Luckily, 3rd parties don't get to make the choice, the kernel developers do.