Build breakage in linux-next

Posted Dec 15, 2025 13:32 UTC (Mon) by error27 (subscriber, #8346)
Parent article: Best practices for linux-next

Hardware companies should really be testing against linux-next. With the new EU CRA, everyone is going to have to upgrade their kernel a couple times. If you test linux-next, that's the earliest place to prevent breakage. I can understand Ted saying linux-next is too unstable for file system testing, but for testing boards it works fine. Everything in linux-next is being actively worked on, but by the time it reaches mainline a month later then the discussion is already stale.

Most of the problems in linux-next are build errors on less used architectures.

It sounds like complaining, but it's just objectively true that the number one cause of build breakage on x86 and arm was Rust. The Rust people should get a bunch of qemu images with common distributions and test for build breakage. Some times it's unavoidable that you have to upgrade your Rust packages but other times it was a kernel issue.

The other annoying thing is when you find a bug and the developer responds "Oops! I will fix it!" And you think they mean they will fix it the next day, but really they mean in a week. They should drop the patch immediately if it can't be fixed that same day.

to post comments

Build breakage in linux-next

Posted Dec 15, 2025 14:54 UTC (Mon) by pizza (subscriber, #46) [Link] (2 responses)

> Hardware companies should really be testing against linux-next. With the new EU CRA, everyone is going to have to upgrade their kernel a couple times. If you test linux-next, that's the earliest place to prevent breakage.

No, as the article repeatedly states, linux-next is a horrible place for preventing arbitrary breakage, as you don't know if it's your code or someone else's that's responsible, and if something you care about breaks today on -next, tomorrow (or next week) it may just work with no action on your part.

Hardware companies (as in those shipping devices with Linux on it) are generally best off sticking with the latest LTS kernel at the time the product development was ongoing, because that gives them a reasonably static target. If their device lifecycle is longer than the LTS lifecycle, then they may need to rebase their work against a newer LTS kernel every couple of years.

Now if they wish to try to get stuff upstream that's going to require dealing with -next and random breakages that come from that as you independently work to get your various local changes into the appropriate subsystems, but nobody has the resources to rebase everything and then meaningfully QA the entire product every time -next changes. Remember, it's not enough to merely *compile*; you have to actually *boot* and *test* it!

I suppose it's worth distinguishing between "hardware == retail product" and "hardware == a peripheral/component"; Sticking with LTS makes the most sense for the former, but getting stuff upstream (and thus, dealing with -next) is a lot more relevant for the latter.

Build breakage in linux-next

Posted Dec 15, 2025 19:52 UTC (Mon) by error27 (subscriber, #8346) [Link]

I've been doing linux kernel testing on next, mainline, and LTS for the past couple years as part of my work. Bugs are easier to fix when you find them in linux-next.

Obviously, no one develops new drivers against linux-next. Kernel developers who work upstream just download an -rc kernel and work against that. There are people who don't care about upstream and they use LTS kernels, but I think it's going to be more and more difficult to ignore upstream with laws like the CRA. Products based on Android will need to go through several major kernel upgrades during their ten year life time. The more out of tree code, the more complicated each kernel upgrade becomes.

But I wasn't really talking about development, I was talking about hardware which are already supported by upstream and the vendor want to make sure it keeps working. The best tree to test against is linux-next. If that's not possible then you still want to be as close as reasonable to upstream.

Android has a tree which is updated within a couple weeks of when Linus releases a kernel. This is a good model. People think they are saving time by only upgrading once every other year instead of every eight weeks, but it's actually harder. When you upgrade regularly, then that's a smaller upgrade. You learn to be good at it. Also since everyone else is upgrading there is a built in support system if you have questions. But if you upgrade randomly after two years, no one will remember the old API or how it changed.

Build breakage in linux-next

Posted Dec 16, 2025 16:55 UTC (Tue) by broonie (subscriber, #7078) [Link]

It depends a lot what the objective of your testing is - shipping a product on -next is obviously silly, but if what you're doing is trying to ensure that what comes down the pipeline to your production systems is in reasonable shape then -next is a great place to do testing, you'll tend to catch things much closer to when they're introduced and therefore find it a lot easier to get things looked into. There's limits on what you'd want to test but it's valuable to pay attention. Similarly if what you're testing is functionality you use as opposed to functionality you write it doesn't really matter who introduced issues, you just want to find them and fix them.

For example I run testing of -next not so much because it's the thing I directly want to use but more because each merge window it turns into the baseline I'm testing against so if any of the boards or tests I rely on break then I'd much rather catch that while it's not impacting the testing I'm actually trying to do.