Testing in the Yocto Project
Testing in the Yocto Project
Posted May 18, 2019 16:39 UTC (Sat) by dvdeug (guest, #10998)Parent article: Testing in the Yocto Project
Posted May 18, 2019 18:20 UTC (Sat)
by mtaht (subscriber, #11087)
[Link] (7 responses)
I think having a diverse automated testbase so one group with an automated test suite could get verified by another group with a different automated testsuite, could possibly evolve out of all this. Ultimately we'll end up with AI's stacked upon AIs competing to keep the computing environment clean...
... duking it out with AIs stacked upon AIs competing to find holes and exploit them before the other AIs do... and other AIs ripping out the work of other AIs to find a place for themselves to execute....
... and befuddled humans as the bystanders in that rapidly escalating war hoping that no AI notices that the AIs don't actually need to keep the humans alive and around in order to thrive in the complexity collapse.
I just finished reading a pretty decent new SF book out about this sort of stuff: https://www.amazon.com/Genesis-Geoffrey-Carr-ebook/dp/B07...
Posted May 19, 2019 2:54 UTC (Sun)
by roc (subscriber, #30627)
[Link]
Why would you need that? I can't think of any project that requires a bug to be confirmed in two independent test suites before investigating it. I don't even know how you would know it was the same bug.
> Ultimately we'll end up with AI's stacked upon AIs competing to keep the computing environment clean...
Maybe, but the kernel community is still far behind what other big projects consider basic testing requirements. (And those basics don't require AI.)
Posted May 19, 2019 4:19 UTC (Sun)
by dvdeug (guest, #10998)
[Link] (5 responses)
Posted May 19, 2019 9:20 UTC (Sun)
by knuto (subscriber, #96401)
[Link] (4 responses)
Posted May 19, 2019 10:37 UTC (Sun)
by roc (subscriber, #30627)
[Link] (3 responses)
> That way the author of a change keeps the burden of proof for the change, instead of the test owner.
That's a good point, but if you're a kernel developer and you can keep the burden of testing, triaging, diagnosing and reporting bugs on other people, it makes sense to carry on with that system. Even if in many cases you're shifting work from paid kernel developers to unpaid open source project maintainers. Even if it means you're forcing downstream consumers to duplicate tons of work maintaining multiple test infrastructures and running the same sorts of tests, or even exactly the same tests. Even if it strangles the feedback loop between detecting and fixing regressions.
Posted May 20, 2019 9:16 UTC (Mon)
by metan (subscriber, #74107)
[Link] (2 responses)
Well in SUSE we do run LTP compiled with -m32 on x86-64 which catches bugs in the compat layer from time to time, but the coverage of ioctls() is sparse at best, we do have some tests for /dev/tty*, /dev/net/tun, block device ioctls, /dev/random, namespace ioctls and some Btrfs ones and that's it.
Posted May 20, 2019 13:14 UTC (Mon)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted May 20, 2019 13:21 UTC (Mon)
by metan (subscriber, #74107)
[Link]
Posted May 20, 2019 15:35 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (20 responses)
Posted May 20, 2019 16:04 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
See this bug report to git: https://marc.info/?t=151242550100001&r=1&w=2 And the followup patch after just one day: https://marc.info/?t=151251216400002&r=1&w=2. Could I have come up with that `read_cache()` line? Probably, but not in that timescale. The commit message would also certainly have been far inferior.
Posted May 20, 2019 17:35 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link]
Apart from that, paraphrasing a famous quote, "the kernel is just a program".
Posted May 20, 2019 18:14 UTC (Mon)
by dvdeug (guest, #10998)
[Link]
That philosophy works great for code that somebody threw up on GitHub because someone else might find it useful. But if maintainers get dismayed that a feature is being dismissed as broken, with no bug reports, or that some group is running an ancient version of the kernel, this is why. The fact that updating the kernel could cause random problems, and even dedicated searching for the bug, where you can provide a reproducable test case and the commit that's causing the problem, is not going to help, is a reason to lock the kernel to one version or use a local patch if one can figure out what's wrong.
I see the problem. But if you want to make a program that's used by a wide audience, sometimes you have to respond to the needs of that wide audience, and when they can give you a test that's failing and even a commit that made the change, perhaps you should take a look and fix it. Otherwise, people will fork or stay on older versions or switch to another program that does the same thing. As you say, the kernel is just a program, and if Linux's rapid changes are a downside, no other Unix kernel is changing nearly as fast.
(I'd point out that there's not this attitude inside the kernel community; if a commit won't bootstrap on ARM, it's not the problem of the ARM community, that's the problem of the commit submitter. There's a good argument that a problem like this doesn't need time to work out what's going on; once a particular commit was known to be causing a userspace problematic change, just rollback the commit and if the submitter of the commit wants the change, they get to make it work right or explain why the problem is not a kernel bug.)
Posted May 20, 2019 20:51 UTC (Mon)
by roc (subscriber, #30627)
[Link] (16 responses)
The Linux kernel is special. Linux is so dominant that downstream users are more or less stuck with it, so kernel development is structured to offload as many costs as possible to downstream. Even though it's a terribly inefficient setup and painful for downstream, downstream users have no leverage to alter the deal.
This will continue to make sense until Linux has serious competition, and then it won't.
Posted May 20, 2019 21:33 UTC (Mon)
by rweikusat2 (subscriber, #117920)
[Link] (13 responses)
If *you* think this is terrible, you're perfectly entitled to that but other people who are also "downstream users" might disagree. The kernel is really just a program and as such, it can be made to dance to whatever tune someone currently prefers. It's even a pretty organized program, hence, changing it is reasonably easy compared to other open source software (I happen to have encountered).
Posted May 20, 2019 23:16 UTC (Mon)
by roc (subscriber, #30627)
[Link] (12 responses)
Imagine if Firefox and Chrome didn't do their own testing, and instead Linux distros, device vendors and Web developers felt obliged to set up independent browser-testing projects with duplicated but not always identical tests, their own triage and false-positive screening, various different channels for reporting test failures upstream (many of which are ignored), etc. It would obviously be insane. Yet the kernel does this and everyone seems to accept it.
Posted May 20, 2019 23:30 UTC (Mon)
by corbet (editor, #1)
[Link] (6 responses)
The problem with the kernel is that so many of the problems people run into are workload and hardware dependent. Developers increasingly test for the things they can test, but they can never test all of the available hardware, and they can never test that your workload, in particular will not regress. So, while the project still very much needs to improve its act in this regard, I'm not sure I see an alternative to a lot of downstream testing anytime soon.
(And yes, a better answer to bug tracking is sorely needed, but there are many ghetto areas in kernel development that nobody wants to pay for, and that is one of them).
Posted May 21, 2019 0:06 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted May 21, 2019 0:20 UTC (Tue)
by roc (subscriber, #30627)
[Link]
The Linux Foundation seems to have taken on the role of trying to ensure that organizations that profit from Linux put money back into it. I'm not privy to the workings of LF so I don't know whether they can take this on. Still, if there was a consensus in the kernel community that central testing infrastructure is needed and we just have to figure out how to pay for it, that would be a big step forward.
One thing that might help: the kernel community could try to normalize the expectation that writing tests, building better test frameworks, and investigating failed tests is something every kernel developer needs to do (including the expectation that every bug fix and every new feature comes with automated tests). This is normal for a lot of other projects, from tiny no-money projects to large big-money projects. Then the companies paying people to do kernel development would effectively be taxed to support the testing effort.
> The problem with the kernel is that so many of the problems people run into are workload and hardware dependent.
The kernel workload is highly variable. Then again, for browsers your workload is *every Web page in the world*, from gigantic static pages to GMail to twitchy 3D shooters and everything in between. Mostly written in obfuscated code by tens of millions of developers with ... a range of skill levels ... whom you mostly can't talk to. Browsers are sensitive to hardware variation too, though less so than the kernel, sure. But desktop applications have their own special challenges: myriad combinations of software that your software depends on and interacts with, including ultra-invasive stuff like antivirus and malware that patches your code, all changing underneath you day to day, almost entirely closed-source and often controlled by your competitors. Mobile has similar issues.
I talk a lot about browsers because that's what I know. I don't think kernels and browsers are uniquely difficult.
I get that testing the kernel is hard. That's why greater cooperation and more efficient use of testing resources is so important.
> I'm not sure I see an alternative to a lot of downstream testing anytime soon.
Sure but so far I don't even see that the kernel community sees lack of upstream testing as a problem that needs to be solved.
Look at it this way: kernel people emphasize how important it is for people to upstream their code --- "upstream first", etc. That reduces duplication of effort, ensures things stay in sync, etc. But for some reason* they don't apply the same logic to tests. Tests, test harnesses, and test infrastructure should be upstream-first too for exactly the same reasons. Yes, there will always be stuff that is hard to upstream.
* Come on, we all know what the real reason is. Testing is less fun that coding features or fixing bugs. Figuring out test failures is even less fun (though I'm working on that).
Posted May 21, 2019 0:26 UTC (Tue)
by roc (subscriber, #30627)
[Link] (3 responses)
You're paying for poor bug tracking already, the cost is just smeared out as people waste time finding and diagnosing bugs people have already found and diagnosed.
I assume the actual hardware/software running costs would be insignificant, so is the problematic cost what you'd have to pay people to do triage or what? I always assumed the barrier to better bug tracking was the difficulty of coming up with a single system and getting people to pay attention to it.
Posted May 21, 2019 0:31 UTC (Tue)
by roc (subscriber, #30627)
[Link] (2 responses)
Posted May 23, 2019 4:25 UTC (Thu)
by tbird20d (subscriber, #1901)
[Link] (1 responses)
Posted May 23, 2019 21:42 UTC (Thu)
by roc (subscriber, #30627)
[Link]
Posted May 20, 2019 23:47 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Even in this case some duplication will still inevitably take place, but at least it can be greatly minimized.
Posted May 21, 2019 9:04 UTC (Tue)
by metan (subscriber, #74107)
[Link] (1 responses)
Posted May 23, 2019 4:33 UTC (Thu)
by tbird20d (subscriber, #1901)
[Link]
I'm aware of 3 different get-togethers to continue working on pushing stuff forward: A testing microconference at Plumbers, a kernel testing summit immediately after Plumbers, and the 2019 Automated Testing Summit after OSSEU/ELCE in France. Some of these have not gotten much visibility yet, but hopefully that will change shortly.
Posted May 25, 2019 17:14 UTC (Sat)
by flussence (guest, #85566)
[Link] (1 responses)
That insanity is already old news. Are you completely unaware of the long tail of blogs posting cargo-culted tweaks to get things like basic GPU acceleration and sound to work in browsers?
“This web browser best viewed with Nvidia on Windows 10” is not a good look, but everyone seems to accept it.
Posted May 27, 2019 23:11 UTC (Mon)
by roc (subscriber, #30627)
[Link]
I'm aware that real-world combinations of software and hardware vastly exceed anything you can test in any reasonable test farm. Firefox's upstream tests aren't perfect, and can't ever be, so tweaks to Firefox's configuration in unusual situations remain necessary. (Keep in mind, though, that many of those "cargo-culted tweaks" aren't addressing Firefox bugs but are choosing different points in a tradeoff, e.g. choosing performance over stability by using GPU drivers with known bugs.)
That doesn't affect what I wrote, though. It is still true that, unlike the kernel, Firefox and Chrome upstreams do massive testing in a way that's tightly integrated with the development process, and take responsibility for detecting, diagnosing and fixing regressions.
Posted May 23, 2019 22:01 UTC (Thu)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted May 27, 2019 23:11 UTC (Mon)
by roc (subscriber, #30627)
[Link]
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
> structured to offload as many costs as possible to downstream. Even though it's a terribly inefficient setup and painful for
> downstream, downstream users have no leverage to alter the deal.
Testing in the Yocto Project
Kernel testing needs to improve a lot — and has moved significantly in that direction. But I do wonder how you might envision this working; what organization, in particular, would do the testing that you are looking for? Who should pay for it?
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Please come to the Automated Testing Summit in Lyon, France, in October. It's a new event, where I hope we can talk about (and listen to testing experts on) exactly the issues you raise. The CFP should be available shortly, and personally I'd love to hear from you and other testers about your experiences. I'll announce the CFP on the automated testing e-mail list when it's ready (which should be within a few weeks).
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
See https://elinux.org/Automated_Testing_Summit_2018 and https://lwn.net/Articles/771782/
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project
Testing in the Yocto Project