Testing in the Yocto Project

Ideally this would be an independent lab funded by the Linux Foundation (or any other trade group). A large warehouse stuffed with various hardware that runs realistic workloads.÷

> But I do wonder how you might envision this working; what organization, in particular, would do the testing that you are looking for? Who should pay for it?

The Linux Foundation seems to have taken on the role of trying to ensure that organizations that profit from Linux put money back into it. I'm not privy to the workings of LF so I don't know whether they can take this on. Still, if there was a consensus in the kernel community that central testing infrastructure is needed and we just have to figure out how to pay for it, that would be a big step forward.

One thing that might help: the kernel community could try to normalize the expectation that writing tests, building better test frameworks, and investigating failed tests is something every kernel developer needs to do (including the expectation that every bug fix and every new feature comes with automated tests). This is normal for a lot of other projects, from tiny no-money projects to large big-money projects. Then the companies paying people to do kernel development would effectively be taxed to support the testing effort.

> The problem with the kernel is that so many of the problems people run into are workload and hardware dependent.

The kernel workload is highly variable. Then again, for browsers your workload is *every Web page in the world*, from gigantic static pages to GMail to twitchy 3D shooters and everything in between. Mostly written in obfuscated code by tens of millions of developers with ... a range of skill levels ... whom you mostly can't talk to. Browsers are sensitive to hardware variation too, though less so than the kernel, sure. But desktop applications have their own special challenges: myriad combinations of software that your software depends on and interacts with, including ultra-invasive stuff like antivirus and malware that patches your code, all changing underneath you day to day, almost entirely closed-source and often controlled by your competitors. Mobile has similar issues.

I talk a lot about browsers because that's what I know. I don't think kernels and browsers are uniquely difficult.

I get that testing the kernel is hard. That's why greater cooperation and more efficient use of testing resources is so important.

> I'm not sure I see an alternative to a lot of downstream testing anytime soon.

Sure but so far I don't even see that the kernel community sees lack of upstream testing as a problem that needs to be solved.

Look at it this way: kernel people emphasize how important it is for people to upstream their code --- "upstream first", etc. That reduces duplication of effort, ensures things stay in sync, etc. But for some reason* they don't apply the same logic to tests. Tests, test harnesses, and test infrastructure should be upstream-first too for exactly the same reasons. Yes, there will always be stuff that is hard to upstream.

* Come on, we all know what the real reason is. Testing is less fun that coding features or fixing bugs. Figuring out test failures is even less fun (though I'm working on that).

> (And yes, a better answer to bug tracking is sorely needed, but there are many ghetto areas in kernel development that nobody wants to pay for, and that is one of them).

You're paying for poor bug tracking already, the cost is just smeared out as people waste time finding and diagnosing bugs people have already found and diagnosed.

I assume the actual hardware/software running costs would be insignificant, so is the problematic cost what you'd have to pay people to do triage or what? I always assumed the barrier to better bug tracking was the difficulty of coming up with a single system and getting people to pay attention to it.

At least people from various kernel QA departments started to talk to each other in a last year or so. I suppose that there is enough funding already but the real problem is that every company works more or less in isolation. Maybe we can solve that with standard interfaces and components so that each company can plug in a hardware they care for into a virtual kernel testing lab, at least that seem to be a way things started to move slowly on. The problem now is that the problem is complex and as far as I can tell there is no significant manpower behind it.

> Are you completely unaware of the long tail of blogs posting cargo-culted tweaks to get things like basic GPU acceleration and sound to work in browsers?

I'm aware that real-world combinations of software and hardware vastly exceed anything you can test in any reasonable test farm. Firefox's upstream tests aren't perfect, and can't ever be, so tweaks to Firefox's configuration in unusual situations remain necessary. (Keep in mind, though, that many of those "cargo-culted tweaks" aren't addressing Firefox bugs but are choosing different points in a tradeoff, e.g. choosing performance over stability by using GPU drivers with known bugs.)

That doesn't affect what I wrote, though. It is still true that, unlike the kernel, Firefox and Chrome upstreams do massive testing in a way that's tightly integrated with the development process, and take responsibility for detecting, diagnosing and fixing regressions.