Testing and the stable tree

Posted May 29, 2019 20:04 UTC (Wed) by mathstuf (subscriber, #69389)
Parent article: Testing and the stable tree

> The merits of an include list versus an exclude list were also discussed. Ts'o said that an include list will never get updated when people get busy, so new tests won't end up being run. With an exclude list, failures are noise that will get attention. French said that it is important to test with both kinds of lists, but it is equally important to collect and use different kernel configurations.

What I've wanted to do for our setup is that it is a blacklist for failing/known "bad" tests. But each week, schedule a "run excluded tests" run. If any come back green, take them off the blacklist again.

> Those files also need to be commented so that it is clear why tests are being skipped.

Agreed. Our blacklists used to be very ad hoc and without a way to tell what's up (issue links are best), the tests just rot on that platform combination.

Testing and the stable tree

Posted May 30, 2019 14:04 UTC (Thu) by Grimthorpe (subscriber, #106147) [Link] (1 responses)

> What I've wanted to do for our setup is that it is a blacklist for failing/known "bad" tests. But each week, schedule a "run excluded tests" run. If any come back green, take them off the blacklist again.

If failing / bad tests show up as green then I think it's a good idea to investigate why that has happened.

If they've genuinely been fixed then that's good, but if there has not been a targeted fix for the problem then making sure it hasn't caused a false positive is important.

Testing and the stable tree

Posted Jun 2, 2019 16:45 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

Of course. Our exclusion list is maintained by hand, so any removals would presumably go with an analysis of the now-passing tests and a commit message.