Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 20:29 UTC (Sat) by ehiggs (subscriber, #90713)
Parent article: Python gets a "Developer-in-Residence"

> As you can see, re-running concurrent_futures tests now takes 2.5s instead of over 82s.

> Interestingly, this not only increases speed of the re-runs but also makes it more likely that they will succeed. You see, the fewer tests we need to re-run, the less likely we are to hit another intermittent failure.

I'm aware/familiar of the operational realities of flaky tests and also requiring a green build to merge. But concurrent futures is likely non-deterministic code. IME if concurrent tests fail ever then it's **definitely** wrong and any green test result is luck. Rerunning to tease out a green test suite run is not a good useful in the slightest.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 20:30 UTC (Sat) by ehiggs (subscriber, #90713) [Link]

> Rerunning to tease out a green test suite run is not a good useful in the slightest.

Bad edit :(

Rerunning to tease out a green test suite run is not {a good idea | not useful} in the slightest.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 21:34 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

I don't think that's talking about "rerunning until it passes", but rather "if the test suite determines you may have affected it, it no longer takes more than a minute". But there are tests that are useful to "rerun until pass or N failures" (in one project I work on, context menus are opened which are display manager modal in X and serializing the tests is not feasible as it'd easily make the test suite take 5x longer to run). Tests involving inter-thread communication are better to "rerun until failure or N passes" though.