The costs of continuous integration

Posted Mar 5, 2020 3:16 UTC (Thu) by JohnVonNeumann (guest, #131609)
In reply to: The costs of continuous integration by roc
Parent article: The costs of continuous integration

What clopez said and in my original comment I did make a proviso around running test suites, I'm well aware of the size of some test suites. But you don't need to run all the tests for CI builds, just the test for the area in which you are working, which should be managable by your local device. But the crux of my argument didn't revolve around the tests. It was around the stuff that can be done locally, like static analysis and linting primarily.

Whilst I also get what you're saying about the tests, the simple fact is that most places are not firefox, and people should therefore stop acting as if their test suites are so large that they can't run them locally.

The costs of continuous integration

Posted Mar 5, 2020 3:28 UTC (Thu) by gus3 (guest, #61103) [Link] (1 responses)

"you don't need to run all the tests for CI builds, just the test for the area in which you are working"

But are you absolutely, positively certain that your work doesn't touch any other areas?

If the system integrator says otherwise, you'll need to run *all* the CI build tests.

The costs of continuous integration

Posted Mar 5, 2020 4:58 UTC (Thu) by JohnVonNeumann (guest, #131609) [Link]

No of course you're not certain your work doesn't touch other areas, that's why the CI pipeline still runs the full test suite.

The costs of continuous integration

Posted Mar 5, 2020 19:57 UTC (Thu) by iabervon (subscriber, #722) [Link] (1 responses)

The right way to say that is: run the tests for the area you touched first, and if they fail, don't bother finding out if you also broke other things. It's worth minimizing the cost of finding the most likely problems. For that matter, the full CI looking for subtle and unexpected interactions should probably be after code review if it's expensive, since it's pretty likely that there will be a few rounds of not-quite-clear code, and nobody needs to be more than 99% sure these don't cause subtle problems, while they need close to 100% for the final code.

The costs of continuous integration

Posted Mar 6, 2020 15:16 UTC (Fri) by marcH (subscriber, #57642) [Link]

"Testing can only prove the existence of bugs, not their absence", yet it's difficult for some developers inexplicably still not interesting in quality to understand that testing is not just one button and not just one green/red light but various coverage/resources/time tradeoffs.

Several times in my careers I had surreal discussions with sometimes senior developers where I would ask tracking unfortunately large lists of "known failing" tests to immediately detect and report regressions in passing tests and measure progression. Very basic validation stuff yet the answer was: "no, let's fix all the tests instead" which would of course never happen.