LWN: Comments on "The costs of continuous integration"

The costs of continuous integration

nix — Mon, 16 Mar 2020 22:24:54 +0000

Exactly. The cost of sending a few terabytes around my local network is basically nil (I suppose it costs a few micropennies in extra electricity, but that's utterly unnoticeable).

The costs of continuous integration

immibis — Thu, 12 Mar 2020 15:54:17 +0000

A carbon tax would make sure the emissions are priced into that number.

The costs of continuous integration

immibis — Thu, 12 Mar 2020 15:53:30 +0000

Also that the people who decide how much the CI system should do aren't the ones paying for it.

The costs of continuous integration

immibis — Thu, 12 Mar 2020 15:50:52 +0000

> It's a massive waste of CI compute that is barely felt because most orgs aren't struggling to pay for the CI time. I think that perhaps you'll find a similar situation is occuring with this as well.

This. When you have a physical server running CI (or a handful of servers with different hardware configurations that you want to test), you *want* to run it as much as possible because otherwise it's doing nothing. It's not until you hit your capacity limit that you have to find the most economical level of testing.

Pay-by-the-hour/gigabyte cloud hosting just makes it harder to know what your limits are. I have no doubt that most of these projects could get by with a much smaller rate of CI builds on many fewer platforms. But the power of CI is that it helps you find obscure bugs quickly, and then it wouldn't do that any more.

Imagine this scenario: everyone tests their own code, and the CI test runs only once a day on each platform. You would have a one-day turnaround on obscure platform bugs, but the cost would be way way way lower.

The costs of continuous integration

pizza — Tue, 10 Mar 2020 12:56:19 +0000

Running on your own systems does push the costs elsewhere, but it's disingenuous to claim that doing so is automatically more expensive, especially if your cost is dominated by data transfer.

For my group, keeping everything in-house is both faster _and_ cheaper.

The costs of continuous integration

judas_iscariote — Tue, 10 Mar 2020 12:34:14 +0000

Yeah, but running on your own computers is not going to make it cheaper at all, it will just push the cost elsewhere.

The costs of continuous integration

pkolloch — Tue, 10 Mar 2020 06:18:28 +0000

I see a lot of CI pipelines that build and rerun everything nowadays, even though saner build systems like nix or bazel will cache appropriately. You still have to structure your build dependencies a bit carefully but then you can often avoid a lot of work. nix caches by transative input hashes right now, so bazel caching by immediate inputs is often more precise, e.g. the reformatting mentioned in other comments would cause the directly affected build units to rebuild but their output would be the same (given a reproducible compiler) so the rest is skipped.

The costs of continuous integration

wookey — Mon, 09 Mar 2020 16:56:25 +0000

Obviously the cost is an issue, but potentially more of an issue is the energy use and thus emissions of the modern tendency to do epic amounts of CI. How many tonnes does that $90,000 represent? Is it one, ten or thousands?

This probably pales into insignificance in comparison to the madness that is bitcoin, but we should consider how much CI we are doing and where we are doing it and whether the emissions are worth the benefit we get. Being wasteful is not excused by doing it on behalf of free software. '10 years to halve emissions' might include being a bit more responsible/targeted with our CI.

Maybe a dumb question

marcH — Fri, 06 Mar 2020 16:25:34 +0000

> Text is tiny compared to the resulting binaries that you have to transfer to the machines that run the tests.

Cache these too and transfer with rsync? It performs miracles when the binaries are not compressed.

The costs of continuous integration

marcH — Fri, 06 Mar 2020 16:23:44 +0000

> I have been banging on about this for a bit now, using simple stuff like githooks and docker to have all of your non-functional CI tasks run client side (linting, testing to some degree, formatting), anything that isn't "deploy this to x" but I don't seem to have much luck convincing people ...

How "simple" is that stuff really? The typical vicious circle is:
- test dependencies and setups are complex, so few developers run them locally
- because it affects only a few developers, the validation/devops team keeps increasing the complexity, asking these few developers to "upgrade" to new test frameworks etc.
- fewer and fewer developers run tests locally
- repeat and the gap widens each time

Ideally, anyone can copy/paste the command straight from the CI log[*] and at worst get a "program not found" fixable with a simple apt/dnf install. A test setup should not be less automated than a build setup - ideally they should be treated as one.

[*] and githooks just a couple lines long, personal preference.

The costs of continuous integration

marcH — Fri, 06 Mar 2020 15:22:29 +0000

> Many people will consider that an opportunity: https://xkcd.com/303/ ("My code's compiling").

Simple, more visible gap between developers who care a little bit about their productivity vs not really? Approved by HR immediately! Not sure I'm really against it either.

The costs of continuous integration

marcH — Fri, 06 Mar 2020 15:16:08 +0000

"Testing can only prove the existence of bugs, not their absence", yet it's difficult for some developers inexplicably still not interesting in quality to understand that testing is not just one button and not just one green/red light but various coverage/resources/time tradeoffs.

Several times in my careers I had surreal discussions with sometimes senior developers where I would ask tracking unfortunately large lists of "known failing" tests to immediately detect and report regressions in passing tests and measure progression. Very basic validation stuff yet the answer was: "no, let's fix all the tests instead" which would of course never happen.

The costs of continuous integration

mathstuf — Fri, 06 Mar 2020 14:13:37 +0000

My reading has been that the biggest cost has been the bandwidth usage, not CPU usage. So moving builders to be even further away is unlikely to help with the biggest problem.

The costs of continuous integration

Seirdy — Fri, 06 Mar 2020 10:10:17 +0000

Isn't it possible to set up a mirror to gitlab.com and run expensive tests there
while running quicker tests on the self-hosted instance? Expansive tests can be run
on the local instance for new tags/versions, but for every little commit it should be
okay to run on an extenal instance.

The costs of continuous integration

geert — Fri, 06 Mar 2020 09:32:19 +0000

The previous generation checked their code meticulously for syntax and other errors before submitting their punched cards to the datacenter.
My generation relies on local tools to catch such errors.
The next generation just pushes everything to the CI.

The costs of continuous integration

geert — Fri, 06 Mar 2020 09:28:23 +0000

Many people will consider that an opportunity: https://xkcd.com/303/ ("My code's compiling").

The costs of continuous integration

fw — Fri, 06 Mar 2020 08:29:37 +0000

The computers wouldn't be cheaper in all cases (only in some), but the network traffic would be, by over an order of magnitude. Closer to two, actually. And that's before factoring in bulk discounts.

The branded public cloud overcharges a lot for network traffic. It's quite ridiculous. You don't even have to roll out your own network to avoid these costs. Just switching to an off-brand cloud helps a lot.

The costs of continuous integration

smurf — Fri, 06 Mar 2020 07:51:22 +0000

Probably not, but there's a middle road – host the instances in a data center with low traffic cost and admin them yourself, instead of paying Google $$$ for what's $ or $$ (additional) cost to them.

Yes I know that economy of scale wouldn't work if *everybody* was only willing to pay their marginal cost, but that's why hosting commercial code should be (and often is, cf. Github) priced differently than free software.

The costs of continuous integration

pkern — Fri, 06 Mar 2020 07:16:05 +0000

Sure, but you can monitor for that. Whatever public infrastructure you offer will eventually be abused. That might even be true of any private infrastructure. ;-) At the same time people will only care about that if there are actual costs they see.

The costs of continuous integration

mathstuf — Fri, 06 Mar 2020 05:14:13 +0000

> using simple stuff like githooks and docker to have all of your non-functional CI tasks run client side (linting, testing to some degree, formatting)

The issues we found is that the set of tools available on client-side are either not easily available everywhere (e.g., clang-format on Windows), or have silly version-specific behaviors (also clang-format). Plus, `git commit -n` exists, so you're stuck doing server-side enforcement *anyways*.

We're migrating to gitlab-ci at work and we do have a robot that does those kinds of checks (fairly cheaply; it's a 2 core VM and only really gets unhappy if you toss MRs with thousands of novel commits in them at it). There's another ~3x perf benefit laying around I haven't gotten to because it hasn't been necessary (and having to plumb libgit2 through everything rather than just exposing "here's a way to run an arbitrary git command" is more painful than I'd like it to be). Hooking things up so that the robot blocks testing if some subset of checks aren't passing (e.g., formatting is bad vs. some WIP marker detection) doesn't seem to be *too* hard (though may requires manual intervention of some kind on the first stab).

It works with any Gitlab instance (Github too for those curious, but it's an app there, so on the deprecation route; getting it to work in an action is on the TODO list) if folks are interested (fully FOSS, just bad "marketing" since I've had it on my list to blog about it for…a long time now).

The costs of continuous integration

gerdesj — Fri, 06 Mar 2020 01:34:34 +0000

>What I've gleaned is that all human organizations of size (> Dunbar's Number) tend toward the Tower of Babel. Inevitably. Because #people.

Well yes they do for obvious reasons. I'll play straight man here. Dunbar's number is a measure of your direct contacts. Remember that each person has their own circle and it is subtly different from all the others. Changes in language, accent, dialect etc will propagate gradually across these circles of influence. Give it time and modern English, Dutch and German are very different languages, despite having fairly similar roots.

The costs of continuous integration

smitty_one_each — Fri, 06 Mar 2020 01:16:18 +0000

Well, we can't have J. Random Crapper making these contributions, can we?

What I've gleaned is that all human organizations of size (> Dunbar's Number) tend toward the Tower of Babel. Inevitably. Because #people.

The costs of continuous integration

gerdesj — Fri, 06 Mar 2020 01:04:40 +0000

>> . . .75k USD this year, and 90k USD next year. . .

> "You mean: two to three seconds," said the U.S. DoD.

> I kills me that open source is so beggared. I enjoy being an FSF member, because we have to feed these good tools.

It is slightly annoying to me as a tax payer that the likes of DoD 'n' co could be so wasteful that you think it should be routine to look down on sums like that because they can be consumed in seconds.

Here in the UK we laugh at how much it costs for a football (soccer) player to take a shit on a contract at something like £200,000 p/w. Allow five mins in the bog: 200,000*52/365/24/12 = £99. Now that is obviously value for money but I suspect that a right thinking tax payer's largesse consumer would slap a £5000 per turd testing and initial deployment fee on top. Obviously there will be turd maintenance fees for 30 years which will be forecast scaled (inappropriately) and billed for 50 years, with an overrun of extra charges cascaded for another 20 years (because: "nonsense").

That's a very simplified (and only slightly jaded) view of military spending by ... some military lot.

Maybe a dumb question

daniels — Thu, 05 Mar 2020 23:16:32 +0000

They are caching.

The costs of continuous integration

iabervon — Thu, 05 Mar 2020 19:57:00 +0000

The right way to say that is: run the tests for the area you touched first, and if they fail, don't bother finding out if you also broke other things. It's worth minimizing the cost of finding the most likely problems. For that matter, the full CI looking for subtle and unexpected interactions should probably be after code review if it's expensive, since it's pretty likely that there will be a few rounds of not-quite-clear code, and nobody needs to be more than 99% sure these don't cause subtle problems, while they need close to 100% for the final code.

Maybe a dumb question

kaesaecracker — Thu, 05 Mar 2020 19:20:32 +0000

The code/git is not the problem here. Text is tiny compared to the resulting binaries that you have to transfer to the machines that run the tests.

Maybe a dumb question

rahvin — Thu, 05 Mar 2020 19:00:52 +0000

Why isn't the CI build client comparing it's local cache against the git instance and only downloading the changes? That bandwidth figure leads me to believe each CI run is re-downloading the whole git tree. The CI server should be able to store or cache the git tree and download only the changes which should dramatically lower bandwidth.

The costs of continuous integration

Cyberax — Thu, 05 Mar 2020 18:34:09 +0000

If you have them locally it probably won't be cheaper.

The costs of continuous integration

flussence — Thu, 05 Mar 2020 17:59:51 +0000

Sounds like the bulk of the cost comes from a dependency on Someone Else's Computers.

The costs of continuous integration

NYKevin — Thu, 05 Mar 2020 17:48:03 +0000

Unfortunately, you can't "just" treat a machine as untrusted and think your job is done. Black hats will use it for DoS attacks against third parties, to mine whatever cryptocurrency is popular, and in various other nefarious ways that don't require trust.

The costs of continuous coffee consumption

NYKevin — Thu, 05 Mar 2020 17:45:46 +0000

I was not aware that Captain Janeway was maintaining the CI system.

The costs of continuous integration

smitty_one_each — Thu, 05 Mar 2020 17:40:11 +0000

> . . .75k USD this year, and 90k USD next year. . .

"You mean: two to three seconds," said the U.S. DoD.

I kills me that open source is so beggared. I enjoy being an FSF member, because we have to feed these good tools.

The costs of continuous integration

daniels — Thu, 05 Mar 2020 16:16:21 +0000

> I think that perhaps you'll find a similar situation is occuring with this as well.

No, I'd be pretty impressed if we'd managed to burn so much money running clang-format.

GStreamer runs an extensive conformance suite (gst-validate) over its changes, as well as builds on several distributions and architectures. Mesa runs builds on several architectures (x86, x86-64, armv7, armv8, s390x, ppc), a couple of build systems (Meson for Linux, SCons for Windows), and then runs tests such as the OpenGL (ES) conformance test suite and Piglit across the software rasteriser and a couple of different hardware platforms (Panfrost for newer Arm Mali GPUs, Lima for older Mali GPUs, Freedreno for Qualcomm GPUs).

The costs of continuous integration

martin.langhoff — Thu, 05 Mar 2020 15:24:50 +0000

Having a short, fast smoketest set of tests, and an extensive set of tests is as easy as marking the tests for different test suites. It's pretty much standard practice in my book.

Gitlab and others have the ability to run different test suites for the PRs and on a configurable "cron" schedule, so you run the smoketests on every PR, and "full" test suite on a daily basis on the master branch. If the full test suite broke, you can bisect to the commit or PR that broke it easily.

If data transfer of VM/Docker images and other assets are dominating the bill, a local image repo or just a cache should wipe 99% of that cost.

All tractable issues, IMO. And CI is a net win.

The costs of continuous integration

blackwood — Thu, 05 Mar 2020 11:52:35 +0000

It's not containers (we have zero illusions they're going to hold an attacker up), it's cheap throwaway machines that do nothing else than run CI jobs. You bomb one, we reinstall it. There's some cross-project credentials you might be able to sneak from other jobs, stuff like "I've rebuilt, need to kick of the next project in the depedency chain to do it's rebuilding". But given that all the code is open anyway the amount of damage you can inflict by taking over CI runners is really minimal. Breaking into the main servers is a different thing entirely.

The costs of continuous integration

pkern — Thu, 05 Mar 2020 10:43:26 +0000

I would argue that as long as no artefacts are copied off those machines and reused for anything but testing on another test machine, you can treat those machines as entirely untrusted. Of course that also means that those machines must not have any privileged access. With containers and VMs you can at least isolate the slightly more privileged runner from the actual test workload.

The costs of continuous integration

farnz — Thu, 05 Mar 2020 10:19:09 +0000

That's not too far off what VS Code Remote Development lets you do. Granted, it's an Electron app, not a web site, but that's nearly the same thing.

The benefit is that instead of having to have a powerful laptop, you can simply choose one that's nice to work on, and then do all the compiler running (gcc etc) on a nice fast machine in a datacentre somewhere.

The costs of continuous integration

rwmj — Thu, 05 Mar 2020 09:07:45 +0000

The logical conclusion of this is developers won't write the software locally at all. They'll simply edit it in some kind of web interface and everything will happen on the CI system. What a world ...

The costs of continuous integration

gfernandes — Thu, 05 Mar 2020 08:32:51 +0000

You're thinking about *sequential* workers. Many workers, work in parallel. So it's still worth it for them to "outsource" the build/test to the CI pipeline, and use that time to work on something else.

The costs of continuous integration

marcH — Thu, 05 Mar 2020 07:13:06 +0000

> The lack of any oversight of what gets run in the CI system and which projects are responsible for it is part of the problem, Airlie said. "You can't have a system in place that lets CI users burn [large] sums of money without authorisation, and that is what we have now."

From a security perspective, the ability for basically anyone to run arbitrary code at ease never ceases to amaze me. I know: containers etc. but still.