Risk management anyone? [LWN.net]

Risk management anyone?

Posted Oct 3, 2024 12:32 UTC (Thu) by aragilar (subscriber, #122569) [Link]

While I do run both PyPI and npm caches (both of which were fairly easy to set up), I would argue the majority blame should be shifted to the hosted CI providers (especially GitHub via its Actions ecosystem as it has the worst setup for caching I've used), who could all provide a caching server and make it easy to use (either by setting the required environment variables by default, or via whatever configuration mechanism makes sense for their system), rather than the developers themselves.

Currently, for hosted CI, you would need to spin up your own cache servers, which given the value of hosted CI is to not need to run servers, would seem to be a much larger ask of individual developers.

Risk management anyone?

Posted Oct 3, 2024 16:11 UTC (Thu) by dskoll (subscriber, #1630) [Link] (8 responses)

Yes. I've blocked downloads of some of my software from various places that re-download the same thing every single time they do a build. There's no excuse for that sort of abuse.

Risk management anyone?

Posted Oct 3, 2024 16:52 UTC (Thu) by Wol (subscriber, #4433) [Link] (7 responses)

How easy is it for a remote download to tell nothing has changed?

Running gentoo, I'm conscious that seems to download everything every time, but it also doesn't download unless something has changed (be it ebuild, use flags, whatever).

I would think it *should* do a shallow git clone, and keep that lying around unless the user explicitly clears it (it leaves enough stuff lying around, why not that), so even if use flags and stuff has changed, it would have no need to get the source afresh unless there really is an upgrade.

But not knowing python, or the guts of portage, I have no way of knowing if that's actually the case ...

Cheers,
Wol

Risk management anyone?

Posted Oct 3, 2024 16:59 UTC (Thu) by pizza (subscriber, #46) [Link] (4 responses)

> How easy is it for a remote download to tell nothing has changed?

There's another aspect to that -- caching is important, but another thing the CI needs to test for is that the original resource is still available.

...I've had plenty of CI runs that _falsely_ succeeded because they used a cached copy of a no-longer-available resource, leading to unexpected failures when (eg) doing a production build or spinning up a new developer environment.

Risk management anyone?

Posted Oct 3, 2024 21:36 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (3 responses)

Is it vital to check that the original file is still online 5 thousands times per day?

And what if it isn't?

Debian is full of packages whose original websites are gone. Every once in a while someone uses the last .tar.gz from debian to make a fork.

Risk management anyone?

Posted Oct 3, 2024 21:50 UTC (Thu) by SLi (subscriber, #53131) [Link]

It certainly is vital to check it often if your build does fetch it from somewhere external. Now fetching it from somewhere external may not be the right thing to do, but absolutely, a CI should detect if a build breaks, and a cache would only mask this failure. A proper mirror with no unpredictable expiry rules is another matter.

Risk management anyone?

Posted Oct 3, 2024 21:55 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> Is it vital to check that the original file is still online 5 thousands times per day?

For a single organization? Probably not. But if it's 50000+ different orgs each checking once a day?

> And what if it isn't?

Then you have to determine why, and adjust your system's data source accordingly.

(Note "original file" can easily point at a private/internal mirror or some sort of SW BoM artifact storage. Granted, some ecosystems make this sort of thing ...challenging to set up and transparently utilize)

Risk management anyone?

Posted Oct 4, 2024 8:22 UTC (Fri) by LtWorf (subscriber, #124958) [Link]

In my own experience, it's most usually few organizations spamming, rather than many organizations. Easy to see because overnight the daily downloads might halve or double. And I doubt it's because thousands of organizations all went to use a different library all in the same day.

Risk management anyone?

Posted Oct 3, 2024 19:46 UTC (Thu) by dskoll (subscriber, #1630) [Link] (1 responses)

It's pretty easy. In my case, the build code was downloading a tarball over HTTPS, so it could easily have used the If-Modified-Since: header.

However, I suspect the build was done one a virtual machine that was spun up from scratch anew each time, so there was no existing tarball for it to check the timestamp against.

Risk management anyone?

Posted Oct 3, 2024 21:39 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

At work at least for local builds I made our system use a cache that gets mounted into the container. It saves considerable amount of time (and allows working on mobile connections) to not download several hundreds of MB of things every time.

Risk management anyone?

Posted Oct 3, 2024 18:19 UTC (Thu) by raven667 (subscriber, #5198) [Link] (7 responses)

Yeah, mirroring and caching web content is far less emphasized in general than it used to be when connection speeds were slower. I don't hear about people setting up Squid proxies or creating internal mirrors, both of which used to be pretty routine IT infrastructure. Now directly updating/installing from public mirrors is fast enough in most cases that people don't keep working to optimize their process and move on, whereas before when you might only have a 1Mbit connection to the Internet shared by your whole office, you couldn't afford the time spent downloading the same thing more than once

Risk management anyone?

Posted Oct 4, 2024 0:53 UTC (Fri) by yeltsin (guest, #171611) [Link]

Depends on where you live. I'm thousands of kilometers away from major datacenters, and every developer worth anything sets up aggressive dependency caching for every CI job because it's unbearable to use otherwise.

Risk management anyone?

Posted Oct 4, 2024 21:40 UTC (Fri) by kleptog (subscriber, #1183) [Link] (5 responses)

Squid proxies only work for HTTP traffic. For HTTPS it doesn't help at all. Since all the package managers (rightfully) depend on HTTPS for security any caching cannot be transparent.

So even of GitLab of Azure DevOps wanted to provide caching for npm or PyPi, they couldn't do it in a way that's transparent. And if they provide a non-transparent mechanism, it makes it a potential MITM.

Ideally there'd be an extension to HTTPS to allow clients to opt into caching, while still preserving the authentication properties of HTTPS, but I think the ship has sailed on that one.

Risk management anyone?

Posted Oct 5, 2024 1:08 UTC (Sat) by dskoll (subscriber, #1630) [Link] (3 responses)

Apt doesn't require https. In fact, all of my sources.list entries are http.

If your packages are signed and you verify the signature, https doesn't buy you anything. If a package is validly-signed, then it doesn't really matter where you downloaded it from.

Risk management anyone?

Posted Oct 5, 2024 2:11 UTC (Sat) by intelfx (subscriber, #130118) [Link] (2 responses)

> If your packages are signed and you verify the signature, https doesn't buy you anything.

That's not strictly true. At the very least, transport-layer encryption buys you confidentiality.

Risk management anyone?

Posted Oct 5, 2024 7:20 UTC (Sat) by LtWorf (subscriber, #124958) [Link]

Unless your attacker can infer that the debian.org hostname is a mirror and uses that information to understand what you're downloading from the sizes of the files that get downloaded.

Risk management anyone?

Posted Oct 5, 2024 14:16 UTC (Sat) by dskoll (subscriber, #1630) [Link]

I don't think confidentiality is an issue for CI/CD pipelines, especially for open-source products where anyone can just look at what exactly the CI/CD pipeline is doing.

Risk management anyone?

Posted Oct 7, 2024 10:56 UTC (Mon) by james (subscriber, #1325) [Link]

Actually, Squid can "bump" or non-transparently MITM HTTPS traffic (and therefore cache it). In a CI environment, this might be a reasonable thing to do, if you consider Squid to be part of the same security domain as the CI environment.