Risk management anyone?

Posted Oct 3, 2024 16:11 UTC (Thu) by dskoll (subscriber, #1630)
In reply to: Risk management anyone? by LtWorf
Parent article: The WordPress mess

Yes. I've blocked downloads of some of my software from various places that re-download the same thing every single time they do a build. There's no excuse for that sort of abuse.

Risk management anyone?

Posted Oct 3, 2024 16:52 UTC (Thu) by Wol (subscriber, #4433) [Link] (7 responses)

How easy is it for a remote download to tell nothing has changed?

Running gentoo, I'm conscious that seems to download everything every time, but it also doesn't download unless something has changed (be it ebuild, use flags, whatever).

I would think it *should* do a shallow git clone, and keep that lying around unless the user explicitly clears it (it leaves enough stuff lying around, why not that), so even if use flags and stuff has changed, it would have no need to get the source afresh unless there really is an upgrade.

But not knowing python, or the guts of portage, I have no way of knowing if that's actually the case ...

Cheers,
Wol

Risk management anyone?

Posted Oct 3, 2024 16:59 UTC (Thu) by pizza (subscriber, #46) [Link] (4 responses)

> How easy is it for a remote download to tell nothing has changed?

There's another aspect to that -- caching is important, but another thing the CI needs to test for is that the original resource is still available.

...I've had plenty of CI runs that _falsely_ succeeded because they used a cached copy of a no-longer-available resource, leading to unexpected failures when (eg) doing a production build or spinning up a new developer environment.

Risk management anyone?

Posted Oct 3, 2024 21:36 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (3 responses)

Is it vital to check that the original file is still online 5 thousands times per day?

And what if it isn't?

Debian is full of packages whose original websites are gone. Every once in a while someone uses the last .tar.gz from debian to make a fork.

Risk management anyone?

Posted Oct 3, 2024 21:50 UTC (Thu) by SLi (subscriber, #53131) [Link]

It certainly is vital to check it often if your build does fetch it from somewhere external. Now fetching it from somewhere external may not be the right thing to do, but absolutely, a CI should detect if a build breaks, and a cache would only mask this failure. A proper mirror with no unpredictable expiry rules is another matter.

Risk management anyone?

Posted Oct 3, 2024 21:55 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> Is it vital to check that the original file is still online 5 thousands times per day?

For a single organization? Probably not. But if it's 50000+ different orgs each checking once a day?

> And what if it isn't?

Then you have to determine why, and adjust your system's data source accordingly.

(Note "original file" can easily point at a private/internal mirror or some sort of SW BoM artifact storage. Granted, some ecosystems make this sort of thing ...challenging to set up and transparently utilize)

Risk management anyone?

Posted Oct 4, 2024 8:22 UTC (Fri) by LtWorf (subscriber, #124958) [Link]

In my own experience, it's most usually few organizations spamming, rather than many organizations. Easy to see because overnight the daily downloads might halve or double. And I doubt it's because thousands of organizations all went to use a different library all in the same day.

Risk management anyone?

Posted Oct 3, 2024 19:46 UTC (Thu) by dskoll (subscriber, #1630) [Link] (1 responses)

It's pretty easy. In my case, the build code was downloading a tarball over HTTPS, so it could easily have used the If-Modified-Since: header.

However, I suspect the build was done one a virtual machine that was spun up from scratch anew each time, so there was no existing tarball for it to check the timestamp against.

Risk management anyone?

Posted Oct 3, 2024 21:39 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

At work at least for local builds I made our system use a cache that gets mounted into the container. It saves considerable amount of time (and allows working on mobile connections) to not download several hundreds of MB of things every time.