So much wasted energy

Posted Sep 9, 2025 9:03 UTC (Tue) by kleptog (subscriber, #1183)
In reply to: So much wasted energy by Cyberax
Parent article: npm debug and chalk packages compromised (Aikido)

Well, the web community has fought hard against any concept of a "trusted HTTPS caching proxy". Technically such a thing would be possible, allowing clients to have the transport security of HTTPS, while also allowing a third-party to cache certain responses transparently. Instead we have the super-wasteful "download everything from the source every time" model.

Debian package distribution shows it can be done in a non-transparent way. Anybody can setup a Debian mirror without asking Debian for permission.

It's a bit late now. Even if we could get enough people to decide that a HTTPS caching proxy was good idea and figure out an implementation, it would be a decade at least before there was enough support to make it work.

So much wasted energy

Posted Sep 9, 2025 14:16 UTC (Tue) by muase (subscriber, #178466) [Link] (2 responses)

Is it that much more wasteful though?

Because whether I connect to server X for download, or to 3rd-party-server Y, doesn't necessarily make a difference in itself – the total amount of connections and transmitted data stays the same. Localization and server efficiency matters, but that is probably already quite good with NPM. I don't know their infrastructure, but I cannot imagine that they have a single cluster in someone's basement that is serving all this all across the world – I bet they are using one or more of the big CDNs, which in turn effectively boils down to pretty efficient and rather localized cache servers.

It would be interesting to do the numbers here, because a custom "trusted HTTPS caching proxy" is also a piece of infrastructure that needs resources and energy. I'm not sure how realistic it is in practice to setup something that's local enough to be more efficient than a highly optimized CDN, which after all can make use of bigger scaling effects and better resource sharing. Maybe if it sits in your local network?

Tbh I think only obvious improvement would be to increase local caching where it's not done already; browsers do that (with caveats), and for build pipelines you can use a pre-populated npm cache in your build pipeline.

So much wasted time

Posted Sep 10, 2025 7:40 UTC (Wed) by kleptog (subscriber, #1183) [Link] (1 responses)

It is wasteful is that most precious resource: time. We added a local cache to our build infra because it reduced time to test our patches significantly. The difference between a build that takes one minute and one that takes five minutes is huge.

It pains me every time I see a Gitlab or ADO build with steps that takes 10 seconds (or longer!) to download a container image and start it to run a process that completes in 2 seconds.

So much wasted time

Posted Sep 10, 2025 13:21 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Oh how I wish for even 5 minute builds :) .

But yes, we have gitlab-runner just accumulate Docker detritus throughout the week. Compilation cache is distributed (Linux) or machine-local (macOS, Windows, and clang-tidy caches on Linux). At the end of the week, we do a `docker system prune --all --force --volumes` to avoid out-of-disk issues. The nightly scheduled pipelines then end up pulling down the primary images to help with the coming week (not all machines will get all images, but it's not zero).

Other places that seem expensive as well that are not necessarily project-specific:

- cache creation and extraction
- archive creation and extraction
- cloning strategies

There are some settings for compression level that may help if the resulting archives don't explode size limits.

So much wasted energy

Posted Sep 9, 2025 18:36 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

NPM also has quite a bit of legacy. There's "integrity" field now in the package-lock file that can store the hash of the dependency, but it's not yet universal. Once it spreads a bit more, local caching can be made safe.

In comparison, Golang readily supports dumb caches because it stores the hashes of all the dependencies. If a cache tampers with a dependency, the download will fail.