Fetch dependencies? Aaaaaah! [LWN.net]

Fetch dependencies? Aaaaaah!

Posted Sep 12, 2021 1:56 UTC (Sun) by HelloWorld (guest, #56129)
In reply to: Fetch dependencies? Aaaaaah! by mirabilos
Parent article: Cro: Maintain it With Zig

Build tools download dependencies because that's the only way to get stuff to work more or less consistently across different developers' machines. Like it or not, but that's just the way the world works these days, and it's time that the C/C++ community catch up to 2005 when it comes to that.

And actually, people increasingly just specify their entire development environment as a container image, see e. g. Gitpod. That makes most ”it works on my machine“ issues just go away. You run a container and that's it, you're done, no need to fiddle around with your setup.

But hey, I guess if you like messing around with m4 and pkg-config and all that, that's cool too.

to post comments

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 0:22 UTC (Mon) by flussence (guest, #85566) [Link] (14 responses)

No thanks.

Modern development practices should not be merely gentrifying the experience of downloading multi-gigabyte mystery meat zip files from random xda-developers threads, or jumping through the same miserable hoops as setting up a semi-proprietary embedded environment in 2005. Don't throw overflowing trashbags of mystery binaries and other people's vendored code with private modifications over the wall and pretend it's open. It's not okay when the likes of Google do it, and you certainly aren't as important.

Also, "pkg-config"? Update your FUD, most people switched to pkgconf years ago.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 1:30 UTC (Mon) by HelloWorld (guest, #56129) [Link] (13 responses)

Wow, you clearly have no idea what you're talking about. Build systems don't download “multi-gigabyte mystery meat zip files from random xda-developers threads”, they fetch packages from well-known package repositories such as Maven Central, and they certainly don't encourage vendoring – it's way more convenient not to bundle dependencies and just have the build tool download them.
And jumping through hoops? How is it jumping through hoops when the build tool just does everything for you? Jumping through hoops is when you need to chase down every single library yourself because your build system isn't capable of doing what it should: take care of the boring stuff in a reliable, reproducible way.

Besides, do you think people make modifications to their libraries just for the fun of it? No, they do it because they need those changes for their program to work right. Most developers aren't idiots looking for ways to make their life harder, you know...

> Also, "pkg-config"? Update your FUD, most people switched to pkgconf years ago.
Yeah, like that makes any kind of meaningful difference.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 2:01 UTC (Mon) by mirabilos (subscriber, #84359) [Link] (12 responses)

Right, Maven Central, now that you mention it as prime example, is a cesspit. Sourcless crap, sometimes not even built with Maven or rebuildable at all. You get source JARs only if you’re lucky, and often *cough*Lombok*cough* they don’t correspond to what’s shipped or only partially correspond to it, and they’re clearly not Complete Corresponding Source. Licence metadata in POM files, if it exists at all, is usually outdated or sometimes even so plainly wrong it was never right in the first place, so in $dayjob, where I often have to do licence analysis/compliance for Java™ stuff, I have to look at every single file…

… not that others are better in any way. Look at the npm package for jQuery once. The consider that jQuery bundles other libraries in its… compilate. Also look at how they ship an older jQuery binary as part of the testsuite runner. Continue screaming.

No, I know why I fully subscribe to the Debian schema of building everything themselves. I learnt in MirPorts, ages ago, that you *always* have to regenerate e.g. configure scripts, period, for example.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 5:07 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

Maven is the first widely used build system that downloads dependencies automatically (well, maybe apart from CPAN), so it's showing its age.

Most of more modern systems don't have this problem. They provide complete bit-for-bit identical build environments via lockfiles and it's also common to only provide source code, not prebuilt binary artifacts.

In particular, jQuery NPM package doesn't require any third-party downloadable libraries: https://www.npmjs.com/package/jquery

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 12:11 UTC (Mon) by mirabilos (subscriber, #84359) [Link] (8 responses)

Haha, jQuery.

It’s still there: https://github.com/jquery/jquery/blob/main/test/data/jque...

Also, includes sizzle.js, not stating the version, and not including that at compile time (or, better, at runtime!), no, it’s bundled, and its licence possibly not even honoured.

I also found “interesting” things in npm but not yet the 5 MiB coffee cup… I did report a few short of 100(!) licence violations to SwaggerUI though, which to fix required them to rethink their entire build system.

This MUST be thought of BEFOREHAND. npm makes it easy to do the wrong thing, even more so than Maven. Debian makes it easy to do the right thing.

And, wtf, do you really say that Arch Linux DOESN’T disable network access during package compilation‽

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 17:36 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> Also, includes sizzle.js, not stating the version, and not including that at compile time (or, better, at runtime!), no, it’s bundled, and its licence possibly not even honoured.

Sizzle.js is copyrighted by jQuery and started as a part of jQuery. Anything more serious?

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 18:39 UTC (Mon) by karkhaz (subscriber, #99844) [Link] (6 responses)

> And, wtf, do you really say that Arch Linux DOESN’T disable network access during package compilation‽

That's correct, the Arch build system downloads the source as part of the build. There's no centralized storage of package sources. See for example the package for glibc [1], the PKGBUILD file includes a "sources" array with the glibc source tarball. The package build system first downloads that and verifies the checksums, then runs the prepare() function (to apply arch-specific patches), etc.

A few years ago I tried building every Arch Linux package from source, and didn't get terribly far with that, but reported this list [2] of packages for which the URL in the PKGBUILD was incorrect or broken (they've since all been fixed).

[1] https://github.com/archlinux/svntogit-packages/tree/packa...
[2] https://archlinux.org/todo/packages-with-missing-sources/

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 18:56 UTC (Mon) by mirabilos (subscriber, #84359) [Link] (5 responses)

No, not that.

Downloading the source you actually want to build (if you verify it) is fine.

But during the actual build process, network access shouldn’t be available. I personally added this to pbuilder in Debian for Linux, using unshare(1). I don’t currently have a solution for nōn-Linux yet.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 19:14 UTC (Mon) by karkhaz (subscriber, #99844) [Link] (4 responses)

I see. Well, that exists too. I can't remember details but I'm almost certain that LibreOffice does some bizarre thing where it fetches its various language packs from the server during the build.

Indeed you can see in the build() function of the PKGBUILD [1] that the autogen.sh script takes a "--with-gdrive-client-id" flag, which implies that it's pulling something out of Google Drive, and a few lines down it writes a touchfile called "src.downloaded". This is obviously a script written by upstream. So no idea how Debian deals with this. I think I've seen other awful stuff like this, but LibreOffice's build stood out in my memory as being exceptionally painful.

[1] https://github.com/archlinux/svntogit-packages/blob/packa...

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 20:47 UTC (Mon) by Wol (subscriber, #4433) [Link]

How long ago was that?

Bear in mind, that one of the major differences between OpenOffice and LibreOffice in the early years was LO paid off a hell of a lot of technical debt, including a *massive* rewrite of the build system.

Which is why LO can now release new versions every few months, and I can build it at regular intervals on gentoo, while OpenOffice last I heard took 14 months to release a bodged fix for a major security problem because nobody could get the build system to build ...

Cheers,
Wol

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 22:41 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

The gdrive client ID thing is because LibreOffice itself can interact with Google Drive: this configure option lets you bake a client ID to do that access into the binary. It doesn't access Google Drive during the build. It's not *that* crazy. :)

It does download a fairly big pile of tarballs: you can do this before the final make, but after configure runs, via 'make fetch': the tarballs can be kept between runs. This just means that your build system needs to be able to split configure and make apart and run them separately, and run things that allow network access in between: it doesn't mean that you need to allow network access during the configure or make phases.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 22:56 UTC (Mon) by mirabilos (subscriber, #84359) [Link]

Almost.

Fetch them beforehand, by using package system means.

Backdoored configure scripts are a thing.

Fetch dependencies? Aaaaaah!

Posted Sep 14, 2021 11:07 UTC (Tue) by karkhaz (subscriber, #99844) [Link]

Thanks for these details. I did remember that LibreOffice failed to build when I blocked its network access but didn't have time to dig into it too deeply now, good to know that it doesn't download from GDrive for the build :)

However, I think the point still remains. Sure, there's a difference between the configure and build stage from an upstream perspective, but all of that is part of the "build" stage for a package manager. In particular, the Arch build system has a means to automatically check the hashes of all of the package sources that are declared in the "sources" array, before doing anything else. Obtaining source files later in the build process breaks reproducible builds and all the guarantees you get from that. Maybe the autogen script checks those hashes itself, and maybe it doesn't, but sidechaining the OS's own validation mechanisms doesn't really bode well.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 5:51 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

Hehehe, npm packages.

Some weird stuff I found inside js packages:

* .c and .h files (why??)
* .py files
* A library made by a tiny js file, coming with HTML documentation, and a 5MiB example image of a coffee cup.
* windows exe files
* fonts
* configuration files for every possible IDE and editor

I too do not trust languages that autodownload stuff. Plus the build agents are harder to secure.

Fetch dependencies? Aaaaaah!

Posted Sep 13, 2021 10:32 UTC (Mon) by scientes (guest, #83068) [Link]

> I know why I fully subscribe to the Debian schema of building everything themselves.

I talked to some Arch developers about how with Debian you could download the entire source archive and build the whole thing from source offline, and they were like "that is a niche feature.".......

While some git integration would be nice, not leaving Debian anytime soon.