"Critical" projects and volunteer maintainers [LWN.net]

"Critical" projects and volunteer maintainers

Posted Jul 14, 2022 6:46 UTC (Thu) by NYKevin (subscriber, #129325)
In reply to: "Critical" projects and volunteer maintainers by logang
Parent article: "Critical" projects and volunteer maintainers

The problem is, developers are lazy. If pip install doesn't work, they will just copy/paste instead.* And that's even worse, because pip install at least leaves behind a metadata trail, so that when the inevitable CVE comes in, you can find and fix all the projects using a given piece of code. If you copy/paste everything, then there's no metadata, and you have no way of knowing what code has propagated where.

* "License? Eh, it's only a couple of lines,** nobody will ever know."
** Do not do this. This is not how copyright law works.

to post comments

"Critical" projects and volunteer maintainers

Posted Jul 14, 2022 9:06 UTC (Thu) by Sesse (subscriber, #53779) [Link] (11 responses)

On the flip side: It's really easy to pull in a hundred projects as (indirect) dependencies. It's really hard to cut and paste the same amount of code.

"Critical" projects and volunteer maintainers

Posted Jul 14, 2022 16:39 UTC (Thu) by hkario (subscriber, #94864) [Link] (10 responses)

You are underestimating the capacity of people to "vendor" code in.

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 15:50 UTC (Fri) by salimma (subscriber, #34460) [Link] (9 responses)

Especially Git-based projects -- it's so easy to just use Git submodules and vendor in code. Though luckily those are also easy to spot when packaging for distributions.

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 16:34 UTC (Fri) by Sesse (subscriber, #53779) [Link] (8 responses)

Does not compute. It's _really hard_ to vendor in a hundred C++ projects and have them compile and work. (For an extreme outlier, look at Chromium: It has a bit over 300, and a small army of people and tooling to keep that going.) npm create-react-app gives you 2000+ subdependencies without a hitch, for hello world.

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 16:42 UTC (Fri) by hkario (subscriber, #94864) [Link] (7 responses)

That's the neat thing about updating vendored code: you don't.

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 17:49 UTC (Fri) by Sesse (subscriber, #53779) [Link] (6 responses)

Note that I said “compile and work”, not “maintain”.

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 18:05 UTC (Fri) by hkario (subscriber, #94864) [Link] (5 responses)

If you don't change the compiler, then it's unlikely you need to change vendored libraries, even if you modify your project.
And even if you do change the compiler, all that's needed to make the compile again, is make it use older standard explicitly or ignore some warnings, not completely rewrite large pieces of code...

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 18:13 UTC (Fri) by Sesse (subscriber, #53779) [Link] (1 responses)

I have no idea what this means. What does the compiler have to do with anything?

More to the point, do you have real-world examples of OSS projects vendoring 1000+ C++ libraries?

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 19:47 UTC (Fri) by Wol (subscriber, #4433) [Link]

What gcc are we on? When did the linux kernel upgrade guidance from - what was it - 4.9?

That experience says compiler upgrades break working code ...

Cheers,
Wol

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 20:56 UTC (Fri) by bartoc (guest, #124262) [Link] (2 responses)

The problem is that each time a vendored library updates you need to do a bunch of work to ensure your rewritten build system keeps up with any changes in upstream's build system.

You can mitigate this by just like, writing a bash script that downloads and builds all your dependencies and plops them into somewhere where your buildsystem can find them. Then again .... how do you think linux package managers got started :D?

"Critical" projects and volunteer maintainers

Posted Jul 15, 2022 23:37 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

The problem is that each time a vendored library updates you need to do a bunch of work to ensure your rewritten build system keeps up with any changes in upstream's build system.

This assumes you intend to keep in sync with upstream. More frequently, projects vendor libraries precisely because they want to diverge from upstream, and copy/paste/edit is easier in the short term than trying to maintain their changes as a patch. When upstream changes, they don't adopt those changes because they aren't relevant, and the vendored library gradually diverges from upstream.

"Critical" projects and volunteer maintainers

Posted Jul 20, 2022 9:58 UTC (Wed) by bartoc (guest, #124262) [Link]

Yeah, there are two "modes" of vendoring I think. If you're using git sub-modules you probably do want to keep up with _some_ upstream, even if it's your own fork. If you want to make changes you might use git-subtree instead. Or derive from "upstream" by simply reading and "rewriting" their code. The last case may not even be considered vendoring, you intend to maintain the thing separately and also want to perhaps change behavior.

To be honest I think all these methods can be fine sometimes, esp the submodule thing is basically a poor-man's package manager and if you share a buildsystem with upstream (or use meson's cmake support thing, I guess) it can be fine.

Hell even the gnulib thing of extracting particular (possible differently licensed) components can work, they do keep things up to date ... most of the time. Ofc a lot of gnulib is much less useful than you would like.

I do kinda wish when people vendored stuff they were better about changing the names of functions, sometimes the vendored library doesn't "escape" across ABI boundaries and can be used with the upstream version just fine, if you change all the symbol names (or dynamically link). Actually have any linkers had support for doing this when static linking, like you can specify that undef symbols should be resolved from different archives for different sets of objects?