LWN: Comments on "Debian discusses vendoring—again"

Debian discusses vendoring—again

j0057 — Mon, 06 Nov 2023 11:43:20 +0000

I agree, the distro model is something from the world of C where system-provided dynamic libraries are the only way to re-use code, and even there many applications have started vendoring their dependencies.

Languages other than C have proper packaging support, and it's now up to the distros to adapt and make use of the information there is. Hunting down which application uses which library is really not that difficult, unless you're unable to tap into the metadata that a given packaging ecosystem already provides.

Debian discusses vendoring—again

plugwash — Tue, 09 Feb 2021 06:22:50 +0000

The problems I see are.

1. Many upstream's idea of "LTS" is far shorter than Debian's idea of regular support.
2. While some projects may be on top of security issues in their dependencies I would wager the majority are not.

For Firefox they have resorted to moving to new upstream "LTS" release series within stable releases of the distro, that it just about tolerable for an end-user app like Firefox but it's really not reasonble for things that are key infrastructure components (and even for firefox it's problematic because firefox updates force rustc updates...........)

Debian discusses vendoring—again

calumapplepie — Tue, 26 Jan 2021 04:42:29 +0000

The central issue is that we don't want to have multiple versions of a package, even when we can. Putting the version in the name is long-established practice for core packages that have major versions: it's the best way to handle upgrades, for instance. However, tracking three different versions of a package means that security and bugfix support becomes almost impossible: we now need to backport fixes not once, but many times.

The fact that apt doesn't permit per-package dependency trees is intentional. We don't want duplicate (or nearly-duplicate) files on our systems, our mirrors, or our security trackers. NPM's ability to let you have a dozen versions of every package just leads to headaches when you need to update one.

Debian discusses vendoring—again

HelloWorld — Fri, 22 Jan 2021 05:19:10 +0000

This definition isn't bizarre at all.

Debian discusses vendoring—again

khim — Fri, 22 Jan 2021 02:48:42 +0000

You couldn't release anything weekly. There are just not enough time to find and fix problems in any new features.

The most you can promise if you “release” weekly is “hey, it passed our automatic testing and therefore we hope it's not completely broken”. That's not a release. 20 years ago it would be called “alpha version” (not even “beta”: beta is supposed to be tested by some testing teams before release, not just pass CI bot). I'm not even sure that thing which they call “an LTS release” can be compared to “normal” stable release of XX century software which you can actually trust.

The really strange result of all that activity is the fact that as “release cadence” shortens the actual, tangible, changes become less and less frequent. But resource consumption and number of bugs skyrockets. Simply because nobody can understand how the whole thing is supposed to work — and the amount of band-aids applied to pile of existing band-aids which ensure that the whole thing doesn't crash immediately upon startup is just endlessly grows.

As Hoare (original inventor of Quicksort) said: there are two ways of constructing a software design — one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

What we are looking on in that pile of weekly “releases” and pile of “containers” is the end result of application of 2nd principle.

Debian discusses vendoring—again

Cyberax — Fri, 22 Jan 2021 01:57:05 +0000

> As for my suggestion about actually solving the vendoring into packages issue, I would love to see the dependencies downloading into the source build and patches sent upstream that allow npm to work in an offline mode.
Go supports this ('go mod download'). We use it to optimize our Docker builds.

Another solution might a Debian-hosted third-party package repository mirror. I know that people are going to attack me with pitchforks, but maybe adopting third-party package managers is actually a better idea than trying to fit everything into deb-src?

Debian discusses vendoring—again

nix — Fri, 22 Jan 2021 01:08:03 +0000

In the Rust ecosystem, Cargo can already do exactly this -- a feature that was added because its first major use case, Firefox, needed it. (Firefox has a *lot* of vendored-in Rust packages, and it wanted to a) use Cargo like everything else, b) build without downloading stuff and c) not have to maintain this massive collection of vendored-in packages by hand.)

Debian discusses vendoring—again

amck — Mon, 18 Jan 2021 13:10:48 +0000

This is confusing two issues.

De-vendoring is a pain, but possible.

Updates to stable for security fixes need to be done on a stable branch, porting just the security fix to the update - not including any other changes relative to existing stable. This is painful but possible.

The versioning naming done makes this possible. e.g. apache2-bin on my stable machine has version "2.4.38-3+deb10u4", the +version part showing the change which falls between stable and testing as required.

The real challenge is (a) bundling (b) overwhelming human resources and the resolution manager in apt.

Debian discusses vendoring—again

foom — Sat, 16 Jan 2021 03:44:42 +0000

I bet the vast majority of software already in Debian does not provide any LTS support, whatsoever.

Now, there won't be CVEs for most issues that were fixed only in main devhead in most software, because nobody is really looking closely enough.

But in the rare case that there is such a CVE, generally Debian would just have to backport the patch.

Debian discusses vendoring—again

MKesper — Fri, 15 Jan 2021 15:50:39 +0000

Pile of shit, I guess?

Debian discusses vendoring—again

farnz — Fri, 15 Jan 2021 15:20:47 +0000

It was - TIOCLINUX subcode 13 (undocumented, but existed) let you control the scrollback from user space. There are documented ioctls for copy and paste you can use to programmatically extract data from a VC.

Debian discusses vendoring—again

LibreTan — Fri, 15 Jan 2021 14:49:20 +0000

I feel that one of the possible solutions to this might be as follows:

Debian can only provide LTS for those Packages which provides LTS as upstream.

Example:
Firefox provides ESR so include it in Debian Stable release.

For all other software which does not provide LTS upstream it should work as OSTree layering only or work through Flatpak only.

If upstream is not providing LTS for their software then how can Debian?

Debian discusses vendoring—again

jafd — Fri, 15 Jan 2021 14:13:43 +0000

A "Piece Of Excrement".

Debian discusses vendoring—again

jafd — Fri, 15 Jan 2021 14:12:16 +0000

Not always, and certainly not in the javascript/NodeJS world.

I won't get into the madness of the neighboring thread (where they are discussing a user-visible change which is not an API change), but here's a very real scenario I'm seeing in a proprietary app that's my job to work on:

1) for $REASONS, we're stuck using a tool of the major version N. It's rather expensive to move to a tool of major version N+1 or N+2.
2) by now, some fourth-order dependencies of the tool had security vulnerabilities discovered, and the only path forward is to bump their major versions. Because JavaScript developers don't believe in backports. A dozen small libraries, but they are all over the place.
3) bumping them will need also bumping the third-order dependencies and so on, which is going to break the tool of version N, no longer maintained.
4) this kind of yak-shaving can take a lot of time, and I'm not counting the regressions that crept in because the migration documentation is far from being complete or accurate. Oh, and new versions of the tool produced buggy output which wasn't our fault. Back to the drawing board we are.

So, basically, until we find time to bring the tool to N+2, we're hosed according to "npm audit". Thankfully the tool is not user-facing, but you can see how "minor fixes" sometimes necessitate bigger changes a project may not be ready to embrace yet.

Debian discusses vendoring—again

jafd — Fri, 15 Jan 2021 14:01:29 +0000

Was the scrollback something you could use programmatically via a kernel API? Or maybe it exposed an ioctl(2) API? Or was there another driver that stopped working entirely because it was using some of the functions that had been a part of scrollback and got obliterated?

I'm confident the world could be better without this very bizarre definition of "API" both you and mjg59 are using here.

Debian discusses vendoring—again

murukesh — Fri, 15 Jan 2021 06:25:41 +0000

According to https://www.jenkins.io/download/lts/, they seem to have settled on a 12-week cadence. But considering they say the normal release cadence is weekly (!), 12 weeks seems like a very long time in comparison (comparing with, say, Ubuntu, where the ratio of LTS/normal support duration is 60 months/9 months ~ 6.66).

Debian discusses vendoring—again

mathstuf — Fri, 15 Jan 2021 04:36:07 +0000

I think you want `"=1.2.3"` there for that. Without any comparison operator, you get `^` or `~` (depending on a zero major version) which means "any compatible version" in practice.

Debian discusses vendoring—again

VolkerWeissmann — Fri, 15 Jan 2021 03:37:26 +0000

If you have
mydependency = "1.2.3"
in your Cargo.toml, then cargo won't update it if you rebuild it.

Debian discusses vendoring—again

VolkerWeissmann — Fri, 15 Jan 2021 03:37:20 +0000

> Which, in turn, means that any JS application is POS which can and would be owned by malicious input presented to it.

What does POS stand for?

Debian discusses vendoring—again

smcv — Thu, 14 Jan 2021 17:07:23 +0000

The resulting binary package has similar properties, but the way you get there is different. With a package that uses Built-Using, a no-changes rebuild will make it pick up the current versions of the packages that it was Built-Using (for example its statically linked copy of glibc, or its statically linked copy of a Go or Rust library). With a package that uses vendoring, the vendored versions of its dependencies are part of the package's source code and will not change without an update to the source package.

Debian discusses vendoring—again

gnu_lorien — Thu, 14 Jan 2021 16:07:09 +0000

I hope that Debian and other distributions find a solution to this problem. About a decade ago when I wanted to check out a new piece of server software I'd often install it from the package manager. In the last five years or so I don't even look for the distribution package. I look for the container image. I look for the curl -X | bash script. I, personally, trust my distribution more than these other methods, but the software isn't there.

Looking at the landscape I hope that Debian sees this as something they must figure out how to do rather than rejecting these other software packaging methods as "bad" or as "not fitting our constraints." It's clear from the explosion of modern packaging systems that what developers want is the ability to vendor and choose the libraries they want to use for themselves. I know this has often been my biggest impediment to packaging software for any particular distribution too: That I only have the given choices of libraries.

As for my suggestion about actually solving the vendoring into packages issue, I would love to see the dependencies downloading into the source build and patches sent upstream that allow npm to work in an offline mode. Once you have the files locked in place as part of your source then the distro can work with the major package managers to extract the data that Debian needs. Right now there are clearly proprietary tools to do this and automatically scan for vulnerabilities. It would be nice to see them pulled into a larger free software effort.

Debian discusses vendoring—again

pj — Thu, 14 Jan 2021 14:34:06 +0000

The other problem is that debian enforces 'one version of a thing' while other package managers don't - npm, for example, lets every package have its own dependency sub-tree, so there may be multiple versions of the same package in use (even major versions, so they have different APIs!). Fixing this for debian requires the version-in-the-name hack, which isn't too bad, but figuring out which packages _need_ this fix might be more difficult. I guess Debian could package them all with the major version, maybe with a kind of 'realm' prefix: npmlib-foojs-1 for foojs v1.x.

Debian discusses vendoring—again

pizza — Thu, 14 Jan 2021 13:56:04 +0000

Okay, fine, if you want to split hairs, I'll be more explicit.

The linux kernel API that allowed for software-based console scrollback was completely removed as part of a "stable kernel security update" from my distribution.

Sure, the patch that removed it originated with the upstream kernel, but it was still an API change that the distro pushed out to its users.

Debian discusses vendoring—again

LtWorf — Thu, 14 Jan 2021 13:15:14 +0000

> That's a solved problem, I believe.

It is, for software that respects licenses, policies, and is not a nightmare to package.

But you just said you don't want to do any of those things with the software you write. So it's not solved for your software.

Debian discusses vendoring—again

LtWorf — Thu, 14 Jan 2021 13:11:28 +0000

That's not an API.

Debian discusses vendoring—again

pabs — Thu, 14 Jan 2021 03:34:43 +0000

The problem with language ecosystem packages is that they often differ from the upstream VCS repo, sometimes even missing source code or build tools to build some generated files.

That said, there are already tools to convert language ecosystem packages to Debian source packages. I think Debian needs to go further than that and integrate more of those things into debhelper and similar.

https://wiki.debian.org/AutomaticPackagingTools

Solved problem in computer science

davecb — Thu, 14 Jan 2021 02:32:10 +0000

First seen in Multics, reinvented in Solaris and again in glibc. Due to be reinvented any time now (;-))

--dave

TL;DR, in reverse order by date:
https://leaflessca.wordpress.com/2018/09/03/avoiding-an-n...
https://cacm.acm.org/magazines/2009/11/48444-you-dont-kno...
https://www.usenix.org/legacy/publications/library/procee...
https://research.swtch.com/vgo-mvs
https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-...
https://lwn.net/Articles/712318/
https://research.swtch.com/version-sat

Debian discusses vendoring—again

dvdeug — Thu, 14 Jan 2021 00:58:29 +0000

Except we're not talking about Kubernetes; we're talking about GSA. GSA almost certainly doesn't have an order of magnitude more security people than Debian. And when we're done with GSA, we're going to be talking about an RPG written in an open source JS framework. Part of the problem is there is no line here; the big items may have support, but the small items are what make up most of the system, and they're endless.

Debian discusses vendoring—again

plugwash — Wed, 13 Jan 2021 21:50:50 +0000

"quite a few" yes, but still a small minority of the total packages. Packaging multiple major versions of the same software is neither explicitly prohibited nor explicitly permitted. In practice it is strongly frowned-upon, though it does happen.

If you just package a new version every time upstream pushes semver and don't go back and clean things up you can quickly end up with a whole pile of versions of a whole pile of libraries.

Some people advocate that it's better to package a new version first in parallel, then go back and port software, rather than trying to do the transition in one bang. The fear is without the carrot of getting the new version of your pet project into testing and/or the stick of having your pet project removed from testing, the work will never get done and the version proliferation will become a reality.

And it gets even worse if the upstream community doesn't see version proliferation or proliferation of tiny unmaintained packages as problems.

Debian discusses vendoring—again

lkppo — Wed, 13 Jan 2021 21:48:48 +0000

That sounds like a good alternative. Another solution would be to have scripts to automatically bulk package the libraries of node, pip, composer, etc. managers in DEB format, independently of the applications that use them. All that remains for the application maintainer is to use the dependencies he needs and possibly plug the holes.

Debian discusses vendoring—again

plugwash — Wed, 13 Jan 2021 21:15:42 +0000

> The issue is we don't want to spend a lot of time packaging a lot a NPM/Python modules in DEB packages.

I don't think this is correct.

Actually producing a package that builds and installs takes negligable time. What takes the time is.

1. Documenting/researching the copyright/license situation and removing anything that violates Debian standards.
2. Waiting for the ftpmasters to validate that what you have done for 1.
3. Deciding what do do about dependency versions, trying to find the path between having piles of versions of the same thing hanging around (which is bad from a security point of view) and breaking stuff when a package is updated.

Debian discusses vendoring—again

drawks — Wed, 13 Jan 2021 20:09:55 +0000

Your description of how it works doesn't sound too different to vendoring, especially for the go/rust type of situations where you are literally vendoring by way of statically linking. I'd be unopposed ot using a new metadata tag that is more specifically scoped for this case, but my point still stands that this is a usable mechanism in all the ways I have described.

Debian discusses vendoring—again

dottedmag — Wed, 13 Jan 2021 19:45:44 +0000

> And as a user, I don't want to download my applications each from a different source. I don't want to remember to update them, each using a different tool.

That's a solved problem, I believe.

> And I want to reduce the risks involved with updates breaking my workflows.

I feel for you and many more (myself included) who are either stuck with old version of software and no support or forced to upgrade to incompatible version. As developers we ought to treat our users better and break less, that's for sure.

Debian discusses vendoring—again

dottedmag — Wed, 13 Jan 2021 19:36:04 +0000

This equally applies to every open source library ever published, doesn't it? :)

Debian discusses vendoring—again

smcv — Wed, 13 Jan 2021 19:35:38 +0000

This isn't what Built-Using is for. A new metadata field could absolutely be used to track what's vendored into each package, but Built-Using is not it.

Built-Using is for the situation when we have some code in one Debian package, and we copy that code into a different Debian package during the second package's build. For instance, if you build busybox-static while you have libc6-dev_2.31-9 installed, version 2.31-9 of libc.a gets statically linked into the busybox-static binary. Now, as long as we have that copy of busybox-static in the archive, we also have to keep a copy of the glibc_2.31-9 source package around, otherwise we'll be missing the source code for some of our binaries (which is a LGPL violation).

Which upstream packages are vendored into the source package for which Debian packages could be tracked in a decentralized way by inventing a new metadata field, vaguely similar to Fedora's use of Provides, but at the moment it's tracked centrally instead: https://salsa.debian.org/security-tracker-team/security-t...

Debian discusses vendoring—again

pizza — Wed, 13 Jan 2021 19:31:49 +0000

oh, quite right.

It's just that when you make that decision to go your own way, you can't assume that anyone else will ever contribute anything, much less shoulder the majority burden of developing/maintaining it.

Debian discusses vendoring—again

drawks — Wed, 13 Jan 2021 18:57:52 +0000

I recall years ago that there was some suggestion that debian could similarly use the "Built-Using:" metadata tags for indicating that a given package has some vendored code included in it. The "Built-Using:" tag already exists for tagging the handful of statically compiled packages which cannot avoid vendoring (busybox, various installer components, etc). The existing presumption is that the vendored components are available at package build time as satisfiable "Build-Depends:". The difference in the npm situation is that at build time the dependencies which are to be bundled would not be packaged separately, but would come from $ambiguous_source and the build system would have to be trusted to tag in the "Built-Using:" data.

This type of solution is a find technical solution if not 100% philosophically "clean" in the way debian intends. I would posit that such a solution would be fine for third party package providers and it would be a big step towards solving real world problems if Debian would just outline that this is a best practice for third parties and that first party debian packages would still be required to provide individually packaged packaged dependencies at build time. Softening the the limits on the number of versions of a library can be in the debian repositories would further make this a maintainable situation moving forward with a reasonable path for all various upstreams to cut between major versions of their dependencies.

at any rate thats my .02 maybe someone can tell me why I'm completely wrong :)

Debian discusses vendoring—again

hkario — Wed, 13 Jan 2021 18:36:53 +0000

(3) also makes sense if any of the existing solutions don't share the goals you have (platform support, maintenance windows, etc.)

and just because you develop in-house doesn't mean you have a). started from zero, b). don't open-source it so that others that do share your goals can contribute

Debian discusses vendoring—again

hkario — Wed, 13 Jan 2021 18:27:49 +0000

many upstreams don't maintain a stable API, let alone a stable branch

Debian discusses vendoring—again

cortana — Wed, 13 Jan 2021 16:26:40 +0000

Debian's package of docker ("docker.io") vendors some of its dependencies. From <https://packages.debian.org/source/sid/docker.io#pdownload> you can see that cli, libnetwork and swarmkit are included within the docker.io source package, rather than packaged separately.

There's some info about this at <https://www.collabora.com/news-and-blog/blog/2018/07/04/d...>.

But I guess these are the easy cases: where the dependency has no other reverse-dependencies. Where there are other reverse-dependencies, vendoring becomes less acceptable...