Debian discusses vendoring—again
The problems with "vendoring" in packages—bundling dependencies rather than getting them from other packages—seems to crop up frequently these days. We looked at Debian's concerns about packaging Kubernetes and its myriad of Go dependencies back in October. A more recent discussion in that distribution's community looks at another famously dependency-heavy ecosystem: JavaScript libraries from the npm repository. Even C-based ecosystems are not immune to the problem, as we saw with iproute2 and libbpf back in November; the discussion of vendoring seems likely to recur over the coming years.
Many application projects, particularly those written in languages like JavaScript, PHP, and Go, tend to have a rather large pile of dependencies. These projects typically simply download specific versions of the needed dependencies at build time. This works well for fast-moving projects using collections of fast-moving libraries and frameworks, but it works rather less well for traditional Linux distributions. So distribution projects have been trying to figure out how best to incorporate these types of applications.
This time around, Raphaël Hertzog raised the issue with regard to the Greenbone Security Assistant (gsa), which provides a web front-end to the OpenVAS vulnerability scanner (which is now known as Greenbone Vulnerability Management or gvm).
The Debian policy forbids download during the build so we can't run the upstream build system as is.
Hertzog suggested three possible solutions: collecting all of the dependencies into the Debian source package (though there would be problems creating the copyright file), moving the package to the contrib repository and adding a post-install step to download the dependencies, or removing gsa from Debian entirely. He is working on updating gsa as part of his work on Kali Linux, which is a Debian derivative that is focused on penetration testing and security auditing. Kali Linux does not have the same restrictions on downloading during builds that Debian has, so the Kali gsa package can simply use the upstream build process.
He would prefer to keep gsa in Debian, "but there's only so
much busy-work that I'm willing to do to achieve this goal
". He
wondered if it made more sense for Debian to consider relaxing its
requirements. But Jonas Smedegaard offered
another possible approach: analyzing what packages are needed by gsa and
then either using existing Debian packages for those dependencies or
creating new ones for those that are not available. Hertzog was convinced
that wouldn't be done, but Smedegaard said that the
JavaScript team is already working on that
process for multiple projects.
Hertzog ran the analysis script described on that page and pointed to
the output from
the package.json file of gsa. He said that it confirmed his
belief that there are too many dependencies to package;
"Even if you package everything, you will never ever have the right
combination of version of the various packages.
"
To many, that list looks daunting at best, impossible at worst, but
Smedegaard seemed
unfazed, noting several reasons to believe that those dependencies can
be handled. But
Hertzog pointed
out that the work is not of any real benefit, at least in his mind. He
cannot justify spending lots of time packaging those npm modules (then
maintaining them) for a single package "when said package was updated in Kali in a matter of
hours
". He thinks the distribution should focus its efforts
elsewhere:
He said that Debian is failing to keep up with the paradigm change in these other
ecosystems, which means that "many useful things
" are not
being packaged. Pirate Praveen agreed that there are useful things going
unpackaged, but disagreed
with Hertzog's approach of simply using the upstream download-and-build
process. Praveen thinks that a mix of vendoring (bundling) for
ultra-specific dependencies and creating
packages for more generally useful modules is the right way forward. It
comes down to distributions continuing to provide a particular service for their users:
One of the reasons Smedegaard felt that the dependencies for gsa could be
handled via Debian packages is that gsa (and other large projects) tend to
overspecify the versions required; in many cases, other versions (which
might already be packaged for Debian) work just fine. But figuring that
out is "a substantial amount of work
", Josh Triplett said in a lengthy
message. He
cautioned against the "standard tangent
" where complaints
about the number of dependencies for these types of projects are aired.
He said that disregarding a project's guidance on the versions for its dependencies is fraught, especially for dynamically typed languages where problems may only be detected at run time. For those ecosystems, the normal Debian practice of having only one version of a given library available may be getting in the way. Relaxing that requirement somewhat could be beneficial:
Triplett outlined the problems that developers encounter when trying to package a project of this sort. They can either try to make it work with the older libraries available in Debian, upgrade the libraries in Debian and fix all the resulting problems in every package that uses them, or simply bundle the required libraries. The first two are enormously difficult in most cases, so folks settle for bundling, which is undesirable but unavoidable:
Adrian Bunk is concerned with handling security problems in a world with multiple library versions. He said that these ecosystems seem to not be interested in supporting stable packages for three to five years, as needed by distributions such as Debian stable or Ubuntu LTS. More library proliferation (version-wise) just means more work for Debian when the inevitable CVE comes along, he said.
But Triplett said that he is not expecting there to be a lot of different library versions, but that at times it might make sense to have more than one:
That's different from requiring *exactly one* version of xyz, forcing all packages to transition immediately, and preventing people from uploading packages because they don't fork upstream and port to different versions of dependencies.
It seems safe to say that few minds were changed in the course of the discussion. Bunk and Triplett seemed to talk past each other a fair bit. And no one spoke up with some wild new solution to these problems. But the problems are not going to disappear anytime soon—or ever. Without some kind of shift, bundling will likely be the path of least resistance, at least until some hideous security problem has to be fixed in enough different packages that bundling is further restricted or prohibited. That would, of course, then require a different solution.
The approach currently being taken by Smedegaard, Praveen, and others to tease out the dependencies into their own packages has its attractions, but scalability and feasibility within a volunteer-driven organization like Debian are not among them. The size and scope of the open-source-creating community is vastly larger than Debian or any of its language-specific teams, so it should not come as a surprise that the distribution is not keeping up. Debian is hardly alone with this problem either, of course; it is a problem that the Linux distribution community will continue to grapple with.
Posted Jan 13, 2021 1:07 UTC (Wed)
by jafd (subscriber, #129642)
[Link] (38 responses)
Chasing all the little packages for all the little libraries is a huge waste of energy, both electrical and cognitive. In that ecosystem, it makes sense to package whole applications and ship all their node_modules with them, rather than every single bit and piece.
I hope Debian ends up doing the same, it's madness otherwise.
Posted Jan 13, 2021 7:50 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (36 responses)
Due to how bad js dependencies work, a single project can include multiple copies of the same library. So only within a single project there can be duplicates that need to be hunted down and fixed.
Fedora has none of those issues since it is a testing distribution for whatever new thing red hat wants to experiment this month.
Posted Jan 13, 2021 8:31 UTC (Wed)
by epa (subscriber, #39769)
[Link] (22 responses)
This suggests that the tracking of security bugs should follow the same granularity as packages. If you package an application from upstream, and it includes its own copy of some library code, it is upstream's job to check for vulnerabilities and publish new releases which fix them. Debian's job is to publish new versions of the library as a standalone package, for the benefit of applications that don't bundle it but do things "the Debian way". This seems like a fairer way to share out the work, so that upstream takes some of the extra costs of bundling, not just taking the benefits and leaving the distribution to do the extra work.
If upstream cannot commit to keep on top of security holes and publish updates for a five year period, or can't commit to providing a stable series rather than "that release is obsolete, upgrade everything", then I guess the application is not suitable for inclusion in a distribution with a 5 year maintenance window. That applies equally well whether or not the application bundles its own libraries.
Posted Jan 13, 2021 8:53 UTC (Wed)
by bangert (subscriber, #28342)
[Link] (1 responses)
say you have a function that is vulnerable to a privilege escalation, but the application in question is not calling it, so it is "safe".
apart from that, figuring out if a specific function bundled in a dependency is used in a project is orders of magnitudes harder, than figuring out if the project includes a given dependency (which itself can be difficult enough).
Posted Jan 13, 2021 10:06 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 13, 2021 9:05 UTC (Wed)
by Lionel_Debroux (subscriber, #30014)
[Link] (1 responses)
Posted Jan 13, 2021 18:27 UTC (Wed)
by hkario (subscriber, #94864)
[Link]
Posted Jan 13, 2021 9:52 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (17 responses)
So instead of patching all the copies of the library you now need to audit each and every function call to the vulnerable library, and then still patch it wherever it is vulnerable.
How is it any less work?
> it is upstream's job to check for vulnerabilities
They use npm, realistically they won't check for vulnerabilities.
But even if they did… would they backport the fixes to whatever version debian is using? If not, then it means they can't be used in a stable distribution, because they'd change everything when fixing the security bug as well.
By the way this behaviour is what pushed distributions to drop mysql and move to mariadb en masse. Because oracle doesn't backport fixes and doesn't say what the issues are. They just say "some security issue is fixed" and so you must upgrade.
> so that upstream takes some of the extra costs of bundling
Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.
The authors write them and forget about them in several cases. Also many of them write on windows and have no idea about linux.
> then I guess the application is not suitable for inclusion in a distribution with a 5 year
The issue is that following your criteria, basically no js application is suitable for inclusion.
Posted Jan 13, 2021 10:08 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (10 responses)
Not having to verify that an update doesn't also break functionality you depend on is less work than having to verify that everything still works fine with a nominally compatible security backport.
Posted Jan 13, 2021 11:52 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (9 responses)
Posted Jan 13, 2021 12:21 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (7 responses)
(No)
Posted Jan 13, 2021 14:45 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (6 responses)
Posted Jan 13, 2021 14:58 UTC (Wed)
by pizza (subscriber, #46)
[Link] (5 responses)
Sure. I lost soft scrollback in my linux consoles, because upstream's "fix" for the security hole was to strip out the feature entirely. Maintaining that feature would have required devising an independent fix (which IIRC never happened) or reverying the upstream changes, leaving that hole open.
Posted Jan 14, 2021 13:11 UTC (Thu)
by LtWorf (subscriber, #124958)
[Link] (4 responses)
Posted Jan 14, 2021 13:56 UTC (Thu)
by pizza (subscriber, #46)
[Link] (3 responses)
The linux kernel API that allowed for software-based console scrollback was completely removed as part of a "stable kernel security update" from my distribution.
Sure, the patch that removed it originated with the upstream kernel, but it was still an API change that the distro pushed out to its users.
Posted Jan 15, 2021 14:01 UTC (Fri)
by jafd (subscriber, #129642)
[Link] (2 responses)
I'm confident the world could be better without this very bizarre definition of "API" both you and mjg59 are using here.
Posted Jan 15, 2021 15:20 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
It was - TIOCLINUX subcode 13 (undocumented, but existed) let you control the scrollback from user space. There are documented ioctls for copy and paste you can use to programmatically extract data from a VC.
Posted Jan 22, 2021 5:19 UTC (Fri)
by HelloWorld (guest, #56129)
[Link]
Posted Jan 15, 2021 14:12 UTC (Fri)
by jafd (subscriber, #129642)
[Link]
I won't get into the madness of the neighboring thread (where they are discussing a user-visible change which is not an API change), but here's a very real scenario I'm seeing in a proprietary app that's my job to work on:
1) for $REASONS, we're stuck using a tool of the major version N. It's rather expensive to move to a tool of major version N+1 or N+2.
So, basically, until we find time to bring the tool to N+2, we're hosed according to "npm audit". Thankfully the tool is not user-facing, but you can see how "minor fixes" sometimes necessitate bigger changes a project may not be ready to embrace yet.
Posted Jan 13, 2021 13:03 UTC (Wed)
by khim (subscriber, #9252)
[Link] (5 responses)
Depends on the promises they make.
> Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.
Which, in turn, means that any JS application is POS which can and would be owned by malicious input presented to it.
That's why all these talks about trying to fix libraries, etc is pointless. You don't need to do it.
Instead you have to think what protections you have in place for the inevitable breakage which results. How these applications are compartmentalized is important. How data is protected from them is important.
How these apps are updates is NOT important because it's just pointless waste of time to try to plug holes in the sieve.
Posted Jan 13, 2021 13:37 UTC (Wed)
by pizza (subscriber, #46)
[Link] (1 responses)
That's easy. Not only do they make no promises, they explicitly disclaim warranties, including merchantability and fitness for any purpose.
Posted Jan 13, 2021 16:09 UTC (Wed)
by smoogen (subscriber, #97)
[Link]
Personally I look at all these Javascript/Python/etc to being similar to having every call in libc/libm etc as separate libraries versus 'bundled' together as a single item.
Posted Jan 15, 2021 3:37 UTC (Fri)
by VolkerWeissmann (guest, #144200)
[Link] (2 responses)
What does POS stand for?
Posted Jan 15, 2021 14:13 UTC (Fri)
by jafd (subscriber, #129642)
[Link]
Posted Jan 15, 2021 15:50 UTC (Fri)
by MKesper (subscriber, #38539)
[Link]
Posted Jan 13, 2021 9:01 UTC (Wed)
by zdzichu (subscriber, #17118)
[Link] (1 responses)
First, Fedora offers 13 months of support for each version.
Second, Fedora is fully fledged distribution with stable releases and various variants (for example Workstation, Server). It is NOT "a testing distribution".
I am a Fedora maintainer. I am not associated with Red Hat - most of us aren't.
Posted Jan 13, 2021 9:54 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link]
Posted Jan 13, 2021 10:15 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
It's just not something that Debian should tackle. Sure, it goes against the old model of Linux distros but times change...
Posted Jan 14, 2021 0:58 UTC (Thu)
by dvdeug (guest, #10998)
[Link]
Posted Jan 13, 2021 12:06 UTC (Wed)
by roc (subscriber, #30627)
[Link] (1 responses)
I don't know the NPM world, but if they don't have something like this already, they need it. Distro packagers could use such a tool to detect which packages contain bundled dependencies that need updating.
Posted Nov 6, 2023 11:43 UTC (Mon)
by j0057 (subscriber, #143939)
[Link]
Languages other than C have proper packaging support, and it's now up to the distros to adapt and make use of the information there is. Hunting down which application uses which library is really not that difficult, unless you're unable to tap into the metadata that a given packaging ecosystem already provides.
Posted Jan 13, 2021 12:12 UTC (Wed)
by cesarb (subscriber, #6266)
[Link] (6 responses)
Fedora offers support for 13 months, not 5 months.
> and if some js library turns out to have a security bug, then the security team is supposed to hunt down possibly hundreds of copies of the same library and fix the issue.
Fedora uses a trick to help with that: as explained at https://fedoraproject.org/wiki/Bundled_Libraries every package which includes a bundled library will have a "Provides: bundled(package) = version", so it's easy to find which packages have a given bundled library, and on which version.
Posted Jan 13, 2021 18:57 UTC (Wed)
by drawks (guest, #144158)
[Link] (5 responses)
This type of solution is a find technical solution if not 100% philosophically "clean" in the way debian intends. I would posit that such a solution would be fine for third party package providers and it would be a big step towards solving real world problems if Debian would just outline that this is a best practice for third parties and that first party debian packages would still be required to provide individually packaged packaged dependencies at build time. Softening the the limits on the number of versions of a library can be in the debian repositories would further make this a maintainable situation moving forward with a reasonable path for all various upstreams to cut between major versions of their dependencies.
at any rate thats my .02 maybe someone can tell me why I'm completely wrong :)
Posted Jan 13, 2021 19:35 UTC (Wed)
by smcv (subscriber, #53363)
[Link] (4 responses)
Built-Using is for the situation when we have some code in one Debian package, and we copy that code into a different Debian package during the second package's build. For instance, if you build busybox-static while you have libc6-dev_2.31-9 installed, version 2.31-9 of libc.a gets statically linked into the busybox-static binary. Now, as long as we have that copy of busybox-static in the archive, we also have to keep a copy of the glibc_2.31-9 source package around, otherwise we'll be missing the source code for some of our binaries (which is a LGPL violation).
Which upstream packages are vendored into the source package for which Debian packages could be tracked in a decentralized way by inventing a new metadata field, vaguely similar to Fedora's use of Provides, but at the moment it's tracked centrally instead: https://salsa.debian.org/security-tracker-team/security-t...
Posted Jan 13, 2021 20:09 UTC (Wed)
by drawks (guest, #144158)
[Link] (3 responses)
Posted Jan 14, 2021 17:07 UTC (Thu)
by smcv (subscriber, #53363)
[Link] (2 responses)
Posted Jan 15, 2021 3:37 UTC (Fri)
by VolkerWeissmann (guest, #144200)
[Link] (1 responses)
Posted Jan 15, 2021 4:36 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
Posted Jan 13, 2021 15:35 UTC (Wed)
by ncm (guest, #165)
[Link]
But, what about the rest? The problem to solve is things like Go packages that want to static-link their dependencies.
One way forward is to consider programs that want to static-link as being, effectively, scripts. So, like scripts, they are not packaged as complete executables. Their dependencies do not refer to a specific version, but provide a link to where to get any specific version wanted. The thing in /usr/bin is a script that checks for a cached build and, *if needed*: invokes the build tool, which uses local copies where found and usable, and downloads updates where needed, and links.
A package- or dist-upgrade doesn't pull new versions of dependencies; it just notes where local, cached copies are stale, and flushes. On next execution -- exactly as with scripts -- invalidated builds are scrapped, and rebuilt.
It means that to use a Go program, you need a Go toolchain, but that is not a large burden.
It means that the responsibility of the package system is only to flush caches when breaking or security updates happen. The target itself arranges to be able to start up quickly, on the 2nd run, much like the Python packages we already deal with.
Posted Jan 13, 2021 2:49 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
This already can happen during complex upgrades, and thousands of JS/Go packages will just make it inevitable.
Posted Jan 13, 2021 8:49 UTC (Wed)
by jond (subscriber, #37669)
[Link]
Posted Jan 13, 2021 7:48 UTC (Wed)
by Archimedes (subscriber, #125143)
[Link] (10 responses)
- So others try to rebuild his package (and not "download this docker image to rebuild us" desasters)
The user maybe looses the "newest" version but gains a more checked one and he gets one single way to update the whole systems and can be (at least more) sure that the package is integrated in *his* choices of the distribution.
I personally try to never download build packages from the developer but follow:
If all that fails, i reconsider hard if the package is that interesting and some times another one is then the better suited one.
Posted Jan 13, 2021 8:24 UTC (Wed)
by josh (subscriber, #17465)
[Link] (3 responses)
Posted Jan 13, 2021 9:48 UTC (Wed)
by Wol (subscriber, #4433)
[Link] (2 responses)
So yes it will dump a lot of work on the security people in a way, but the rule would become "any security fixes to a vendored module must include un-vendoring it".
Cheers,
Posted Jan 13, 2021 10:14 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (1 responses)
A vendored module might have patches that are not in the original.
In stable, updates are generally done only for security or serious bugs. In general any bug fix is allowed, but no new features or changes.
So you don't get surprises every day.
The download at compile time is awful because then you basically can't audit what's in the binary, can't have reproducible builds, can't ensure that it will still build tomorrow when the domain it is downloading from is sold to someone else.
Posted Jan 18, 2021 13:10 UTC (Mon)
by amck (subscriber, #7270)
[Link]
De-vendoring is a pain, but possible.
Updates to stable for security fixes need to be done on a stable branch, porting just the security fix to the update - not including any other changes relative to existing stable. This is painful but possible.
The versioning naming done makes this possible. e.g. apache2-bin on my stable machine has version "2.4.38-3+deb10u4", the +version part showing the change which falls between stable and testing as required.
The real challenge is (a) bundling (b) overwhelming human resources and the resolution manager in apt.
Posted Jan 13, 2021 13:41 UTC (Wed)
by kleptog (subscriber, #1183)
[Link] (5 responses)
This gives you reproducible builds because you can simply refer to the Debian recorded versions of those packages (the package build script could provide a list of "associated" packages with versions and hashes). You need to somehow filter the network traffic of the buildbot to only use access these repositories. This also means if you note that one of those "associate" packages has a security issue, you can immediately identify the Debian packages that used it and take measures.
This does mean Debian would need to manage a whole new set of repositories, but there are tools for this. Still a ridiculous amount of work, so it depends on whether the benefits are significant enough.
Posted Jan 13, 2021 21:15 UTC (Wed)
by plugwash (subscriber, #29694)
[Link]
I don't think this is correct.
Actually producing a package that builds and installs takes negligable time. What takes the time is.
1. Documenting/researching the copyright/license situation and removing anything that violates Debian standards.
Posted Jan 13, 2021 21:48 UTC (Wed)
by lkppo (guest, #97462)
[Link] (3 responses)
Posted Jan 14, 2021 3:34 UTC (Thu)
by pabs (subscriber, #43278)
[Link]
That said, there are already tools to convert language ecosystem packages to Debian source packages. I think Debian needs to go further than that and integrate more of those things into debhelper and similar.
Posted Jan 14, 2021 14:34 UTC (Thu)
by pj (subscriber, #4506)
[Link] (1 responses)
Posted Jan 26, 2021 4:42 UTC (Tue)
by calumapplepie (guest, #143655)
[Link]
The fact that apt doesn't permit per-package dependency trees is intentional. We don't want duplicate (or nearly-duplicate) files on our systems, our mirrors, or our security trackers. NPM's ability to let you have a dozen versions of every package just leads to headaches when you need to update one.
Posted Jan 13, 2021 8:16 UTC (Wed)
by xophos (subscriber, #75267)
[Link]
Posted Jan 13, 2021 9:25 UTC (Wed)
by Karellen (subscriber, #67644)
[Link] (4 responses)
It seems odd to me if this hasn't been part of the discussion already. Debian already includes packages for gtk-2, gtk-3 and gtk-4, and qt-4, qt-5 and qt-6 as two very prominent examples of parallel semver major releases, but I know I've seen quite a few other similar smaller examples browsing the package lists. Why would that not be an obvious part of the background for everyone involved?
Posted Jan 13, 2021 13:27 UTC (Wed)
by khim (subscriber, #9252)
[Link] (2 responses)
The big difference is in timing. GTK+ and QT+ have different versions, sure. But upstream supports these. There are parallel releases — but these are supposed to be parallel-installed. And the whole discussion is about things like Jenkins where two versions are not supposed to be installed and where about three months (but we don't know how much, exactly) is supposed to be called “Long Term Support” (I wish I was joking, I'm not).
Posted Jan 15, 2021 6:25 UTC (Fri)
by murukesh (subscriber, #97031)
[Link] (1 responses)
Posted Jan 22, 2021 2:48 UTC (Fri)
by khim (subscriber, #9252)
[Link]
You couldn't release anything weekly. There are just not enough time to find and fix problems in any new features. The most you can promise if you “release” weekly is “hey, it passed our automatic testing and therefore we hope it's not completely broken”. That's not a release. 20 years ago it would be called “alpha version” (not even “beta”: beta is supposed to be tested by some testing teams before release, not just pass CI bot). I'm not even sure that thing which they call “an LTS release” can be compared to “normal” stable release of XX century software which you can actually trust. The really strange result of all that activity is the fact that as “release cadence” shortens the actual, tangible, changes become less and less frequent. But resource consumption and number of bugs skyrockets. Simply because nobody can understand how the whole thing is supposed to work — and the amount of band-aids applied to pile of existing band-aids which ensure that the whole thing doesn't crash immediately upon startup is just endlessly grows. As Hoare (original inventor of Quicksort) said: there are two ways of constructing a software design — one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. What we are looking on in that pile of weekly “releases” and pile of “containers” is the end result of application of 2nd principle.
Posted Jan 13, 2021 21:50 UTC (Wed)
by plugwash (subscriber, #29694)
[Link]
If you just package a new version every time upstream pushes semver and don't go back and clean things up you can quickly end up with a whole pile of versions of a whole pile of libraries.
Some people advocate that it's better to package a new version first in parallel, then go back and port software, rather than trying to do the transition in one bang. The fear is without the carrot of getting the new version of your pet project into testing and/or the stick of having your pet project removed from testing, the work will never get done and the version proliferation will become a reality.
And it gets even worse if the upstream community doesn't see version proliferation or proliferation of tiny unmaintained packages as problems.
Posted Jan 13, 2021 10:29 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (18 responses)
Moreover, the talk of security updates is obsolete for a large part of ecosystem: as long as the upstream cares about portability to Windows and macOS, that upstream already has to care about security updates, so having a heads-up from a distribution security team is helpful, but it is ultimately a duplicate effort, except for the narrow case of a security vulnerability in a dependency specific to Linux.
Posted Jan 13, 2021 10:41 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (17 responses)
And even if they care, would they backport the security fixes or just tell users to upgrade to the next major version that changes everything as well?
The "use the latest" approach to security doesn't work for stable distributions.
Posted Jan 13, 2021 11:17 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (16 responses)
To repeat my original comment: "as long as the upstream cares about portability to Windows and macOS, that upstream already has to care about security updates". If upstream does not care about security on other platforms, then using this software is dangerous no matter what.
> The "use the latest" approach to security doesn't work for stable distributions.
As an upstream author I don't really care about distributions, all I care about is users who can use my software. So bundling everything into a single static binary is the approach that works best for me and for them (I don't even care about libc in this case, yay stable kernel interfaces and X11 / Wayland!), relying on a small set of platform libraries is the next best thing (there are weird corner cases, but more-or-less it works), giving a distributor full control over dependencies is a nightmare.
"Everything in a static bundle" works only for projects that are serious about tracking security bugs in their dependencies, of course. However portability to anything beyond Linux forces one to think about it.
Posted Jan 13, 2021 12:00 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (12 responses)
However developers also introduce incompatibilities all the time.
For example in gnome updates break extensions, so users after upgrading need to patch their extensions or not use them until someone else does this job.
I understand you don't care, but users care. Users don't use one software, they use hundreds of them and might have scripts to use many of them at once, so when they change, they need to fix whatever broke.
If your newest and greatest version doesn't have any feature that they need, they might not care about installing it to make you happy, and might prefer to keep using the old one so that they don't have to fix their scripts, configurations, and so on.
So that is why stable distributions exist, and why people use them, regardless of what you personally might wish.
> yay stable kernel interfaces and X11 / Wayland!
See, you appreciate stability yourself but don't want to provide it… Proving my point that stability is needed.
What if you use gtk3 and they tell you "there is a security bug but you need to use gtk4 for the fix"? Then you'd have to rewrite your software for gtk4, it would no longer work for any platforms that got dropped, you need to rewrite all of your tests and so on.
All in all, a very selfish comment :)
Posted Jan 13, 2021 12:31 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (9 responses)
Well, I don't break interfaces. Shocking! Problem solved.
> What if you use gtk3 and they tell you "there is a security bug but you need to use gtk4 for the fix"? Then you'd have to rewrite your software for gtk4, it would no longer work for any platforms that got dropped, you need to rewrite all of your tests and so on.
That's the gist of the problem, the CADT model: https://www.jwz.org/doc/cadt.html
The solution is simple. Do not depend on libraries whose authors regularly drop backward compatibility or don't provide extended security support for several major versions. These libraries are not worth the hassle.
Posted Jan 13, 2021 12:39 UTC (Wed)
by pizza (subscriber, #46)
[Link] (8 responses)
Ah, okay, so you just excluded pretty much everything web facing (and nearly the entirety of the javascript, python, go, and rust ecosytems)
Meanwhile, in the real world, the choice is "use this library" or "write something equivalent from scratch". Repeated dozens of times.
Posted Jan 13, 2021 12:51 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (7 responses)
Fortunately, not everything is security-sensitive, so the requirements may be relaxed there, and untrusted input boundaries are not that hard to contain if that's understood as a necessary part of engineering process.
> Meanwhile, in the real world, the choice is "use this library" or "write something equivalent from scratch". Repeated dozens of times.
Easy to reimplement, bad security policy => drop. Over time I find more and more code falls into this bucket.
Hard to reimplement, good security policy => use. Precious. Rare.
Easy to reimplement, bad security policy => use if fits exactly. Freebies. Nice to have.
Posted Jan 13, 2021 12:52 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link]
This one should read "good security policy"
Posted Jan 13, 2021 13:36 UTC (Wed)
by pizza (subscriber, #46)
[Link] (5 responses)
Welcome to software development in the real world, where the _only_ consideration [1] is "how much will it cost this quarter"
(FWIW I don't disagree with anything you have written; but at the end of the day, the folks paying our salaries decide what our development priorities are)
[1] Beyond what is legally mandated by law or private contract (eg required by your payment processor or insurance carrier)
Posted Jan 13, 2021 13:45 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (4 responses)
«This critical-and-not easy-to-replace library has N 0-day exploits with RCE per year. The choices are: 1) use it as is and sustain M break-ins, each conservatively estimated to cost $X / year in lost revenue, cleanups, damage control etc plus $Q for initial use of this library; 2) proactively monitor and patch vulnerabilities, estimated cost is $Y / year plus $Q for initial use of this library; 3) develop a replacement in-house, estimated cost is one-time $Z and $W / year maintenance»
Sure it takes some experience and data to start making these decisions, but once the data is there it becomes much easier to decide. Also over time (3) becomes cheaper as problems don't tend to be unique every time.
Posted Jan 13, 2021 14:47 UTC (Wed)
by pizza (subscriber, #46)
[Link] (3 responses)
You left out the cost of cleaning up after exploits found in your in-house library. Because your in-house code is far (far!) more likely to be full of holes than random F/OSS libraries that are used by a variety of other folks.
You also left out the fact that if it's in-house, $Z and $W are paid entirely by your organization, so while (3) might get smaller (relative to itself) over time, it will forever remain much larger than using a F/OSS library whose $Z and $W approach $0, as you're not shouldering the entire price of development and maintenance yourself.
In the end, most organizations go with (1) because they simply don't have the resources to do anything differently. (3) tends to only happen in areas where cost is not a primary consideration (eg military, medical, safety, and other highly regulated industries) or when there are specific requirements that cannot be met any other way. Varying degrees of (2) is where most organizations end up, typically balanced on the trailing edge of cost-benefit analyses.
Posted Jan 13, 2021 18:36 UTC (Wed)
by hkario (subscriber, #94864)
[Link] (2 responses)
and just because you develop in-house doesn't mean you have a). started from zero, b). don't open-source it so that others that do share your goals can contribute
Posted Jan 13, 2021 19:31 UTC (Wed)
by pizza (subscriber, #46)
[Link] (1 responses)
It's just that when you make that decision to go your own way, you can't assume that anyone else will ever contribute anything, much less shoulder the majority burden of developing/maintaining it.
Posted Jan 13, 2021 19:36 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link]
Posted Jan 13, 2021 12:32 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link]
BTW that's why I wouldn't touch anything in this ecosystem with a very long pole.
Posted Jan 13, 2021 12:36 UTC (Wed)
by pizza (subscriber, #46)
[Link]
I guess users need to put their money where their mouth is, eh?
Posted Jan 13, 2021 15:52 UTC (Wed)
by liori (guest, #117124)
[Link] (2 responses)
And as a user, I don't want to download my applications each from a different source. I don't want to remember to update them, each using a different tool. And I want to reduce the risks involved with updates breaking my workflows.
I want to outsource all of this to the distribution. And, for me, when an application is prepared so that outsourcing all this work to the distribution is easy is a feature I value.
This is why I'm using a Linux distribution and not, let say, LFS.
Posted Jan 13, 2021 19:45 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link] (1 responses)
That's a solved problem, I believe.
> And I want to reduce the risks involved with updates breaking my workflows.
I feel for you and many more (myself included) who are either stuck with old version of software and no support or forced to upgrade to incompatible version. As developers we ought to treat our users better and break less, that's for sure.
Posted Jan 14, 2021 13:15 UTC (Thu)
by LtWorf (subscriber, #124958)
[Link]
It is, for software that respects licenses, policies, and is not a nightmare to package.
But you just said you don't want to do any of those things with the software you write. So it's not solved for your software.
Posted Jan 13, 2021 15:22 UTC (Wed)
by pabs (subscriber, #43278)
[Link]
Posted Jan 13, 2021 16:26 UTC (Wed)
by cortana (subscriber, #24596)
[Link]
There's some info about this at <https://www.collabora.com/news-and-blog/blog/2018/07/04/d...>.
But I guess these are the easy cases: where the dependency has no other reverse-dependencies. Where there are other reverse-dependencies, vendoring becomes less acceptable...
Posted Jan 14, 2021 2:32 UTC (Thu)
by davecb (subscriber, #1574)
[Link]
--dave
TL;DR, in reverse order by date:
Posted Jan 14, 2021 16:07 UTC (Thu)
by gnu_lorien (subscriber, #44036)
[Link] (5 responses)
Looking at the landscape I hope that Debian sees this as something they must figure out how to do rather than rejecting these other software packaging methods as "bad" or as "not fitting our constraints." It's clear from the explosion of modern packaging systems that what developers want is the ability to vendor and choose the libraries they want to use for themselves. I know this has often been my biggest impediment to packaging software for any particular distribution too: That I only have the given choices of libraries.
As for my suggestion about actually solving the vendoring into packages issue, I would love to see the dependencies downloading into the source build and patches sent upstream that allow npm to work in an offline mode. Once you have the files locked in place as part of your source then the distro can work with the major package managers to extract the data that Debian needs. Right now there are clearly proprietary tools to do this and automatically scan for vulnerabilities. It would be nice to see them pulled into a larger free software effort.
Posted Jan 15, 2021 14:49 UTC (Fri)
by LibreTan (guest, #144205)
[Link] (2 responses)
Debian can only provide LTS for those Packages which provides LTS as upstream.
Example:
For all other software which does not provide LTS upstream it should work as OSTree layering only or work through Flatpak only.
If upstream is not providing LTS for their software then how can Debian?
Posted Jan 16, 2021 3:44 UTC (Sat)
by foom (subscriber, #14868)
[Link]
Now, there won't be CVEs for most issues that were fixed only in main devhead in most software, because nobody is really looking closely enough.
But in the rare case that there is such a CVE, generally Debian would just have to backport the patch.
Posted Feb 9, 2021 6:22 UTC (Tue)
by plugwash (subscriber, #29694)
[Link]
1. Many upstream's idea of "LTS" is far shorter than Debian's idea of regular support.
For Firefox they have resorted to moving to new upstream "LTS" release series within stable releases of the distro, that it just about tolerable for an end-user app like Firefox but it's really not reasonble for things that are key infrastructure components (and even for firefox it's problematic because firefox updates force rustc updates...........)
Posted Jan 22, 2021 1:08 UTC (Fri)
by nix (subscriber, #2304)
[Link]
Posted Jan 22, 2021 1:57 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Another solution might a Debian-hosted third-party package repository mirror. I know that people are going to attack me with pitchforks, but maybe adopting third-party package managers is actually a better idea than trying to fit everything into deb-src?
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
now assume, another part of the application has a remote code execution bug - boom, the "safe" privilege escalation is all of a sudden not so safe any more...
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
2) by now, some fourth-order dependencies of the tool had security vulnerabilities discovered, and the only path forward is to bump their major versions. Because JavaScript developers don't believe in backports. A dozen small libraries, but they are all over the place.
3) bumping them will need also bumping the third-order dependencies and so on, which is going to break the tool of version N, no longer maintained.
4) this kind of yak-shaving can take a lot of time, and I'm not counting the regressions that crept in because the migration documentation is far from being complete or accurate. Oh, and new versions of the tool produced buggy output which wasn't our fault. Back to the drawing board we are.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
mydependency = "1.2.3"
in your Cargo.toml, then cargo won't update it if you rebuild it.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
One (small to big) thing the developers of the software (which understandably are not interested in packaging for myriads of linux distributions) looses the input of others which need to incooperate his package in a larger infrastructure.
- so follow the configuration of the build (and add/remove features of it)
- follows the (build) dependencies (and might discover problematic ones)
- follows the run dependencies (like needs db or webserver or ... and provides configuration integration)
- puts the results in the "correct place" (depending on the style of the distribution).
- adds the package to the "system" like add to systemd, add to rsyslog (so depending on the distribution choices)
- 1. try the distribution one
- 2. try the "experimental/unstable" version of the package
- 3. download the source, try to build and (thus have to ) self maintain the version.
Debian discusses vendoring—again
Debian discusses vendoring—again
Wol
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
2. Waiting for the ftpmasters to validate that what you have done for 1.
3. Deciding what do do about dependency versions, trying to find the path between having piles of versions of the same thing hanging around (which is bad from a security point of view) and breaking stuff when a package is updated.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Those who don't value their privacy and security still have the option of installing that crap without the help of debian. It's even easy.
Debian discusses vendoring—again
I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it.
Debian discusses vendoring—again
According to https://www.jenkins.io/download/lts/, they seem to have settled on a 12-week cadence. But considering they say the normal release cadence is weekly (!), 12 weeks seems like a very long time in comparison (comparing with, say, Ubuntu, where the ratio of LTS/normal support duration is 60 months/9 months ~ 6.66).
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Hard to reimplement, bad security policy => isolate. This one is tough. Containerization helps somehow, but it's best if this set is empty.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
> Also over time (3) becomes cheaper as problems don't tend to be unique every time.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Solved problem in computer science
https://leaflessca.wordpress.com/2018/09/03/avoiding-an-n...
https://cacm.acm.org/magazines/2009/11/48444-you-dont-kno...
https://www.usenix.org/legacy/publications/library/procee...
https://research.swtch.com/vgo-mvs
https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-...
https://lwn.net/Articles/712318/
https://research.swtch.com/version-sat
Debian discusses vendoring—again
Debian discusses vendoring—again
Firefox provides ESR so include it in Debian Stable release.
Debian discusses vendoring—again
Debian discusses vendoring—again
2. While some projects may be on top of security issues in their dependencies I would wager the majority are not.
Debian discusses vendoring—again
Debian discusses vendoring—again
Go supports this ('go mod download'). We use it to optimize our Docker builds.