Debian discusses vendoring—again
Debian discusses vendoring—again
Posted Jan 13, 2021 1:07 UTC (Wed) by jafd (subscriber, #129642)Parent article: Debian discusses vendoring—again
Chasing all the little packages for all the little libraries is a huge waste of energy, both electrical and cognitive. In that ecosystem, it makes sense to package whole applications and ship all their node_modules with them, rather than every single bit and piece.
I hope Debian ends up doing the same, it's madness otherwise.
Posted Jan 13, 2021 7:50 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (36 responses)
Due to how bad js dependencies work, a single project can include multiple copies of the same library. So only within a single project there can be duplicates that need to be hunted down and fixed.
Fedora has none of those issues since it is a testing distribution for whatever new thing red hat wants to experiment this month.
Posted Jan 13, 2021 8:31 UTC (Wed)
by epa (subscriber, #39769)
[Link] (22 responses)
This suggests that the tracking of security bugs should follow the same granularity as packages. If you package an application from upstream, and it includes its own copy of some library code, it is upstream's job to check for vulnerabilities and publish new releases which fix them. Debian's job is to publish new versions of the library as a standalone package, for the benefit of applications that don't bundle it but do things "the Debian way". This seems like a fairer way to share out the work, so that upstream takes some of the extra costs of bundling, not just taking the benefits and leaving the distribution to do the extra work.
If upstream cannot commit to keep on top of security holes and publish updates for a five year period, or can't commit to providing a stable series rather than "that release is obsolete, upgrade everything", then I guess the application is not suitable for inclusion in a distribution with a 5 year maintenance window. That applies equally well whether or not the application bundles its own libraries.
Posted Jan 13, 2021 8:53 UTC (Wed)
by bangert (subscriber, #28342)
[Link] (1 responses)
say you have a function that is vulnerable to a privilege escalation, but the application in question is not calling it, so it is "safe".
apart from that, figuring out if a specific function bundled in a dependency is used in a project is orders of magnitudes harder, than figuring out if the project includes a given dependency (which itself can be difficult enough).
Posted Jan 13, 2021 10:06 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 13, 2021 9:05 UTC (Wed)
by Lionel_Debroux (subscriber, #30014)
[Link] (1 responses)
Posted Jan 13, 2021 18:27 UTC (Wed)
by hkario (subscriber, #94864)
[Link]
Posted Jan 13, 2021 9:52 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (17 responses)
So instead of patching all the copies of the library you now need to audit each and every function call to the vulnerable library, and then still patch it wherever it is vulnerable.
How is it any less work?
> it is upstream's job to check for vulnerabilities
They use npm, realistically they won't check for vulnerabilities.
But even if they did… would they backport the fixes to whatever version debian is using? If not, then it means they can't be used in a stable distribution, because they'd change everything when fixing the security bug as well.
By the way this behaviour is what pushed distributions to drop mysql and move to mariadb en masse. Because oracle doesn't backport fixes and doesn't say what the issues are. They just say "some security issue is fixed" and so you must upgrade.
> so that upstream takes some of the extra costs of bundling
Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.
The authors write them and forget about them in several cases. Also many of them write on windows and have no idea about linux.
> then I guess the application is not suitable for inclusion in a distribution with a 5 year
The issue is that following your criteria, basically no js application is suitable for inclusion.
Posted Jan 13, 2021 10:08 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (10 responses)
Not having to verify that an update doesn't also break functionality you depend on is less work than having to verify that everything still works fine with a nominally compatible security backport.
Posted Jan 13, 2021 11:52 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (9 responses)
Posted Jan 13, 2021 12:21 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link] (7 responses)
(No)
Posted Jan 13, 2021 14:45 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link] (6 responses)
Posted Jan 13, 2021 14:58 UTC (Wed)
by pizza (subscriber, #46)
[Link] (5 responses)
Sure. I lost soft scrollback in my linux consoles, because upstream's "fix" for the security hole was to strip out the feature entirely. Maintaining that feature would have required devising an independent fix (which IIRC never happened) or reverying the upstream changes, leaving that hole open.
Posted Jan 14, 2021 13:11 UTC (Thu)
by LtWorf (subscriber, #124958)
[Link] (4 responses)
Posted Jan 14, 2021 13:56 UTC (Thu)
by pizza (subscriber, #46)
[Link] (3 responses)
The linux kernel API that allowed for software-based console scrollback was completely removed as part of a "stable kernel security update" from my distribution.
Sure, the patch that removed it originated with the upstream kernel, but it was still an API change that the distro pushed out to its users.
Posted Jan 15, 2021 14:01 UTC (Fri)
by jafd (subscriber, #129642)
[Link] (2 responses)
I'm confident the world could be better without this very bizarre definition of "API" both you and mjg59 are using here.
Posted Jan 15, 2021 15:20 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
It was - TIOCLINUX subcode 13 (undocumented, but existed) let you control the scrollback from user space. There are documented ioctls for copy and paste you can use to programmatically extract data from a VC.
Posted Jan 22, 2021 5:19 UTC (Fri)
by HelloWorld (guest, #56129)
[Link]
Posted Jan 15, 2021 14:12 UTC (Fri)
by jafd (subscriber, #129642)
[Link]
I won't get into the madness of the neighboring thread (where they are discussing a user-visible change which is not an API change), but here's a very real scenario I'm seeing in a proprietary app that's my job to work on:
1) for $REASONS, we're stuck using a tool of the major version N. It's rather expensive to move to a tool of major version N+1 or N+2.
So, basically, until we find time to bring the tool to N+2, we're hosed according to "npm audit". Thankfully the tool is not user-facing, but you can see how "minor fixes" sometimes necessitate bigger changes a project may not be ready to embrace yet.
Posted Jan 13, 2021 13:03 UTC (Wed)
by khim (subscriber, #9252)
[Link] (5 responses)
Depends on the promises they make.
> Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.
Which, in turn, means that any JS application is POS which can and would be owned by malicious input presented to it.
That's why all these talks about trying to fix libraries, etc is pointless. You don't need to do it.
Instead you have to think what protections you have in place for the inevitable breakage which results. How these applications are compartmentalized is important. How data is protected from them is important.
How these apps are updates is NOT important because it's just pointless waste of time to try to plug holes in the sieve.
Posted Jan 13, 2021 13:37 UTC (Wed)
by pizza (subscriber, #46)
[Link] (1 responses)
That's easy. Not only do they make no promises, they explicitly disclaim warranties, including merchantability and fitness for any purpose.
Posted Jan 13, 2021 16:09 UTC (Wed)
by smoogen (subscriber, #97)
[Link]
Personally I look at all these Javascript/Python/etc to being similar to having every call in libc/libm etc as separate libraries versus 'bundled' together as a single item.
Posted Jan 15, 2021 3:37 UTC (Fri)
by VolkerWeissmann (guest, #144200)
[Link] (2 responses)
What does POS stand for?
Posted Jan 15, 2021 14:13 UTC (Fri)
by jafd (subscriber, #129642)
[Link]
Posted Jan 15, 2021 15:50 UTC (Fri)
by MKesper (subscriber, #38539)
[Link]
Posted Jan 13, 2021 9:01 UTC (Wed)
by zdzichu (guest, #17118)
[Link] (1 responses)
First, Fedora offers 13 months of support for each version.
Second, Fedora is fully fledged distribution with stable releases and various variants (for example Workstation, Server). It is NOT "a testing distribution".
I am a Fedora maintainer. I am not associated with Red Hat - most of us aren't.
Posted Jan 13, 2021 9:54 UTC (Wed)
by LtWorf (subscriber, #124958)
[Link]
Posted Jan 13, 2021 10:15 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
It's just not something that Debian should tackle. Sure, it goes against the old model of Linux distros but times change...
Posted Jan 14, 2021 0:58 UTC (Thu)
by dvdeug (guest, #10998)
[Link]
Posted Jan 13, 2021 12:06 UTC (Wed)
by roc (subscriber, #30627)
[Link] (1 responses)
I don't know the NPM world, but if they don't have something like this already, they need it. Distro packagers could use such a tool to detect which packages contain bundled dependencies that need updating.
Posted Nov 6, 2023 11:43 UTC (Mon)
by j0057 (subscriber, #143939)
[Link]
Languages other than C have proper packaging support, and it's now up to the distros to adapt and make use of the information there is. Hunting down which application uses which library is really not that difficult, unless you're unable to tap into the metadata that a given packaging ecosystem already provides.
Posted Jan 13, 2021 12:12 UTC (Wed)
by cesarb (subscriber, #6266)
[Link] (6 responses)
Fedora offers support for 13 months, not 5 months.
> and if some js library turns out to have a security bug, then the security team is supposed to hunt down possibly hundreds of copies of the same library and fix the issue.
Fedora uses a trick to help with that: as explained at https://fedoraproject.org/wiki/Bundled_Libraries every package which includes a bundled library will have a "Provides: bundled(package) = version", so it's easy to find which packages have a given bundled library, and on which version.
Posted Jan 13, 2021 18:57 UTC (Wed)
by drawks (guest, #144158)
[Link] (5 responses)
This type of solution is a find technical solution if not 100% philosophically "clean" in the way debian intends. I would posit that such a solution would be fine for third party package providers and it would be a big step towards solving real world problems if Debian would just outline that this is a best practice for third parties and that first party debian packages would still be required to provide individually packaged packaged dependencies at build time. Softening the the limits on the number of versions of a library can be in the debian repositories would further make this a maintainable situation moving forward with a reasonable path for all various upstreams to cut between major versions of their dependencies.
at any rate thats my .02 maybe someone can tell me why I'm completely wrong :)
Posted Jan 13, 2021 19:35 UTC (Wed)
by smcv (subscriber, #53363)
[Link] (4 responses)
Built-Using is for the situation when we have some code in one Debian package, and we copy that code into a different Debian package during the second package's build. For instance, if you build busybox-static while you have libc6-dev_2.31-9 installed, version 2.31-9 of libc.a gets statically linked into the busybox-static binary. Now, as long as we have that copy of busybox-static in the archive, we also have to keep a copy of the glibc_2.31-9 source package around, otherwise we'll be missing the source code for some of our binaries (which is a LGPL violation).
Which upstream packages are vendored into the source package for which Debian packages could be tracked in a decentralized way by inventing a new metadata field, vaguely similar to Fedora's use of Provides, but at the moment it's tracked centrally instead: https://salsa.debian.org/security-tracker-team/security-t...
Posted Jan 13, 2021 20:09 UTC (Wed)
by drawks (guest, #144158)
[Link] (3 responses)
Posted Jan 14, 2021 17:07 UTC (Thu)
by smcv (subscriber, #53363)
[Link] (2 responses)
Posted Jan 15, 2021 3:37 UTC (Fri)
by VolkerWeissmann (guest, #144200)
[Link] (1 responses)
Posted Jan 15, 2021 4:36 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
Posted Jan 13, 2021 15:35 UTC (Wed)
by ncm (guest, #165)
[Link]
But, what about the rest? The problem to solve is things like Go packages that want to static-link their dependencies.
One way forward is to consider programs that want to static-link as being, effectively, scripts. So, like scripts, they are not packaged as complete executables. Their dependencies do not refer to a specific version, but provide a link to where to get any specific version wanted. The thing in /usr/bin is a script that checks for a cached build and, *if needed*: invokes the build tool, which uses local copies where found and usable, and downloads updates where needed, and links.
A package- or dist-upgrade doesn't pull new versions of dependencies; it just notes where local, cached copies are stale, and flushes. On next execution -- exactly as with scripts -- invalidated builds are scrapped, and rebuilt.
It means that to use a Go program, you need a Go toolchain, but that is not a large burden.
It means that the responsibility of the package system is only to flush caches when breaking or security updates happen. The target itself arranges to be able to start up quickly, on the 2nd run, much like the Python packages we already deal with.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
now assume, another part of the application has a remote code execution bug - boom, the "safe" privilege escalation is all of a sudden not so safe any more...
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
2) by now, some fourth-order dependencies of the tool had security vulnerabilities discovered, and the only path forward is to bump their major versions. Because JavaScript developers don't believe in backports. A dozen small libraries, but they are all over the place.
3) bumping them will need also bumping the third-order dependencies and so on, which is going to break the tool of version N, no longer maintained.
4) this kind of yak-shaving can take a lot of time, and I'm not counting the regressions that crept in because the migration documentation is far from being complete or accurate. Oh, and new versions of the tool produced buggy output which wasn't our fault. Back to the drawing board we are.
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
Debian discusses vendoring—again
mydependency = "1.2.3"
in your Cargo.toml, then cargo won't update it if you rebuild it.
Debian discusses vendoring—again
Debian discusses vendoring—again
