Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 14:16 UTC (Sat) by LtWorf (subscriber, #124958)
In reply to: Python cryptography, Rust, and Gentoo by MrWim
Parent article: Python cryptography, Rust, and Gentoo

> Imagine you're writing some code and you come across a bug in a library you're using. You can choose to fix the bug upstream, or you can choose to work around it in your downstream code.

You can also send the patch to the distribution directly, or send it to both parties.

> Fixing the bug (or adding the feature) upstream is the path of least resistance.

You claim that the work of:

* forking
* fixing
* making a pull request to upstream
* going through multiple rounds until your patch is good enough to be included upstream and respects their standard of quality
* monitoring upstream's releases to know when a new release with your fix is out
* change your dependencies back to use upstream

is the path of least resistance

LOL.

It isn't. Want to know what people will do? Fork, patch, and point forever to their out of date fork.

Now THAT is the path of least resistance, it only includes 2 of the steps of the previous list. Of course now all this software might contain security vulnerabilities that will never be fixed.

> Now what's the process with distro package managers?

For a bugfix you can patch a package directly in the distribution.

> You fix it upstream,

Or directly downstream, as I said.

> No, waiting, waiting and waiting is not going to fly.

You assume that distributions and upstream projects are maintained by members of 2 different races. Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..

Also you are saying loads of incorrect things and forgetting that distributions can and do patch bugs out.

> I don't see this point brought up very often when discussing cargo, but I consider it to be one of the principal advantages of cargo.

As we have seen, your entire assumption of what the "path of least resistance" is, was completely wrong. So was the conclusion :)

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 14:42 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (2 responses)

> You can also send the patch to the distribution directly, or send it to both parties.

Not all patches should have this done. For example, those which change API are certainly not eligible for direct distro inclusion (IMO). Upstream should have a look before someone else ships a new API in their name for sure. Even for bugfixes, I don't know if my patch is an X/Y problem and that I'm actually patching a symptom and not a root cause. Upstream can certainly help improve these patches better than packagers (on average).

> You claim that the work of: … is the path of least resistance

IME? Yes. Because things like PyPI, crates.io, etc. make releases so easy, once it is in, the release shouldn't be *too* hard. Because I can't publish *my* crate to crates.io while pointing to my fork (unless I publish it as a crate of its own on crates.io, but that requires renaming due to collisions…which is then *more* work on my consuming side).

> For a bugfix you can patch a package directly in the distribution.

"the distribution". As if there's only one.

> Distribution maintainers can be fast, and upstream maintainers can take months to reply.

What does this have to do with anything? The reverse is also certainly possible.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 15:45 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (1 responses)

> those which change API are certainly not eligible for direct distro inclusion (IMO).

Those are not eligible to be accepted anywhere.

> Upstream can certainly help improve these patches better than packagers (on average).

There is an amount of software that distribution maintainers fork and become the "new upstream" because the actual upstream completely abandoned the project.

Yes upstream people abandon projects all the time. See python2 in red hat.

> IME? Yes. Because things like PyPI, crates.io, etc. make releases so easy, once it is in, the release shouldn't be *too* hard.

You can just point to your commit forever. Your software certainly wouldn't break.

> "the distribution". As if there's only one.

Uhm distributions share patches with each other.

> What does this have to do with anything? The reverse is also certainly possible.

It is possible, but you presented it as the only existing possibility.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 22:56 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

> Those are not eligible to be accepted anywhere.

APIs do change. I'm including things like "add a new enum variant for some new OpenSSL feature" kind of API changes in this. These patches certainly have a place, just not in some distro-specific patch (woe be unto anyone relying on distro packages being representative of upstream decisions in this case). See https://lwn.net/Articles/845448/ for a real-world case of this happening.

> There is an amount of software that distribution maintainers fork and become the "new upstream" because the actual upstream completely abandoned the project.

Why would I select such a project for a new dependency? All you're left with is projects that now need to port off of it (at least that would be my decision assuming there wasn't a distro-agnostic maintenance process set up). Case in point: scrot in Fedora (maintainer here). giblib and scrot were abandoned by upstream. The community picked up scrot, but left giblib alone. giblib starts to FTBFS. I don't want to maintain it; it's just a dependency of a project I do care about and I really don't want questions about it outside of that use. I file an issue upstream to port away from giblib. Still nothing. It's certainly not a patchset I want to maintain. So, scrot is currently dead in Fedora because I explicitly do *not* want to become an upstream.

As for something like Python2, yeah, that'll get some distro pickup. giblib? Not worth my time.

> You can just point to your commit forever. Your software certainly wouldn't break.

Not if I want to publish it anywhere (useful); crates.io requires that crates.io provide all your dependencies. I imagine PyPI is probably similar, but don't know.

> Uhm distributions share patches with each other

As if that's typical or even common (I'd like to see evidence). I've had to hunt down distro patches to our project that never got contributed to us, upstream. If they're not sharing upstream (or even filing issues about what they are patching), why would they share with each other? Granted, things have gotten better, but why must upstream be the one prodding here?

> It is possible, but you presented it as the only existing possibility.

Maybe that's MrWim you're thinking of?

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 15:08 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

> It isn't. Want to know what people will do? Fork, patch, and point forever to their out of date fork.
This is true. However, clicking a couple of buttons on a web form and submitting a PR is pretty easy.

> You assume that distributions and upstream projects are maintained by members of 2 different races. Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..
So your users must depend on a whim of an unpaid maintainer for months-to-years? That's a nice model.

> Also you are saying loads of incorrect things and forgetting that distributions can and do patch bugs out.
The other poster actually nails most obvious issues with distros. They simply suck for application writers.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 20:36 UTC (Sat) by roc (subscriber, #30627) [Link] (9 responses)

Submitting changes to distro library packagers instead of upstream would only be my last resort, for the case where upstream is completely unhelpful. (In which case I'll be looking to move off that dependency anyway.) It simply doesn't scale given the number of distributions in use. In fact I have never, ever done this.

What I *have* done, many times, is exactly what MrWim proposed: made local changes to a Rust library via a temporary Cargo [patch], and later submitted those changes upstream --- and had them accepted. The former step is indeed the path of least resistance and lets me make progress in my project. The latter step is justified because there is an ongoing maintenance cost to those patches, so reducing the number of them that we're carrying at any one time pays off long term. The review they get upstream is also valuable. I'm working through this right now at https://github.com/rayon-rs/rayon/issues/562 for example.

Python cryptography, Rust, and Gentoo

Posted Feb 14, 2021 22:10 UTC (Sun) by marcH (subscriber, #57642) [Link] (8 responses)

> made local changes to a Rust library via a temporary Cargo [patch], and later submitted those changes upstream --- and had them accepted. The former step is indeed the path of least resistance and lets me make progress in my project. The latter step is justified because there is an ongoing maintenance cost to those patches, so reducing the number of them that we're carrying at any one time pays off long term.

_This_ is "real" open-source: zero boundary between downloading/using/cloning/branching/forking/experimenting = complete freedom. This is why decentralized version control felt liberating. I would even argue that a project still stuck in centralized/medieval version control cannot really be considered open-source because of the added friction. And don't get me started on directories with sometimes long lists of *.patch files... never heard about branches?

Configuring and building C/C++ code at large is a nightmare and Linux distributions have been performing an amazing and critical job there. However to solve this they had to add layer(s) of indirection between software authors and users which adds friction and delays. So it's really not a surprise to see many authors trying to connect directly with their users. Random and recent example:

git clone some_python_project
pip install --editable .
<hack, test, hack, test>
git push new_pull_request

It should never be more complicated than this.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 6:43 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (3 responses)

Open source has a precise definition that has absolutely nothing to do with your preferred version control system.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 7:38 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

Next time at least pretend to try to get the point.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 10:54 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (1 responses)

Next time make one that makes sense?

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 8:44 UTC (Wed) by marcH (subscriber, #57642) [Link]

FYI, the usual behavior on this site when you don't understand something is to either ask or ignore.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 10:39 UTC (Wed) by MrWim (subscriber, #47432) [Link] (2 responses)

> This is why decentralized version control felt liberating.

Thanks for this analogy, there's definitely something to it. Something about not having to ask permission before acting, but instead being able to develop using the same tools as anyone, publish the results and have the results be judged instead.

Maybe what git is to a project, cargo is to a super project, or dependency graph. Hmm, that doesn't quite feel right because the versioning is still provided by git. It's the lockfile that extends git semantics to your entire dependency graph. Hmm, not sure if that's right, I'll have to think on it, but the analogy is food for thought.

As it is I'm a big fan of lockfiles, which can even be applied to whole distros and I believe that good tooling can unlock functional freedom, although I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 19:15 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

I think it's closer to true than you might expect. The GPL requires that source be provided in the preferred format for making modifications. That was meant to exclude things like generated code (rather than the material used to generate it) and obfuscated source, but it's not that far out to extend it to source being a copy of the version control system rather than just the raw source files.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 22:19 UTC (Thu) by marcH (subscriber, #57642) [Link]

Imagine a few people on the Internet want to collaborate and start some experimental branch for a project hosted with subversion _without_ bothering the maintainers. They would most likely start by cloning the subversion project to git or similar - and submit back to subversion only in the end (if ever).

Python cryptography, Rust, and Gentoo

Posted Jul 9, 2024 18:27 UTC (Tue) by jengelh (subscriber, #33263) [Link]

>_This_ is "real" open-source: zero boundary between downloading/using

What you describe sounds more like "collaboration" than "open-source": a project can achieve open-source status with so little as having a OSI license, even if its sole author rejects all your PRs because he thinks the project only makes sense for him.

>Configuring and building C/C++ code at large is a nightmare and Linux distributions have been performing an amazing and critical job there. However to solve this they had to add layer(s) of indirection between software authors and users

It is thanks to this indirection - or rather: integration -, that gives projects visiblity in the first place. Things could have turned out quite differently: If librsvg could not be integrated again after the language switch, the distro would either sport no svg support, no browser or no users (or any combination).

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 12:29 UTC (Mon) by MrWim (subscriber, #47432) [Link] (3 responses)

> Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..

I agree. Distro maintainers can be fast, or not, and the same is true of upstream maintainers. Patches may be suitable for inclusion in stable distros, or not. Upstream maintainers may request changes to the patches, maybe in several rounds, or not.

The point is that with cargo the process is both asynchronous and uniform.

# Asynchronous

I don't need to wait for the maintainer to evaluate my patches for me to continue with my development. We can iterate over the best way to implement something at leisure without me making demands over their time "Please reply promptly this is very important to us", etc.

The other important asynchrony here is that the modifications we make needn't affect other users of the library until both the modifications and the applications are ready. This means that you can be sure that modifications you've made to a dependency don't break curl for example which also uses this dependency. Your changes are isolated until the point they are confirmed good. The way rust and cargo works is that you can even have multiple versions of the same dependency in your application at the same time.

You might respond that multiple versions of a single dependency on a system is a bad thing - it makes security validation and updates more difficult. I'd agree that the ideal state is that all applications on your system use the same version of a library. What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

I don't like it because I think it makes upgrading a library needlessly fraught, and places too much responsibility on the shoulders of the library maintainer, rather than spreading it more widely among the application maintainers. The library maintainer can review a patch and refuse it on various grounds like it contains bugs, or that it's already working as intended, or that it's likely to break compatibility, etc. They may also refuse it because they're worried that it will break a dependant package. This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

Instead, by removing the technical requirement that the library be upgraded in lockstep across the whole of the system we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version. Application maintainers can validate that an dependency upgrade has not broken their application and apply the upgrade then. This is a much narrower task than validating that a library hasn't broken any application, and the task falls on the person best placed to perform it.

The current process works well for patches that are both small and urgent. This applies to most buffer overflow or integer overflow fixes for example. I think a different process would be better for patches that are either not small or not urgent.

I haven't mentioned the importance of uniformity yet, but this reply is already long enough and has taken enough time so I'll worry about that later.

> > You fix it upstream,
>
> Or directly downstream, as I said.

I was responding to logang's comment. Specifically: "the libraries you are using (or not) need help.". I interpret "the libraries" to mean the upstream library. My point is that it's easier for one to help the upstream libraries when they've got them from cargo, rather than from the distro.

It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

I also believe in distros, but principally as distributors/integrators/maintainers of applications, rather than of libraries.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 6:57 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (2 responses)

> I don't need to wait for the maintainer to evaluate my patches for me to continue with my development.

And what if they completely reject your changes or require substantial changes?

This can be for several reasons.

What would you do at that point? You'd need to either give up using the feature you added or refactor your code.

> What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

The debian policy does not say anything against static linking vs dynamic linking. It does mandate that there can exist only 1 copy of a certain source within the archive. Browsers are granted exceptions because they are irreplaceable and do whatever they want.

> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

That's basically the opposite of Torvald's approach :D

> we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version.

Libraries that are made to support it are normally available in multiple versions for a period of time. Typical example is Qt. Qt4 has been removed only very recently. But point releases replace the previous version, because we don't want to do the go way of depending on a specific commit.

> It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

And when the user ends up with unpatched vulnerabilities because he downloaded some binary from some website because it wasn't included in a distribution because it violated every existing policy… how is the user served well by this?

He has to hope the author has a system in place to recompile when a CVE appears, that there is a repository set up to get the latest version. Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 8:28 UTC (Tue) by matthias (subscriber, #94967) [Link]

>> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.
> That's basically the opposite of Torvald's approach :D
No, it is exactly Torvalds' approach. The kernel policy is no regressions, yes. But Torvalds does not do the QA for all software that runs on linux. That would be simply impossible. He releases -rc versions of the kernel and asks everyone to test. If regressions are reported, the corresponding patches are reverted (or improved). But it is the job of the users (distros, cloud service providers, etc.) to do the testing and validation.

And it is definitely up to the distros to test whether a new kernel version works for them before they include it into the distribution.

The main difference between the kernel and many other projects is, that the kernel developers care much more about the regression reports received from the users and that regression reports have higher priority than new features.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 13:07 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> And what if they completely reject your changes or require substantial changes?

If we had gone with your proposal, now my distro is stuck with a patch rejected by upstream. Yay?

In this case, I rework the patch and adapt my code when I point to the next proposed patch. Same as step 1, just with a different baseline to diff from.

> Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

As if Linux is the only distribution platform for projects these days. You do realize that Linux (and the BSDs) are the oddballs out here, right? Pretty much everything else does vendoring or the like to a large extent. And if I want a turnkey release from my website, a tarball with dependencies embedded is the answer even for Linux without waiting for distros to churn on the new release (which generally takes a month or two to hit the unstable channels for our project; add a release cycle for the stable channel to have a chance).