Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 12:29 UTC (Mon) by MrWim (subscriber, #47432)
In reply to: Python cryptography, Rust, and Gentoo by LtWorf
Parent article: Python cryptography, Rust, and Gentoo

> Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..

I agree. Distro maintainers can be fast, or not, and the same is true of upstream maintainers. Patches may be suitable for inclusion in stable distros, or not. Upstream maintainers may request changes to the patches, maybe in several rounds, or not.

The point is that with cargo the process is both asynchronous and uniform.

# Asynchronous

I don't need to wait for the maintainer to evaluate my patches for me to continue with my development. We can iterate over the best way to implement something at leisure without me making demands over their time "Please reply promptly this is very important to us", etc.

The other important asynchrony here is that the modifications we make needn't affect other users of the library until both the modifications and the applications are ready. This means that you can be sure that modifications you've made to a dependency don't break curl for example which also uses this dependency. Your changes are isolated until the point they are confirmed good. The way rust and cargo works is that you can even have multiple versions of the same dependency in your application at the same time.

You might respond that multiple versions of a single dependency on a system is a bad thing - it makes security validation and updates more difficult. I'd agree that the ideal state is that all applications on your system use the same version of a library. What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

I don't like it because I think it makes upgrading a library needlessly fraught, and places too much responsibility on the shoulders of the library maintainer, rather than spreading it more widely among the application maintainers. The library maintainer can review a patch and refuse it on various grounds like it contains bugs, or that it's already working as intended, or that it's likely to break compatibility, etc. They may also refuse it because they're worried that it will break a dependant package. This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

Instead, by removing the technical requirement that the library be upgraded in lockstep across the whole of the system we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version. Application maintainers can validate that an dependency upgrade has not broken their application and apply the upgrade then. This is a much narrower task than validating that a library hasn't broken any application, and the task falls on the person best placed to perform it.

The current process works well for patches that are both small and urgent. This applies to most buffer overflow or integer overflow fixes for example. I think a different process would be better for patches that are either not small or not urgent.

I haven't mentioned the importance of uniformity yet, but this reply is already long enough and has taken enough time so I'll worry about that later.

> > You fix it upstream,
>
> Or directly downstream, as I said.

I was responding to logang's comment. Specifically: "the libraries you are using (or not) need help.". I interpret "the libraries" to mean the upstream library. My point is that it's easier for one to help the upstream libraries when they've got them from cargo, rather than from the distro.

It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

I also believe in distros, but principally as distributors/integrators/maintainers of applications, rather than of libraries.

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 6:57 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (2 responses)

> I don't need to wait for the maintainer to evaluate my patches for me to continue with my development.

And what if they completely reject your changes or require substantial changes?

This can be for several reasons.

What would you do at that point? You'd need to either give up using the feature you added or refactor your code.

> What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

The debian policy does not say anything against static linking vs dynamic linking. It does mandate that there can exist only 1 copy of a certain source within the archive. Browsers are granted exceptions because they are irreplaceable and do whatever they want.

> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

That's basically the opposite of Torvald's approach :D

> we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version.

Libraries that are made to support it are normally available in multiple versions for a period of time. Typical example is Qt. Qt4 has been removed only very recently. But point releases replace the previous version, because we don't want to do the go way of depending on a specific commit.

> It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

And when the user ends up with unpatched vulnerabilities because he downloaded some binary from some website because it wasn't included in a distribution because it violated every existing policy… how is the user served well by this?

He has to hope the author has a system in place to recompile when a CVE appears, that there is a repository set up to get the latest version. Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 8:28 UTC (Tue) by matthias (subscriber, #94967) [Link]

>> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.
> That's basically the opposite of Torvald's approach :D
No, it is exactly Torvalds' approach. The kernel policy is no regressions, yes. But Torvalds does not do the QA for all software that runs on linux. That would be simply impossible. He releases -rc versions of the kernel and asks everyone to test. If regressions are reported, the corresponding patches are reverted (or improved). But it is the job of the users (distros, cloud service providers, etc.) to do the testing and validation.

And it is definitely up to the distros to test whether a new kernel version works for them before they include it into the distribution.

The main difference between the kernel and many other projects is, that the kernel developers care much more about the regression reports received from the users and that regression reports have higher priority than new features.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 13:07 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> And what if they completely reject your changes or require substantial changes?

If we had gone with your proposal, now my distro is stuck with a patch rejected by upstream. Yay?

In this case, I rework the patch and adapt my code when I point to the next proposed patch. Same as step 1, just with a different baseline to diff from.

> Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

As if Linux is the only distribution platform for projects these days. You do realize that Linux (and the BSDs) are the oddballs out here, right? Pretty much everything else does vendoring or the like to a large extent. And if I want a turnkey release from my website, a tarball with dependencies embedded is the answer even for Linux without waiting for distros to churn on the new release (which generally takes a month or two to hit the unstable channels for our project; add a release cycle for the stable channel to have a chance).