Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 12:45 UTC (Thu) by rahulsundaram (subscriber, #21946)
In reply to: Python cryptography, Rust, and Gentoo by LtWorf
Parent article: Python cryptography, Rust, and Gentoo

> There are lots of C libraries.

Yes there are some but unlike Rust or Python, there is no single place you can go to look for them and the tooling around installing or updating the dependencies isn't as straightforward.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 18:54 UTC (Thu) by logang (subscriber, #127618) [Link] (54 responses)

apt search xyz
apt install libxyz

How is that not straightforward?

The difference between C here and other languages isn't in the straightforwardness of finding and installing libraries but the difficulty of publishing them. Getting a library into pypy/whatever requires zero effort and there is zero quality control. Getting a library into a distribution is a lot harder and as a result the C libraries there tend to be of a higher quality; but the cost of this is that there are fewer choices.

However, I believe this is a good thing. No serious programmer should be choosing to depend on tiny and marginally maintained libraries that often don't care one wit about breaking their consumers. This can create very serious headaches down the road. Thought and care should be put into every dependency. Just because it's trendy these days to do otherwise, doesn't mean it's a good idea.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:31 UTC (Thu) by roc (subscriber, #30627) [Link] (34 responses)

Depending on distro libraries is a nightmare for developers. It creates so many problems.

When I make my software depend on a distro library, I now have to worry about:
-- Adding a step before the build that makes sure the library package is installed, e.g. providing instructions *per-distro* to install it manually, making my software harder to build
-- For distros that don't package the library (or package a version of it that's older than I need), providing instructions to build and install that library manually, making my software even harder to build
-- Making sure my software builds and runs with a range of library versions packaged by different distros and distro versions, potentially packaged in different ways with different directory layouts etc across distros
-- On platforms like Windows, iOS and Android (i.e. where almost all users are), where users cannot or will not build the software themselves and I need to provide binaries, and there definitely will not be a "distro package" I can use, I need to vendor the library myself anyway

Once I vendor the library for Windows/mobile, it's usually easier to just use that for Linux too. This is why big projects like Firefox/Chrome vendor everything.

Example: rr uses Capnproto, BLAKE2 and brotli, but we only depend on Capnproto as an external library; we vendor BLAKE2 and brotli. Even the single Capnproto dependency is horrible to deal with. For example we want to distribute rr binaries that work across distros, which means we want the rr build to support static linking of Canproto, but many distro Capnproto packages don't support static linking.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:18 UTC (Thu) by logang (subscriber, #127618) [Link] (25 responses)

Many of these problems already have solutions and haven't really been that big of a deal in the past. autoconf and Cmake have existed for a long time.

It's hard to avoid the issues with Windows/iOS/Android but if you're developing such an application you are probably not using C. Windows has always had a hellish story for libraries.

Firefox, for one, is distributed by many in an unvendored form.

But the overall theme is that the libraries you are using (or not) need help. If a library you want to use is good, and well maintained but not packaged, help them package it. If an algorithm isn't in a library, find an existing library that is a good fit and add to it (or, in the worst case start a new library, preferably that contains a lot more than just one algorithm). Or maybe the benefits of the latest and greatest compression algorithm are outweighed by older ones due to their accessibility. Develop with library versions that are commonly available, not the latest and greatest. Wait for features to mature (and possibly help them mature) before depending on them. If distros don't package a static library of something, send a patch so they can. Ultimately doing all this work allows you to write software that can be included in a distro and that should be the long term goal that is by far the easiest for all your users and easiest for the people that end up maintaining your software.

There is an awfully large amount of well written C software that has been written this way, has stood the test of time and will likely be around for a long time to come.

Yes, this can take more time and may mean you have to do more work in the short term, or wait for new features to percolate through the process. But the long term end result is a more sustainable ecosystem with a lot less work over the entire community. Vendoring something might make less work for you in the moment, but is more work for other people (or even your future self) down the line and doesn't solve anything for other people with the same problems as you.

If you want to write brittle broken software that needs constant attention and maybe doesn't even work at all in a few years, then yes, go ahead and keep doing things this way. Those that engineer things properly will still be around, still making constant progress.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:36 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> Ultimately doing all this work allows you to write software that can be included in a distro and that should be the long term goal that is by far the easiest for all your users and easiest for the people that end up maintaining your software.
Why mutilating your development process in order to confirm to arbitrary distro whims should be your goal?

A goal of an application developer is to provide value to users. Around 99% of users use iOS/Android/Windows/macOS, not classic distros.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:29 UTC (Thu) by roc (subscriber, #30627) [Link]

>If a library you want to use is good, and well maintained but not packaged, help them package it.

For all distros that any of my users might conceivably use? And then wait for several years for users to actually update to distro versions where the new package is present? No, that is completely unreasonable.

It's actually kind of breathtaking what you're suggesting here --- become a member of many different distro communities, learn all their different processes, persuade all of them to accept the library (what if they don't?), and stay engaged long term. All to avoid vendoring one library. I doubt there is a single person who has ever done this.

> Or maybe the benefits of the latest and greatest compression algorithm are outweighed by older ones due to their accessibility. Develop with library versions that are commonly available, not the latest and greatest. Wait for features to mature (and possibly help them mature) before depending on them.

Yes, creating worse performing, less capable software is definitely an option. I prefer not to.

> If you want to write brittle broken software that needs constant attention and maybe doesn't even work at all in a few years, then yes, go ahead and keep doing things this way.

Your preferred approach "needs constant attention and may not even work at all in a few years" --- you require me to pay constant attention to how distros are packaging my dependent libraries and regularly contribute to that process. In fact, because bugs are found and requirements change, any project with external dependencies requires ongoing attention.

My main project Pernosco is in Rust, has tons of dependencies (because it does a lot), and Rust+cargo have done a great job of managing those dependencies over the last five years. I am happy to keep on doing this this way.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 14:58 UTC (Fri) by MrWim (subscriber, #47432) [Link] (21 responses)

> But the overall theme is that the libraries you are using (or not) need help.

I agree. I think distro library management incentivises working around bugs, while cargo* incentivises helping upstream libraries.

I don't see this point brought up very often when discussing cargo, but I consider it to be one of the principal advantages of cargo.

> Develop with library versions that are commonly available, not the latest and greatest.

I think this is a sensible approach if you limit yourself to libraries, dynamically linked and available in distros. However I think it demonstrates how distro package managers incentivise *not* helping the libraries you're using.

Imagine you're writing some code and you come across a bug in a library you're using. You can choose to fix the bug upstream, or you can choose to work around it in your downstream code. With cargo you clone your dependency's git repo, fix the bug, push the change to a pull request upstream and update your Cargo.toml dependency to point at your new git revision with:

mydep = { git = "https://github.com/me/mydep.git", rev = "9f35b8e" }

You can leave it pointing at that specific revision until the upstream makes a new release at which point you update your Cargo.toml back to:

mydep = "3"

Fixing the bug (or adding the feature) upstream is the path of least resistance. Once you do it others who are using the library can benefit at the time of their choosing. In my mind many small fixes like this **is** the maturation process

Now what's the process with distro package managers? You're working on your new feature for your software. You come across a bug. You fix it upstream, you wait for it to get accepted upstream, you wait for upstream to make a new release and then you wait a few years for it to get into enterprise distros. Then you upgrade your infrastructure to a new major distro version, and only then can you deploy your new software that depends on this bug-fix/feature to get it in the hands of your users.

No, waiting, waiting and waiting is not going to fly. You want to help upstream but depriving your users of the new feature in your software for years is too high a cost to pay. So you work around the bug in your software and maybe if you've got time left over you also submit a fix upstream.

> Wait for features to mature (and possibly help them mature) before depending on them

I think this is the crux of my argument. cargo makes it easy to help features mature. Limiting yourself to distro repos means you have to wait for them to mature.

* possibly other language package managers too, but I'm not sure. I think cargo is best-in-class in this regard and some of its advantages may not apply to non-compiled languages/languages that don't statically link.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 15:21 UTC (Fri) by MrWim (subscriber, #47432) [Link]

Another way cargo encourages upstream collaboration is standardisation. I believe that the biggest barrier to open-source contribution is actually getting the software built in the first place. It's generally easy with rust, because it's always the same, and because the compilation model and cargo seem well designed. Check out the source code and:

cargo build

cargo takes care of finding and building the required dependencies. When you want to test your change it's cargo test. Finding the git repo for a dependency is easy too. It's linked to from its page on crates.io.

Note that nothing I've said above is related to rust as a language, it's all about tooling, but most importantly the culture of the rust community.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 21:35 UTC (Fri) by roc (subscriber, #30627) [Link]

This is a really good point.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 14:16 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (18 responses)

> Imagine you're writing some code and you come across a bug in a library you're using. You can choose to fix the bug upstream, or you can choose to work around it in your downstream code.

You can also send the patch to the distribution directly, or send it to both parties.

> Fixing the bug (or adding the feature) upstream is the path of least resistance.

You claim that the work of:

* forking
* fixing
* making a pull request to upstream
* going through multiple rounds until your patch is good enough to be included upstream and respects their standard of quality
* monitoring upstream's releases to know when a new release with your fix is out
* change your dependencies back to use upstream

is the path of least resistance

LOL.

It isn't. Want to know what people will do? Fork, patch, and point forever to their out of date fork.

Now THAT is the path of least resistance, it only includes 2 of the steps of the previous list. Of course now all this software might contain security vulnerabilities that will never be fixed.

> Now what's the process with distro package managers?

For a bugfix you can patch a package directly in the distribution.

> You fix it upstream,

Or directly downstream, as I said.

> No, waiting, waiting and waiting is not going to fly.

You assume that distributions and upstream projects are maintained by members of 2 different races. Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..

Also you are saying loads of incorrect things and forgetting that distributions can and do patch bugs out.

> I don't see this point brought up very often when discussing cargo, but I consider it to be one of the principal advantages of cargo.

As we have seen, your entire assumption of what the "path of least resistance" is, was completely wrong. So was the conclusion :)

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 14:42 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (2 responses)

> You can also send the patch to the distribution directly, or send it to both parties.

Not all patches should have this done. For example, those which change API are certainly not eligible for direct distro inclusion (IMO). Upstream should have a look before someone else ships a new API in their name for sure. Even for bugfixes, I don't know if my patch is an X/Y problem and that I'm actually patching a symptom and not a root cause. Upstream can certainly help improve these patches better than packagers (on average).

> You claim that the work of: … is the path of least resistance

IME? Yes. Because things like PyPI, crates.io, etc. make releases so easy, once it is in, the release shouldn't be *too* hard. Because I can't publish *my* crate to crates.io while pointing to my fork (unless I publish it as a crate of its own on crates.io, but that requires renaming due to collisions…which is then *more* work on my consuming side).

> For a bugfix you can patch a package directly in the distribution.

"the distribution". As if there's only one.

> Distribution maintainers can be fast, and upstream maintainers can take months to reply.

What does this have to do with anything? The reverse is also certainly possible.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 15:45 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (1 responses)

> those which change API are certainly not eligible for direct distro inclusion (IMO).

Those are not eligible to be accepted anywhere.

> Upstream can certainly help improve these patches better than packagers (on average).

There is an amount of software that distribution maintainers fork and become the "new upstream" because the actual upstream completely abandoned the project.

Yes upstream people abandon projects all the time. See python2 in red hat.

> IME? Yes. Because things like PyPI, crates.io, etc. make releases so easy, once it is in, the release shouldn't be *too* hard.

You can just point to your commit forever. Your software certainly wouldn't break.

> "the distribution". As if there's only one.

Uhm distributions share patches with each other.

> What does this have to do with anything? The reverse is also certainly possible.

It is possible, but you presented it as the only existing possibility.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 22:56 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

> Those are not eligible to be accepted anywhere.

APIs do change. I'm including things like "add a new enum variant for some new OpenSSL feature" kind of API changes in this. These patches certainly have a place, just not in some distro-specific patch (woe be unto anyone relying on distro packages being representative of upstream decisions in this case). See https://lwn.net/Articles/845448/ for a real-world case of this happening.

> There is an amount of software that distribution maintainers fork and become the "new upstream" because the actual upstream completely abandoned the project.

Why would I select such a project for a new dependency? All you're left with is projects that now need to port off of it (at least that would be my decision assuming there wasn't a distro-agnostic maintenance process set up). Case in point: scrot in Fedora (maintainer here). giblib and scrot were abandoned by upstream. The community picked up scrot, but left giblib alone. giblib starts to FTBFS. I don't want to maintain it; it's just a dependency of a project I do care about and I really don't want questions about it outside of that use. I file an issue upstream to port away from giblib. Still nothing. It's certainly not a patchset I want to maintain. So, scrot is currently dead in Fedora because I explicitly do *not* want to become an upstream.

As for something like Python2, yeah, that'll get some distro pickup. giblib? Not worth my time.

> You can just point to your commit forever. Your software certainly wouldn't break.

Not if I want to publish it anywhere (useful); crates.io requires that crates.io provide all your dependencies. I imagine PyPI is probably similar, but don't know.

> Uhm distributions share patches with each other

As if that's typical or even common (I'd like to see evidence). I've had to hunt down distro patches to our project that never got contributed to us, upstream. If they're not sharing upstream (or even filing issues about what they are patching), why would they share with each other? Granted, things have gotten better, but why must upstream be the one prodding here?

> It is possible, but you presented it as the only existing possibility.

Maybe that's MrWim you're thinking of?

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 15:08 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

> It isn't. Want to know what people will do? Fork, patch, and point forever to their out of date fork.
This is true. However, clicking a couple of buttons on a web form and submitting a PR is pretty easy.

> You assume that distributions and upstream projects are maintained by members of 2 different races. Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..
So your users must depend on a whim of an unpaid maintainer for months-to-years? That's a nice model.

> Also you are saying loads of incorrect things and forgetting that distributions can and do patch bugs out.
The other poster actually nails most obvious issues with distros. They simply suck for application writers.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 20:36 UTC (Sat) by roc (subscriber, #30627) [Link] (9 responses)

Submitting changes to distro library packagers instead of upstream would only be my last resort, for the case where upstream is completely unhelpful. (In which case I'll be looking to move off that dependency anyway.) It simply doesn't scale given the number of distributions in use. In fact I have never, ever done this.

What I *have* done, many times, is exactly what MrWim proposed: made local changes to a Rust library via a temporary Cargo [patch], and later submitted those changes upstream --- and had them accepted. The former step is indeed the path of least resistance and lets me make progress in my project. The latter step is justified because there is an ongoing maintenance cost to those patches, so reducing the number of them that we're carrying at any one time pays off long term. The review they get upstream is also valuable. I'm working through this right now at https://github.com/rayon-rs/rayon/issues/562 for example.

Python cryptography, Rust, and Gentoo

Posted Feb 14, 2021 22:10 UTC (Sun) by marcH (subscriber, #57642) [Link] (8 responses)

> made local changes to a Rust library via a temporary Cargo [patch], and later submitted those changes upstream --- and had them accepted. The former step is indeed the path of least resistance and lets me make progress in my project. The latter step is justified because there is an ongoing maintenance cost to those patches, so reducing the number of them that we're carrying at any one time pays off long term.

_This_ is "real" open-source: zero boundary between downloading/using/cloning/branching/forking/experimenting = complete freedom. This is why decentralized version control felt liberating. I would even argue that a project still stuck in centralized/medieval version control cannot really be considered open-source because of the added friction. And don't get me started on directories with sometimes long lists of *.patch files... never heard about branches?

Configuring and building C/C++ code at large is a nightmare and Linux distributions have been performing an amazing and critical job there. However to solve this they had to add layer(s) of indirection between software authors and users which adds friction and delays. So it's really not a surprise to see many authors trying to connect directly with their users. Random and recent example:

git clone some_python_project
pip install --editable .
<hack, test, hack, test>
git push new_pull_request

It should never be more complicated than this.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 6:43 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (3 responses)

Open source has a precise definition that has absolutely nothing to do with your preferred version control system.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 7:38 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

Next time at least pretend to try to get the point.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 10:54 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (1 responses)

Next time make one that makes sense?

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 8:44 UTC (Wed) by marcH (subscriber, #57642) [Link]

FYI, the usual behavior on this site when you don't understand something is to either ask or ignore.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 10:39 UTC (Wed) by MrWim (subscriber, #47432) [Link] (2 responses)

> This is why decentralized version control felt liberating.

Thanks for this analogy, there's definitely something to it. Something about not having to ask permission before acting, but instead being able to develop using the same tools as anyone, publish the results and have the results be judged instead.

Maybe what git is to a project, cargo is to a super project, or dependency graph. Hmm, that doesn't quite feel right because the versioning is still provided by git. It's the lockfile that extends git semantics to your entire dependency graph. Hmm, not sure if that's right, I'll have to think on it, but the analogy is food for thought.

As it is I'm a big fan of lockfiles, which can even be applied to whole distros and I believe that good tooling can unlock functional freedom, although I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 19:15 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

I think it's closer to true than you might expect. The GPL requires that source be provided in the preferred format for making modifications. That was meant to exclude things like generated code (rather than the material used to generate it) and obfuscated source, but it's not that far out to extend it to source being a copy of the version control system rather than just the raw source files.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 22:19 UTC (Thu) by marcH (subscriber, #57642) [Link]

Imagine a few people on the Internet want to collaborate and start some experimental branch for a project hosted with subversion _without_ bothering the maintainers. They would most likely start by cloning the subversion project to git or similar - and submit back to subversion only in the end (if ever).

Python cryptography, Rust, and Gentoo

Posted Jul 9, 2024 18:27 UTC (Tue) by jengelh (subscriber, #33263) [Link]

>_This_ is "real" open-source: zero boundary between downloading/using

What you describe sounds more like "collaboration" than "open-source": a project can achieve open-source status with so little as having a OSI license, even if its sole author rejects all your PRs because he thinks the project only makes sense for him.

>Configuring and building C/C++ code at large is a nightmare and Linux distributions have been performing an amazing and critical job there. However to solve this they had to add layer(s) of indirection between software authors and users

It is thanks to this indirection - or rather: integration -, that gives projects visiblity in the first place. Things could have turned out quite differently: If librsvg could not be integrated again after the language switch, the distro would either sport no svg support, no browser or no users (or any combination).

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 12:29 UTC (Mon) by MrWim (subscriber, #47432) [Link] (3 responses)

> Distribution maintainers can be fast, and upstream maintainers can take months to reply. It depends entirely on the specific project..

I agree. Distro maintainers can be fast, or not, and the same is true of upstream maintainers. Patches may be suitable for inclusion in stable distros, or not. Upstream maintainers may request changes to the patches, maybe in several rounds, or not.

The point is that with cargo the process is both asynchronous and uniform.

# Asynchronous

I don't need to wait for the maintainer to evaluate my patches for me to continue with my development. We can iterate over the best way to implement something at leisure without me making demands over their time "Please reply promptly this is very important to us", etc.

The other important asynchrony here is that the modifications we make needn't affect other users of the library until both the modifications and the applications are ready. This means that you can be sure that modifications you've made to a dependency don't break curl for example which also uses this dependency. Your changes are isolated until the point they are confirmed good. The way rust and cargo works is that you can even have multiple versions of the same dependency in your application at the same time.

You might respond that multiple versions of a single dependency on a system is a bad thing - it makes security validation and updates more difficult. I'd agree that the ideal state is that all applications on your system use the same version of a library. What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

I don't like it because I think it makes upgrading a library needlessly fraught, and places too much responsibility on the shoulders of the library maintainer, rather than spreading it more widely among the application maintainers. The library maintainer can review a patch and refuse it on various grounds like it contains bugs, or that it's already working as intended, or that it's likely to break compatibility, etc. They may also refuse it because they're worried that it will break a dependant package. This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

Instead, by removing the technical requirement that the library be upgraded in lockstep across the whole of the system we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version. Application maintainers can validate that an dependency upgrade has not broken their application and apply the upgrade then. This is a much narrower task than validating that a library hasn't broken any application, and the task falls on the person best placed to perform it.

The current process works well for patches that are both small and urgent. This applies to most buffer overflow or integer overflow fixes for example. I think a different process would be better for patches that are either not small or not urgent.

I haven't mentioned the importance of uniformity yet, but this reply is already long enough and has taken enough time so I'll worry about that later.

> > You fix it upstream,
>
> Or directly downstream, as I said.

I was responding to logang's comment. Specifically: "the libraries you are using (or not) need help.". I interpret "the libraries" to mean the upstream library. My point is that it's easier for one to help the upstream libraries when they've got them from cargo, rather than from the distro.

It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

I also believe in distros, but principally as distributors/integrators/maintainers of applications, rather than of libraries.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 6:57 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (2 responses)

> I don't need to wait for the maintainer to evaluate my patches for me to continue with my development.

And what if they completely reject your changes or require substantial changes?

This can be for several reasons.

What would you do at that point? You'd need to either give up using the feature you added or refactor your code.

> What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

The debian policy does not say anything against static linking vs dynamic linking. It does mandate that there can exist only 1 copy of a certain source within the archive. Browsers are granted exceptions because they are irreplaceable and do whatever they want.

> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

That's basically the opposite of Torvald's approach :D

> we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version.

Libraries that are made to support it are normally available in multiple versions for a period of time. Typical example is Qt. Qt4 has been removed only very recently. But point releases replace the previous version, because we don't want to do the go way of depending on a specific commit.

> It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

And when the user ends up with unpatched vulnerabilities because he downloaded some binary from some website because it wasn't included in a distribution because it violated every existing policy… how is the user served well by this?

He has to hope the author has a system in place to recompile when a CVE appears, that there is a repository set up to get the latest version. Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 8:28 UTC (Tue) by matthias (subscriber, #94967) [Link]

>> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.
> That's basically the opposite of Torvald's approach :D
No, it is exactly Torvalds' approach. The kernel policy is no regressions, yes. But Torvalds does not do the QA for all software that runs on linux. That would be simply impossible. He releases -rc versions of the kernel and asks everyone to test. If regressions are reported, the corresponding patches are reverted (or improved). But it is the job of the users (distros, cloud service providers, etc.) to do the testing and validation.

And it is definitely up to the distros to test whether a new kernel version works for them before they include it into the distribution.

The main difference between the kernel and many other projects is, that the kernel developers care much more about the regression reports received from the users and that regression reports have higher priority than new features.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 13:07 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> And what if they completely reject your changes or require substantial changes?

If we had gone with your proposal, now my distro is stuck with a patch rejected by upstream. Yay?

In this case, I rework the patch and adapt my code when I point to the next proposed patch. Same as step 1, just with a different baseline to diff from.

> Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

As if Linux is the only distribution platform for projects these days. You do realize that Linux (and the BSDs) are the oddballs out here, right? Pretty much everything else does vendoring or the like to a large extent. And if I want a turnkey release from my website, a tarball with dependencies embedded is the answer even for Linux without waiting for distros to churn on the new release (which generally takes a month or two to hit the unstable channels for our project; add a release cycle for the stable channel to have a chance).

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 0:46 UTC (Sat) by hunger (subscriber, #36242) [Link]

> Many of these problems already have solutions and haven't really been that big of a deal in the past. autoconf and Cmake have existed for a long time.

I have seen both autoconf and cmake referred to as a problem more often than as a solution:-)

Watch out: The C/C++ world has tooling for dependency management incoming as well. Sooner or later you will have similar problems with those languages depending on very specific versions of libraries as well. Developers want that stuff, so they will write it. In the end each developer gets the tooling she deserves;-)

> If a library you want to use is good, and well maintained but not packaged, help them package it.

Why? The package will either be too old, or have the functionality my program needs stripped out since some packager did not like a dependency it introduced. Or its crippled or patched to no longer work properly for my application. Or the necessary files for your buildsystem of choice are not installed. Or they can't be found. Or users have crippled the libraries to prevent some imaginary issue or another. I had to deal with all of the above in bug reports already. It is such a huge pain to debug this kind of issue.

As a developer you need to vendor libraries (at thebvery least as a fallback if system libraries are not found!), even when running on distributions that have official packages of the required libraries. And you will get bug reports due to incompatible libraries because some distro packager will unvendor your libraries for you, blissfully ignoring the documented requirements.

> There is an awfully large amount of well written C software that has been written this way, has stood the test of time and will likely be around for a long time to come.

That is survivors bias... tons of poorly written crap written in C got lost and good riddance! Undoubtedly a lot of code written in any other language will not stand the test of time either.

> But the long term end result is a more sustainable ecosystem with a lot less work over the entire community.

There is less code because it takes ages to write and maintain. Is that a benefit to the eco system as a whole? I doubt it: Making something hard does make the persistent people stick around, and many good programmers I know are lazy and easy to distract:-)

> Vendoring something might make less work for you in the moment, but is more work for other people (or even your future self) down the line and doesn't solve anything for other people with the same problems as you.

It is more work for me and other people down the line. You get so many more bugs due to library incompatibilities and such. Those produce extra work for the users, the packagers and the developers. We are just used to this work load, so we ignore it.

> If you want to write brittle broken software that needs constant attention and maybe doesn't even work at all in a few years, then yes, go ahead and keep doing things this way.

If you want brittle software now, then go ahead:-)

Seriously there is great and well tested C code out there. That is wonderful! There is also some great rust code out there. Also awesome! Great code is a wonderful thing to have in any language! In my mind code quality is not related to how hard it is to use 3rd party libraries in the programming language of choice.

I also do not want to see programming as an activity where you need to recite obscure texts that got cargo-culted down to you by your elders! And it is rare to find a medium sized project in C or C++ that does not do exactly that -- at the very least in some dark corner of its build system. Rust projects at least do have less dark corners. Part of the reason is of course that rust has not accumulated so much historical baggage yet:-)

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 11:14 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (7 responses)

> Adding a step before the build that makes sure the library package is installed, e.g. providing instructions *per-distro* to install it manually, making my software harder to build

You mean a README file with a list of dependencies? I'm sure people on a certain distribution know how to use their package manager.
If they don't know, they won't be compiling your software anyway, because they don't know how to install a compiler.

> For distros that don't package the library (or package a version of it that's older than I need), providing instructions to build and install that library manually, making my software even harder to build

Users of stable distributions are familiar with the issue.

> Making sure my software builds and runs with a range of library versions packaged by different distros and distro versions, potentially packaged in different ways with different directory layouts etc across distros.

That is incentive to:
1. Do not depend on amateur libraries that change API
2. Use autotools and let it figure out all this stuff

> On platforms like Windows

There are no package managers on windows. So that is a completely different situation. But anyway you won't be using the same binary on linux and windows.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 13:57 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (4 responses)

> There are no package managers on windows.

There are. They're not as mature as anything Linux has AFAIK, but there is at least (in no particular order):

- vcpkg (probably the most useful for the discussion at hand)
- Conan (CMake-based)
- chocolatey (binary-based, includes Visual Studio/MSVC packages)
- anaconda (scientific/Python oriented, but has other bits too)

I think there's another, but I can't remember it's name. There's also zero surprise from me if there are others I haven't heard of.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 21:58 UTC (Fri) by roc (subscriber, #30627) [Link] (3 responses)

You really need a single standard package manager, preferably shipped with the OS, so there is a high chance users already have it installed and "install package manager" doesn't just make your installation process longer.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 13:35 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (2 responses)

While true, I think what I'd do is use vcpkg for developer management, bundle everything up into a single package and ship that via normal means (installer/relocatable zip) and maybe chocolatey depending on the tool target audience.

Anaconda would probably be better if you're already in that realm though.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 20:39 UTC (Sat) by roc (subscriber, #30627) [Link] (1 responses)

vckpg looks cool, thanks for pointing to it. But it looks more like "cargo for C++" than a distro package manager.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 20:41 UTC (Sat) by roc (subscriber, #30627) [Link]

Of course, as such, it may be a good answer to the problem of "how do I consume C third-party libraries" which was the original issue before we got into a discussion of the "use distro packages" non-solution.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 21:56 UTC (Fri) by roc (subscriber, #30627) [Link] (1 responses)

> You mean a README file with a list of dependencies? I'm sure people on a certain distribution know how to use their package manager.

Sure. One problem is, that package list changes with each distribution and sometimes within versions of each distribution. So we only have instructions for Fedora and Ubuntu, and those instructions are wrong for some versions of those distros.

Anyway, every single step makes the software harder to build.

> Users of stable distributions are familiar with the issue.

So? The issues still exist.

> That is incentive to:
> 1. Do not depend on amateur libraries that change API

Distro policies are responsible for varying file layouts and naming conventions. Distros also make varying decisions about library versions and which features they enable at build time.

> 2. Use autotools and let it figure out all this stuff

Autotools are a nightmare and writing autotools feature tests for everything I care about would be a ton of extra work.

> you won't be using the same binary on linux and windows.

Indeed, but once you've done the work to build Windows binary you can reuse that work to build a Linux binary with vendored libraries.

People arguing that the Right Way to build C/C++ software is to make it Linux only, use distro libraries, do a ton of extra work, and downgrade the performance and functionality of that software to fit the shipped libraries, are not doing C/C++ any favours.

Python cryptography, Rust, and Gentoo

Posted Feb 13, 2021 8:12 UTC (Sat) by abartlet (subscriber, #3928) [Link]

> Sure. One problem is, that package list changes with each distribution and sometimes within versions of each distribution. So we only have instructions for Fedora and Ubuntu, and those instructions are wrong for some versions of those distros.

This got so bad for Samba (both the distribution versions to cover and the Samba versions to cover) that we ended up building a massive infrastructure to:
- Create Docker images for CI
- Test every build in all the supported distributions
- Publish a 'install dependencies for samba' script.

Just looking at the table here: https://wiki.samba.org/index.php/Package_Dependencies_Req... to see where this ends up.

Even the source data for those generated scripts, for a single release is quite complex: https://gitlab.com/samba-team/samba/-/blob/master/bootstr...

So for software of any serious size, it is not just a README with a list of dependencies. Furthermore, Samba has found we have to have configure checks looking for the library (otherwise folks complain that their build failed) and to make those fail by default (not 'auto-detect' and work around) because otherwise features just go missing.

All in all it is hard to argue that this is really a good vision to match.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:56 UTC (Thu) by roc (subscriber, #30627) [Link]

Making consumption of third-party libraries extremely painful is not a good way to address whatever downsides there are of depending on third-party libraries. In reality C/C++ programmers react to that pain by either vendoring libraries (with bad tools, which make updates expensive, which creates security and correctness hazards), or by reimplementation (which on average means lower quality because development effort is spread over more implementations).

For example in our Rust project (https://pernos.co) we use cargo-deny in CI to scan our dependencies for known CVEs and break the build if there is one. This is working very well. Nothing like it exists for C because the infrastructure for consuming third-party libraries in C is hopelessly fractured.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:16 UTC (Thu) by marcH (subscriber, #57642) [Link] (17 responses)

> apt search xyz
> apt install libxyz
> How is that not straightforward?

It seems straight-forward when you ignore all the work that distributions perform behind the scenes to achieve that result.

It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

It seems straight-forward if you've never debugged CMake or (much worse) autotools.

It seems straight-forward as long as you don't need different packages that require different versions of xyz.

It seems straight-forward as long as you don't try to use a package from another distro because it's missing on yours.

It seems straight-forward as long as you don't try to naively "upgrade" the LTS version of your distro with packages from a newer version of the _same_ distro.

If it's so straight-forward, why have brand new projects like flatpak, snap etc. just been created?

Code re-use, software distribution and maintenance is hard, really hard. I'm not claiming rust or anything else cracked that nut, far from it and downloading random code from the Internet (in _any_ language_) is of course a security disaster[*] Pretending on the other hand that this problem has already been solved is either dishonest or incredibly naive and probably why the entire industry is still so bad at this. Have you never heard about "DLL Hell?". We should all keep open mind, take interest in any new approach and ignore anyone recommending to keep doing what they've have been always been doing.

[*] latest and greatest fun: https://www.theregister.com/2021/02/10/library_dependenci...

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:36 UTC (Thu) by logang (subscriber, #127618) [Link] (6 responses)

>It seems straight-forward when you ignore all the work that distributions perform behind the scenes to achieve that result.

Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

>It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

That's an odd statement. I do that multiple times a week on more than a dozen machines.

>It seems straight-forward if you've never debugged CMake or (much worse) autotools.

I've done both. Not that hard.

>It seems straight-forward as long as you don't need different packages that require different versions of xyz.

If libraries are well maintained and care about not breaking their users, and support a range of their own dependencies (instead of essentially vendoring their own dependencies by insisting on a very specific version) this problem tends not to be that bad. Even in python, good well maintained libraries ensure they work on a wide range of python versions and with a range of versions of their own dependencies. But also, in general, long deep dependency trees should be avoided and pushed back against.

>It seems straight-forward as long as you don't try to naively "upgrade" the LTS version of your distro with packages from a newer version of the _same_ distro.

I've done this a lot. For the rare critical package, this is hard and should simply not be done. 9 times out of 10, it is easy.

> If it's so straight-forward, why have brand new projects like flatpak, snap etc. just been created?

No idea. But I avoid those like the plague. They don't solve any of my problems.

> Code re-use, software distribution and maintenance is hard, really hard. I'm not claiming rust or anything else cracked that nut, far from it and downloading random code from the Internet (in _any_ language_) is of course a security disaster[*] Pretending on the other hand that this problem has already been solved is either dishonest or incredibly naive and probably why the entire industry is still so bad at this. Have you never heard about "DLL Hell?". We should all keep open mind, take interest in any new approach and ignore anyone recommending to keep doing what they've have been always been doing.

Absolutely right. But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything. From a security, maintenance and longevity perspective the distros have been doing far better, which is why I always go to them first and strongly resist the newer trends to vendor everything.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:07 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

You know how many users we have using distro-provided deployment mechanisms? Zero (that I hear from). I hear from distributor maintainers and we work to accomodate building with external deps (because *I* care and testing against external versions is the easiest way to set warning bells for API changes coming down the pipe).

Existing deployments are on bespoke machines with oddball dependencies not packaged by distros. They use custom MPI builds that are tuned for the hardware. External libraries compiled against those MPI libraries. And other things too.

I agree that distros do a lot of work and I'm grateful for it, but the "everyone deploys to Linux (or FreeBSD)" mentality (this is especially rampant in the web world too) is short-sighted to me. We vendor the "core" libraries we need. I even made sure we do it properly: no untracked patches to them, mangle the symbols, soname, and header include paths to avoid conflicts with external copies, provide options to *use* external copies, etc. It's a lot of work.

And after all that? I would really rather just drop a `Cargo.lock` file in for stability and have CI churn on new releases to let me know of what's up in the future.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:52 UTC (Thu) by roc (subscriber, #30627) [Link] (3 responses)

> But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything

Effectively Rust wants developers to vendor everything, but a lot of work has gone into Rust+cargo to solve a lot of hard problems. For example:

cargo provides simple commands to update a dependency to the latest version, usually as simple as "cargo update" or "cargo update -p <library>".

cargo makes it easy to override a (possibly indirect) dependency with a patched version, via "[patch]".

rust-sec/advisory-db collects CVEs for Rust libraries and you can configure the cargo-deny tool to automatically break your build if one of your dependencies has an outstanding CVE.

Rust is designed so that by default linking multiple versions of the same library into a single binary works fine (always undesirable, but sometimes a necessary last resort).

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 23:31 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Effectively Rust wants developers to vendor everything

I don't think this is quite right. As I understand it "vendoring" means copying the source code of libraries you used into your own source tree rather than linking to the distribution-provided library at run time. As I understand it, there are a few problems with vendoring:

Library fragmentation. When people on the project discover something wrong with the library- a bug or missing feature- there's a tendency to patch it in the local copy rather than pushing the fix to upstream. Even if the project attempts to push changes upstream, the project may keep them if upstream is uninterested, resulting in fragmentation of the library.
Patch delays. If something upstream gets patched, it takes extra time and effort to push the patch out to all the projects that have vendored the library compared to patching the single distribution provided version. This is annoying with ordinary bugs and a serious danger with security bugs.
Hidden copies. It can be difficult even to track down all the projects that have vendored the library to make sure their copy has been fixed. This further slows patch rollout.

What Rust (and many other languages with their own dependency resolution systems) does is slightly different. They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale. That means they still have problems with patch delays but much less of one with library fragmentation or hidden copies than projects which have truly vendored libraries.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:07 UTC (Fri) by marcH (subscriber, #57642) [Link]

> As I understand it "vendoring" means copying the source code of libraries you used into your own source tree [...] tendency to patch it in the local copy rather than pushing the fix to upstream

In other words forking the source.

> They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale.

In other words forking the binaries but not the source.

There are probably a few other (and incompatible...) "definitions" of vendoring, for instance those that (wrongly) care about where the copy is hosted, but I don't think any other vendoring definition matters besides the two ways of forking above. I suspect we can get rid of that new word and not lose anything - actually gain some clarity. Please prove me wrong!

Duplication is not bad in itself, it's bad only when it leads to Divergence.
https://doc.rust-lang.org/book/ch03-01-variables-and-muta...

I stopped saying "Copy/Paste", now I say Copy/Paste/Diverge. Even the least technical managers understand he latter.

Examples of Duplication that keeps Divergence under control: cache invalidation, RCU, version control, snapshot isolation, transactional memory,...

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:23 UTC (Fri) by roc (subscriber, #30627) [Link]

Yes, I used the term loosely. Sorry about that.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:13 UTC (Fri) by marcH (subscriber, #57642) [Link]

> > It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

(I meant "apt upgrade)

> That's an odd statement. I do that multiple times a week on more than a dozen machines.

Then pause once and try to gauge about how many persons and lines of code you trust every time you install or upgrade a few dozens packages. Maybe pip and cargo won't look that bad after all.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 11:15 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (9 responses)

> It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

Uh?

How is this more risky than having 900 copies of libpng and hoping that all of them will be upgraded when inevitably the next buffer overflow is found?

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 17:03 UTC (Fri) by marcH (subscriber, #57642) [Link]

I'm not the one pretending to know which is more risky.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 22:12 UTC (Fri) by roc (subscriber, #30627) [Link] (7 responses)

Suppose we had a distro where all packages were built with Rust and statically linked the "png" crate, a CVE is issued for that crate, and a new minor version of "png" is available that fixes the bug. It would be very simple to scan all Cargo.lock files for all packages to see which ones are using a vulnerable version of "png". For each affected package, "cargo update -p png" would update to a non-vulnerable version. It would be easy to automate the entire process.

In this hypothetical distro you would also want to run 'cargo-deny' in CI to ensure that every time a package is built, the build fails if there is an outstanding CVE against one of its components.

The big picture here is that Rust+cargo standardize the build process and metadata to make managing dependencies much easier, more consistent and scalable.

(Of course we're ignoring the issue that you will have to do this much less frequently for a Rust PNG library because Rust code isn't prone to buffer overflows...)

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 2:14 UTC (Tue) by dvdeug (subscriber, #10998) [Link] (6 responses)

Let's compare that to what we have right now; we have a CVE in libpng, we upgrade the version of libpng in the distro, and fix all the packages without recompilation. That's already complex enough without literally recompiling almost every program on the system.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 2:55 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

And then some applications randomly break because of an ABI change.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 2:31 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (2 responses)

Recompilation has been known to randomly break applications as well. The art of a good security patch is that it doesn't change anything besides making the security fix.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 4:59 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> Recompilation has been known to randomly break applications as well.
Uhh.... Whut? Recompilation can't break applications, especially with repeatable builds. A bad fix that changes the API can certainly do that.

But it's way better than random breakages because the ABI has subtly changed.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 8:33 UTC (Wed) by geert (subscriber, #98403) [Link]

If the compilation is needed due to a change in a dependency, it is not a repeat of the previous build. If it was, there was no point in recompiling it.
If the compiler has changed, the recompiled application may behave differently, too.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 16:40 UTC (Tue) by foom (subscriber, #14868) [Link] (1 responses)

If we had the ability to cross-compile for slow target architectures, and proper build automation, recompiling everything that depends on libpng wouldn't need to be a problem.

Distributing the update to users might need some adjustments, too, in order to avoid massive bandwidth usage -- a good mechanism to send just binary deltas for the affected files would be more important than it is now.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 22:50 UTC (Thu) by flussence (guest, #85566) [Link]

It's a bit ironic that the software I *most* need cross-compilation for is the stuff most resistant to being cross-compiled…

(Bought more RAM than I thought I'd ever need. The compiler crashes because it runs out of i686 registers now. *sigh*)