Packaging Rust for Fedora

By Jonathan Corbet
October 28, 2022

Linux distributions were, as a general rule, designed during an era when most software of interest was written in C; as a result, distributions are naturally able to efficiently package C applications and the libraries they depend on. Modern languages, though, tend to be built around their own package-management systems that are designed with different goals in mind. The result is that, for years, distributors have struggled to find the best ways to package and ship applications written in those languages. A recent discussion in the Fedora community on the packaging of Rust applications shows that the problems have not yet all been solved.

The initial spark for the discussion was this Fedora 38 change proposal driven by Panu Matilainen. The RPM package manager has long carried its own internal OpenPGP parser for the management of keys and signatures for packages. This parser seemingly pleases nobody; the proposal describes it as "rather infamous for its limitations and flaws" and puts forward a plan to replace it with the Sequoia library, which is written in Rust (and which was covered here in 2020). The use of Rust provides the sort of safety net that is welcome in security-relevant code like this, but it can also be a red flag for developers who worry about how Rust fits into the distribution as a whole.

Inevitably, there were complaints about this proposal. Kevin Kofler, for example, asked why a library written in C had not been chosen. According to Matilainen, efforts to find such a library have been underway for years without success. The most obvious alternative, GPGME, is unsuitable because it is built around communicating with an external GPG process, "which is a setup you do NOT want in the rpm context where chroots come and go etc.". Neal Gompa agreed that the GPGME model creates pain in this context, and seemed to agree that there was no better alternative than Sequoia despite his own disagreements with the Rust community. "So here we are, in a subpar situation created by bad tools because nobody cares enough about security anyway".

Kofler went on to outline his problems with the Rust language. One of those was simply that it's yet another language to deal with, a complaint that didn't draw a lot of sympathy on the list. His other objection, though, struck closer to home:

The worst issue I see with Rust is the way libraries are "packaged", which just implies installing source code and recompiling that source code for every single application. (And as a result, the output obviously gets statically linked into the application, with all the drawbacks of static linking.) I consider a language with no usable shared library support to be entirely unpackageable and hence entirely useless.

Fabio Valentini, who works on packaging Rust crates for Fedora, pointed out that Sequoia is implemented as a shared library with a C ABI, so there will be no need to statically link any Rust code into RPM. He asked Kofler for any constructive suggestions he might have for improving the situation; that request was not addressed in Kofler's response. Fedora project leader Matthew Miller did have some thoughts, though.

Specifically, he agreed with Kofler that Rust applications may, in the end, just be "unpackagable". He mentioned his efforts with the Bevy game engine; he found that invoking the Cargo build system to obtain Bevy's build dependencies will fetch no less than 390 separate crates, about half of which are not currently packaged for Fedora. Trying to package such an application is sure to be painful but, he said, "this is what open source winning looks like". Cargo makes it easy to share and reuse software components, which is a great benefit, but it makes packaging all of those dependencies independently much harder. The fact that many of those dependencies are on specific versions of the crates involved makes the task harder yet.

All of this has led him to question the value of the work that is going into packaging Rust crates for Fedora. Instead, he said, Rust could be an opportunity to explore different approaches. "Something lightweight where we cache crates and use them _directly_ in the build process for _application_ RPMs". The implication was clearly that, by not trying to package all of the dependencies or ship dynamically linked executables, Fedora could work more directly with the Cargo ecosystem, save a lot of work that is (to him) of dubious value, and more easily get applications out to users. Fedora, he concluded, needs to adapt to remain relevant in the current development environment.

Few readers are likely to be surprised by the news that Valentini disagreed with this point of view. Bevy, he said, is a bit of a special case; most Rust applications are relatively easy to package for Fedora because the most popular crates are already packaged. Cargo and RPM, he added, work in similar ways, making the packaging job easier; in many cases, the RPM spec file can be generated automatically from the Cargo metadata. Meanwhile, the packaging effort brings all of the usual benefits, including cross-architecture testing, code and licensing review, and upstream contributions to make packaging easier in the future.

Trying to change the packaging process for Rust applications will, he said, make things worse instead. That is what happened with both Node.js and Java, he said (some of the Java discussions were covered here in June). Overall, he concluded, the situation with Rust is relatively good, and trying to do something other than "plain RPM packages" is likely to create more problems than it solves.

Kofler, instead, decried the ease with which Rust allows the addition of dependencies, calling the result "dependency hell". Rather than Fedora adapting to Rust, he said, Rust is going to have to adapt to become more relevant to Linux distributions. Gompa was not optimistic about that happening, though, saying that his efforts in that direction had met significant resistance in the past.

The conversation wound down at that point without any definitive conclusions. There is one relevant point that wasn't addressed that is worth considering, and which is highlighted by the use of Sequoia in RPM. Language-specific environments can work nicely as long as the developer sticks with the language in question; they can fall down when faced with the need to combine code written in multiple languages. At that point, the distribution model, which tries to make all packages work well together, shows its value. Given that the Rewrite The World In Rust Project is destined to take years to reach its conclusion, it seems likely that the number of mixed-language applications will increase for some time, and distributors will need to be able to package and ship those applications.

For the time being, the packaging of Rust crates for Fedora seems likely to continue without significant changes. But the topic of the intersection between distribution and language-specific package managers seems destined to reappear regularly for the indefinite future. Finding a way to make these independent ecosystems interact more smoothly will not be easy, but it would be beneficial to all involved; it is a problem worth working on.

to post comments

Packaging Rust for Fedora

Posted Oct 28, 2022 15:33 UTC (Fri) by q_q_p_p (guest, #131113) [Link] (41 responses)

"shared library with a C ABI" but without C headers is useless or PITA.

Rust doesn't even support separating interface from implementation, and wastes resources be requiring every dependency to be recompiled for every program.

Now rewrite LLVM in Rust - what will you get? Few hours of compilation for every program that depends on it. Have Fun.

Packaging Rust for Fedora

Posted Oct 28, 2022 16:47 UTC (Fri) by tchernobog (subscriber, #73595) [Link] (40 responses)

> "shared library with a C ABI" but without C headers is useless or PITA.

cbindgen is able to generate the C/C++ headers for your library from the Rust code. So I don't get your point here.

> Rust doesn't even support separating interface from implementation,

Which is only needed in languages designed in the '70s - '80s, like C?

Even C++ is moving towards not requiring this anymore, with C++20 modules.

The performance hit is negligible, if not actually beneficial because the compiler has more information earlier (e.g. to mark what to inline).

> and wastes resources be requiring every dependency to be recompiled for every program.

This is where a global cache, like it was proposed for Fedora, would be appreciated.

The problem is not trivial because of Rust package features; but it's not like a change in, say, the features enabled in OpenSSL during the ./configure step might not force you to have to recompile all reverse dependencies. All of this can be resolved by just being a bit disciplined and writing modules for Rust rather than compile-time features. It's a software engineering problem, not a language issue.

And let's recall that languages such as C++ don't have a well-defined ABI anyway (even name mangling is not part of the standard). C gets away with it mostly because a lot of stuff is missing from function signatures. But I've lost count of the amount of times I saw breakage at runtime because of it.

tl;dr: I don't think Rust here is much different than other compiled languages, if cargo is integrated properly with the rest of the packaging system.

> Now rewrite LLVM in Rust - what will you get? Few hours of compilation for every program that depends on it. Have Fun.

Just use shared libraries? You seem to be under the impression Rust doesn't support them.

Packaging Rust for Fedora

Posted Oct 28, 2022 17:51 UTC (Fri) by khim (subscriber, #9252) [Link] (8 responses)

> Just use shared libraries? You seem to be under the impression Rust doesn't support them.

Rust doesn't really support shared libraries. swift does, though and Rust-the-language forces traits model which makes that support not impossible in the future.

That's hard task, though, not something you may demand while Rust developers are doing that development on shoestring budget.

Packaging Rust for Fedora

Posted Oct 28, 2022 19:33 UTC (Fri) by tchernobog (subscriber, #73595) [Link] (7 responses)

You can use #[repr(C)] if you want a stable ABI now.

Else, you can opt to stick with the same rustc version for a few years, and the unstable ABI producing dylibs (".rlib") will be enough.

I mean, it's not like Perl gets upgraded on my Debian system without hundreds of modules going through a mass rebuild. And C++ ABI got calmer only with libstdc++ versioning.

I am just saying: it's not a particularly new film we are seeing. Nor one which has an unsolvable scenario, given enough willingness to fix it.

Packaging Rust for Fedora

Posted Oct 28, 2022 21:08 UTC (Fri) by khim (subscriber, #9252) [Link] (6 responses)

> You can use #[repr(C)] if you want a stable ABI now.

You can't. Ideomatic Rust is built on top Option<T> and Result<T, E> (some language construct don't even accept anything else!) — and these are not #[repr(C)].

Actually you can't even use other generic types without language support similar to what Swift did.

That, basically, means that instead of using shared libraries as, well, libraries, you are creating modules, boundaries of which are almost as hard to cross as boundaries between processes.

At this point making shared libraries makes no sense. Too much pain for too little gain. Better to link in network library and go with multiple processes.

> And C++ ABI got calmer only with libstdc++ versioning.

No. It got calmer when C++ developers have decided: yes, we do want to support shared libraries. Rust developers have never done that.

> Else, you can opt to stick with the same rustc version for a few years, and the unstable ABI producing dylibs (".rlib") will be enough.

This would have worked if Rust releases, like GNAT releases, happened once a year. And even that would have been problematic (I don't think any distribution releases too many shared Ada libraries). But they are released 8 times a year. That's too much churn to realistically support.

You can't go against upstream, basically. If upstream doesn't want to support shared libraries and actively makes that support problematic then they wouldn't be supported.

Packaging Rust for Fedora

Posted Oct 30, 2022 13:31 UTC (Sun) by ssokolow (guest, #94568) [Link] (4 responses)

No. It got calmer when C++ developers have decided: yes, we do want to support shared libraries. Rust developers have never done that.

Whether or not to build a cdylib from a Rust project is certainly a choice upstream makes, but also bear in mind in mind that, unless things have changed since 2012 on the ABI compatibility front, many of the core issues are just as unsolved in C++... it's just less idiomatic to rely on things like templates in C++ than on things like generics in Rust. (See The impact of C++ templates on library ABI by Michał Górny from 2012 for more on that.)

...not to mention, Rust's dependency counts are misleading when you're trying to make comparisons, because C and C++ are rife with bespoke reimplementations and single-header libraries that maintainers just take in stride, rather than trying to split out into their own packages. (See the Gotta go deeper section of Alopex's Let's Be Real About Dependencies for details on that.)

Packaging Rust for Fedora

Posted Oct 30, 2022 14:50 UTC (Sun) by khim (subscriber, #9252) [Link] (3 responses)

> it's just less idiomatic to rely on things like templates in C++ than on things like generics in Rust.

No. The important thing is standard library. It's ideomatic to use it in both C++ and Rust.

And while it's called Standard Template Library libstdc++ developers have decided that they would find a way to make it work as shared library. And they did. Yes, templates are injected in your code, not in the shared library, but certain discipline on the side of libstdc++ developers made it possible.

Swift as we all know picked different road, compiler-based. But, again: standard types and standard templates are usable when you build shared library.

In Rust land… Rust developers have never done that and not even core is offered in a way suitable for use in shared libraries.

That means that shared-library-Rust doesn't even remotely resemble normal std-based Rust. While in C++ the part which can be used as shared library is very much a subset of what you would do in normal C++.

That's pretty big difference: while in C++ it's possible to write code which would be acceptable both in normal C++ project and in shared library, too… in Rust it's just impossible. Without ability to just return Result or accept Option your code is not Rust, it's different language.

In C++, on the other hand, it's perfectly fine to accept std::option or pass std::unique_ptr even in function which is part of shared library.

Packaging Rust for Fedora

Posted Oct 30, 2022 15:02 UTC (Sun) by ssokolow (guest, #94568) [Link] (2 responses)

Fair point... though an argument has also been made that C++'s focus on ABI stability is killing the standard library.

Packaging Rust for Fedora

Posted Oct 30, 2022 16:51 UTC (Sun) by khim (subscriber, #9252) [Link] (1 responses)

There is no free lunch. Either you have ABI-stable standard library or shared libraries are, basically, impossible.

And it's not as if Rust standard library can do sudden, radical, changes. It just wasn't in use for as long as C++ one.

Packaging Rust for Fedora

Posted Oct 30, 2022 17:09 UTC (Sun) by ssokolow (guest, #94568) [Link]

There is no free lunch. Either you have ABI-stable standard library or shared libraries are, basically, impossible.

Or you marshal your data in some way, like the abi_stable crate does for enabling dlopen-able libraries using higher-level Rust constructs via Rust-Rust FFI through the C ABI.

And it's not as if Rust standard library can do sudden, radical, changes. It just wasn't in use for as long as C++ one.

The Rust standard library has two advantages over C++ on that front:

Static linking and no stable ABI (eg. Rust 1.0 didn't do automatic structure packing. That was added later.)
Stronger API encapsulation (Rust has already done things C++ balked at, such as Rust 1.36 swapping out its original, slower HashMap implementation for a vendored copy of a port of Google's SwissTable. There's currently a PR being refined to replace the internals of std::sync::mpsc with a vendored copy of crossbeam-channel.)

Packaging Rust for Fedora

Posted Oct 31, 2022 11:44 UTC (Mon) by DianaNites (subscriber, #160945) [Link]

`Option<T>` is FFI safe in certain situations. https://doc.rust-lang.org/stable/std/option/index.html#re... https://doc.rust-lang.org/nomicon/ffi.html#the-nullable-p...

Packaging Rust for Fedora

Posted Oct 28, 2022 19:29 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> Even C++ is moving towards not requiring this anymore, with C++20 modules.

Eh. Splitting still has benefits. The isolation is not as air-tight as some might expect from other languages. For example if you (privately) import module X in your module, some compilers may require that when importing your module, they *also* need to be told where the bits for module X are (there's no way to say "I need X, but it doesn't affect by interface" because what affects the interface is…not an exhaustive list typically). So if X is indeed something that doesn't affect your interface (which includes `sizeof()` any public types), putting it into an "implementation unit" can help consumers not need to care (as much) about X.

I do agree about the reaction of there being no separation. Basically only C and its immediate offspring have it and it is…not a benefit overall. The lack of proper namespacing and isolation is what caused that. Rust has good namespacing and isolation, so it's not actually that much of a problem there.

Packaging Rust for Fedora

Posted Oct 30, 2022 1:53 UTC (Sun) by dvdeug (subscriber, #10998) [Link] (27 responses)

> > Rust doesn't even support separating interface from implementation,

> Which is only needed in languages designed in the '70s - '80s, like C?

I'd say separating interface from implementation is needed in every large enough program, at least conceptually. C doesn't do it great; the .h file doesn't stop all symbols from leaking all over the place, and has only hacky ways of hiding information (you can't have a struct without making the details available, though you can have a pointer to a struct without the details for a struct.) Java gets away with it because class files carry the interface information and prohibit abuse of implementation details (at a computer level) and JavaDoc shows just the implementation.

I'm not that familiar with Rust, but I'd say a good language for software engineering needs at least a JavaDoc substitute (for humans) and some effort to keep users from accidentally using implementation details, if it's not going to separate interface from implementation in code. I don't know why everyone has gone away from separate interfaces, but I don't think it's been a good decision.

Packaging Rust for Fedora

Posted Oct 30, 2022 10:22 UTC (Sun) by ssokolow (guest, #94568) [Link] (7 responses)

They've certainly got the former. Rust's JavaDoc equivalent is unadventurously called rustdoc, it's installed as part of the base toolchain, it doesn't require you to manually opt into documenting things like Sphinx for Python does, you can generate and view docs for your entire dependency tree locally with cargo doc --open, and every crate that's published to crates.io automatically gets rustdoc run on it and the result hosted publicly at docs.rs. (Though the docs.rs run is configured to only display the README for binary/non-library crates, unlike cargo doc.)

As for the latter, I don't know what your requirements are, but they certainly make a strong effort. Things like type inference are only allowed inside functions, with function signatures, consts, and statics requiring that you explicitly write out the type signature and structs even requiring you to write out lifetime annotations in places where they could be inferred. There are no exceptions, instead requiring that you declare error paths intended to be recoverable (i.e. not asserts) as part of your return type via Result<T, E>. There's no such thing as "is this class POD?" (You ask for dynamic dispatch by taking a (data, vtable) fat pointer to the object). There are various language features such as method overloading where part of the reason they don't exist is "the interaction with type inference would leak implementation details into the ABI stability characteristics". etc. etc. etc.

The ecosystem has also produced crates like static_assertions (#1 most popular crate in the #compile-time category according to the libs.rs alternative crates.io frontend), which makes it easy to fail the build if invariants like "these two types must have the same alignment requirement" get broken.

Packaging Rust for Fedora

Posted Oct 30, 2022 11:07 UTC (Sun) by ssokolow (guest, #94568) [Link]

"the interaction with type inference would leak implementation details into the ABI stability characteristics"

I keep forgetting not to post in the middle of the night. I meant "API" here.

Packaging Rust for Fedora

Posted Oct 31, 2022 0:41 UTC (Mon) by tialaramex (subscriber, #21167) [Link] (2 responses)

> function signatures, consts, and statics requiring that you explicitly write out the type signature and structs even requiring you to write out lifetime annotations in places where they could be inferred.

Lifetime elision has been enabled in Rust for some time. For example, suppose our function takes a &str and it returns a &str. We could ask, what is the lifetime of these two &strs ? But it doesn't take a genius to guess that they should probably have the same lifetime, whatever that is. The compiler will try that and, if it works, accept the function despite it not specifying the lifetimes.

https://doc.rust-lang.org/reference/lifetime-elision.html

Packaging Rust for Fedora

Posted Oct 31, 2022 1:03 UTC (Mon) by ssokolow (guest, #94568) [Link] (1 responses)

I've been using Rust since 1.0. I know what lifetime elision is. Maybe my phrasing wasn't clear enough that the "even requiring you to write out lifetime annotations in places where they could be inferred" was scoped to the "and structs", not the entire sentence.

I was referring to how, yes, you can elide lifetimes in function signatures, but you'll get a "missing lifetime specifier" error if you try to do something like this:

struct Thing {
    inner: &str,
}

When lifetime elision was implemented, stopping there was a conscious decision.

Packaging Rust for Fedora

Posted Nov 1, 2022 8:28 UTC (Tue) by tialaramex (subscriber, #21167) [Link]

Oh! Now you point it out I think my brain skipped over the "and structs". Sorry.

Packaging Rust for Fedora

Posted Oct 31, 2022 3:36 UTC (Mon) by pabs (subscriber, #43278) [Link] (2 responses)

Is there a recommended way for Rust crates to generate manual pages for command-line tools? The ones I've seen just use help2man to generate a manual page, commit that to git and maybe update it, maybe not.

Packaging Rust for Fedora

Posted Oct 31, 2022 11:28 UTC (Mon) by neal (subscriber, #7439) [Link] (1 responses)

In Sequoia, we started generating man pages from the clap data structures. It works much better than help2man.

Packaging Rust for Fedora

Posted Oct 31, 2022 12:35 UTC (Mon) by pabs (subscriber, #43278) [Link]

Interesting, thanks. As it happens and perhaps as you guessed, I was complaining here about the Sequoia keyring linter, which still does the help2man approach. Would you like a new issue as a reminder? :)

Packaging Rust for Fedora

Posted Oct 30, 2022 13:03 UTC (Sun) by emk (subscriber, #1128) [Link]

> I'm not that familiar with Rust, but I'd say a good language for software engineering needs at least... some effort to keep users from accidentally using implementation details

Rust uses namespaces with the ability to control what gets exported. If it's not exported, you can't access it. You can even say things like "this interface is visible outside of this file, but only inside of this library" (aka "pub(crate)"), or you can export certain symbols only in test mode. Private members of structs are completely hidden, and don't even lie at a predictable offset within the struct. A number of other features, like the orphan rule and trait member visibility, are designed specifically to prevent weird surprises. But Rust doesn't implement this using header files; it's built directly into the language.

Rust libraries are almost precisely limited to what appears in the documentation. And as other posters have noted, Rust's doc tools are solid. By default, if you put sample code in the docs, Rust even compiles and runs it as a unit test. (You can mark individual code snippets "no_run", which will cause Rust to only compile them.)

Packaging Rust for Fedora

Posted Oct 30, 2022 17:01 UTC (Sun) by khim (subscriber, #9252) [Link] (17 responses)

> I don't know why everyone has gone away from separate interfaces, but I don't think it's been a good decision.

It was absolutely good decision because it's bad idea to ask people to do things which computers can do better.

You never need implementation part alone, that's just never usable for anything. You need interface and implementation if you are developing library and you need just the interface if you are using that library.

And it makes much more sense for the computer to take the whole thing and remove implementation (like both Javadoc and rustdoc are doing) then to ask human to mentally combine them when s/he tries to do development.

Packaging Rust for Fedora

Posted Oct 30, 2022 19:30 UTC (Sun) by dvdeug (subscriber, #10998) [Link] (16 responses)

Java has "interfaces", which are different from the type of interfaces we're talking about, but can be used to separate interface from implementation. And many people online ask if every class should have an interface; among other things, that makes the object mockable.

In semver terms, the interface can not be touched in a patch level change, it can be added to but what was there can't be changed, and it can be arbitrarily changed in a major release. Even internally, the interface can't be unilaterally changed without touching other files.

Mixing interface and implementation makes it harder for the programmer to see what changes are breaking the interface. Making the interface into the implementation is usually a copy command and a few minor edits, and the computer could easily keep them compatible. Either way, the computer can help the issue, but I've seen nothing that highlights interface changes or otherwise makes them obvious to the programmer.

Packaging Rust for Fedora

Posted Oct 30, 2022 20:34 UTC (Sun) by khim (subscriber, #9252) [Link] (14 responses)

> Java has "interfaces", which are different from the type of interfaces we're talking about, but can be used to separate interface from implementation.

Not perfectly. Default implementation of methods for these interfaces (like with Rust analogue) goes in the file where interface is described.

And you can add classes to them, too:

interface MyInterface {
    void interface_not_implementation();
    static class MyImplementation implements MyInterface {
       public void interface_not_implementation() {
           System.out.println("Someone talks about separation?");
       }
    }
}

> And many people online ask if every class should have an interface; among other things, that makes the object mockable.

Yes. That what I hate about Java most. Not about Java-the-language, but about Java-the-ecosystem. Useless pile of abstractions where the actual code is very hard to find.

Why would you make every class mockable? What's the point? What are you trying to achieve? There are no implementation inheritance in Rust which means 90% of problems which this total mocking is supposed to fix just don't exist!

Traits are a bit harder to abuse for that, but, more importantly, Rust doesn't consider blind cargo-culting a viable way to write code.

> Mixing interface and implementation makes it harder for the programmer to see what changes are breaking the interface.

Maybe if you randomly change your code. Yes, trait definition goes into the same file as it's implemntation, usually, but in practice it's not hard to notice whether change is modifying trait or only it's implementation (just look for impl keyword).

> Either way, the computer can help the issue, but I've seen nothing that highlights interface changes or otherwise makes them obvious to the programmer.

It's same in Java. If you replace Object with T then it's backward-compatible change as easily can be seen, but nothing highlight that AFAICS. Same as with C or Rust: you have to look on the change to understand whether it's backward-compatible or not.

Packaging Rust for Fedora

Posted Oct 30, 2022 20:47 UTC (Sun) by mpr22 (subscriber, #60784) [Link] (13 responses)

> Why would you make every class mockable? What's the point? What are you trying to achieve?

If every class is mockable, then a unit test of class Fred can use mocks of any classes that class Fred invokes.

This is highly attractive for some people.

Packaging Rust for Fedora

Posted Oct 30, 2022 21:33 UTC (Sun) by khim (subscriber, #9252) [Link] (12 responses)

> This is highly attractive for some people.

That's not the answer why they like to compulsively mock everything endlessly.

If someone likes to get drunk it's not a good reason to make alcohol more easily available to him (or her).

Note that C doesn't allow you to easily mock something and yet Linux kernel is more reliable than creations of these guys who praise Java for the easy mockability of everything.

Which means mockability doesn't translate into higher reliability.

If people do understand why they need to mock everything religiously (and that still doesn't help entirely) then it would be easy for them to see why they don't need that in Rust.

If they don't understand where and why that cargo-cult started then I don't see why we need to even help them.

P.S. Hint: it starts with L and includes the name.

Packaging Rust for Fedora

Posted Oct 30, 2022 22:54 UTC (Sun) by dvdeug (subscriber, #10998) [Link] (11 responses)

> Note that C doesn't allow you to easily mock something and yet Linux kernel is more reliable than creations of these guys who praise Java for the easy mockability of everything.

> Which means mockability doesn't translate into higher reliability.

That's a ridiculous comparison. You're comparing a kernel which has had some of the best programmers in the world work on it to ... everything ever written in Java?

Mocking a class permits testing just that exact class in isolation. That has clear advantages for testability and hence for reliability. Is it the best tool? I don't know, but the interesting question for most users is how does it work for standard business programmers working under time pressure, not how does it work for Linus Torvalds when he's been working on the same program for 30 years.

Packaging Rust for Fedora

Posted Oct 31, 2022 0:53 UTC (Mon) by khim (subscriber, #9252) [Link] (9 responses)

> You're comparing a kernel which has had some of the best programmers in the world work on it to … everything ever written in Java?

No, just typical “enterprise app” whose development cost comparable sum to what development of kernel cost.

And you can not say that kernel is written by “some of the best programmers in the world” and then turn around and claim that they just don't understand the good design when they see it.

> Mocking a class permits testing just that exact class in isolation.

Nope, you don't need mocks for that. If you class permits the disconnect in production (e.g. if you permit choice of either mysql or postgresql) then you can change that part for testing. If there are no possibility of separating two components or classes in production then why the heck do you want to separate them in your tests? Mocks are not the way to test interfaces, they increase coupling instead of decreasing it!

> That has clear advantages for testability and hence for reliability.

Quite the contrary. Simple example: suppose I have a parser and I decided to improve it's performance by adding some buffering to it… it would pass all tests you can imagine without mocks… but mocks would make it fail (because mock wouldn't provide enough data to fill the buffer).

Now, I have to change mocks and would make them match new expectations from what must be provided for my parser — which means I'm writing new code without tests (there are no tests for mocks because if you start going that way when would you stop).

And it's easy to write mock in a way that it always fills the buffer requested and never returns less data then what was requested (unlike the real thing). Eventually you reach the state where your mocks model everything badly and incorrectly. And then your tests pass but code throws exceptions left and right if it can be used at all.

Yes, mocks allow you to test code in isolation, but that's a problem, not an advantage. It reduces reliability of the whole system, not increases it. Some even say that they only way to properly test something is to test “everything like in production”. I wouldn't go as far, but the larger piece of your system is covered by a single test then higher chance it gives to you that test verifies something important.

Mocks go in the opposite direction and thus are not needed nor desired for reliability. Try again.

> I don't know, but the interesting question for most users is how does it work for standard business programmers working under time pressure, not how does it work for Linus Torvalds when he's been working on the same program for 30 years.

Maybe, but from observations it doesn't work all that well for them either. Git (entirely new project) went from rought idea to usable product much faster then any Java-business project I know. And yes, I know that Linus Torvalds is brighter than “most standard business programmers”. That wasn't the big reason.

The big reason was that Linus knew what he wanted and he knew what tests he needs and what tests he doesn't need. “Standard business programmer” just does motions which were designed to achieve things he doesn't know about and then tries to fix issue which not need fixing while simultaneously missing things that do need fixing.

It's telling that you couldn't event explain what property mocks are trying (and failing) to achieve and which, indeed, is hard, almost impossible to test without mocks in Simula67-style OOP languge like Java.

Packaging Rust for Fedora

Posted Oct 31, 2022 2:30 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (1 responses)

> No, just typical “enterprise app” whose development cost comparable sum to what development of kernel cost.

https://lwn.net/ml/linux-kernel/CAHk-=wj6y5fipM2A5kEuOO9q... lists over 100 contributors to Linux 6.1-rc1. Assuming that has lasted over the past 30 years, I don't know how many enterprise apps there are with 3000 man years in them.

> And you can not say that kernel is written by “some of the best programmers in the world” and then turn around and claim that they just don't understand the good design when they see it.

That's not what I said. I will say that the best anything in the world frequently don't understand how the average person in a field can best do something.

> Some even say that they only way to properly test something is to test “everything like in production”.

Which is a cause of a huge number of bugs. For one thing, the hardware your code will run on in a few years usually doesn't exist. For another, it's hard to predict what production will be like. At worst, that comes down to "it works on my system".

> The big reason was that Linus knew what he wanted

Yeah, I don't think a single programmer writing a source-control system to his own whims is very closely related to writing a large team writing code to external specifications.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:49 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> https://lwn.net/ml/linux-kernel/CAHk-=wj6y5fipM2A5kEuOO9q... lists over 100 contributors to Linux 6.1-rc1. Assuming that has lasted over the past 30 years, I don't know how many enterprise apps there are with 3000 man years in them.

Amazon has something like 50000 developers working on AWS. I believe Microsoft and Google have similar numbers. Even given that the average AWS developer is far from the level of kernel developers, the aggregate amount of code that they produce is staggering.

You really underestimate the size of modern large applications.

Packaging Rust for Fedora

Posted Oct 31, 2022 8:18 UTC (Mon) by Wol (subscriber, #4433) [Link] (5 responses)

> Maybe, but from observations it doesn't work all that well for them either. Git (entirely new project) went from rought idea to usable product much faster then any Java-business project I know. And yes, I know that Linus Torvalds is brighter than “most standard business programmers”. That wasn't the big reason.

> The big reason was that Linus knew what he wanted and he knew what tests he needs and what tests he doesn't need. “Standard business programmer” just does motions which were designed to achieve things he doesn't know about and then tries to fix issue which not need fixing while simultaneously missing things that do need fixing.

Which is why I bang on about how Pick projects are better than SQL. Pick makes programming accessible to the END USER. Which is why projects actually do what is required at the first attempt!

Okay, without proper programmer/analyst input right from the start, the result can easily be a horrendous ball of spaghetti, but when you have programmers and end users working TOGETHER, then the result is brilliant. Far too many projects, the end user tells the analyst what they want, the analyst specs the system, then maybe the designer designs it, before the programmer writes it. Chinese Whispers all the way. The programmer never sees an end user, half the people in the chain are clueless as to what the system is really wanted to do, and the result is a late, over budget, half-implemented ass of a solution.

Cheers,
Wol

Packaging Rust for Fedora

Posted Oct 31, 2022 12:04 UTC (Mon) by khim (subscriber, #9252) [Link] (4 responses)

> Which is why I bang on about how Pick projects are better than SQL.

SQL is awful abomination in theory, but there are many decent free implementation which work reasonably well in practice.

Pick is better in theory but there are no decent free implementations which makes it non-starter for many projects.

I would love to use Pick instead of SQLite for my [pretty] modest needs but SQLite is free and integrates very well with other tools which Pick…

Well, maybe someone would solve that problem. Hey, before Rust I was 100% sure the mistake that world made quarter century ago (when it picked OOP and GC as the main road which obviously leads us nowhere) would never be fixed.

Maybe something like that would, eventually, happen with databases, too?

Packaging Rust for Fedora

Posted Nov 1, 2022 12:47 UTC (Tue) by Wol (subscriber, #4433) [Link] (3 responses)

SQL is an awful abhomination in practice ...

I've just been involved in a migration from SQL/Relational to SQL/Relational. It's over-run its cost budget, it's over-run its time budget, it's been a bit of a disaster ... (Not in causing any real problem to the company, but just in the endless delays ...)

Part of this has been "where is this field defined?". We've got assorted measures like DPR (part of our efficiency metric), which is calculated in god-only-knows how many places, the definition has chnged over time, where the hell is the current definition and is it used consistently ...

Because Pick does NOT distinguish between tables and views, because Pick has virtual columns, because Pick is (or can be) object orientated, it's EASY to have just the one definition, in the one place, and THAT'S IT. As soon as you try defining virtual columns in Relational, they end up strewn over multiple tables, and managing it is a nightmare.

Then there's the simplicity of defining your FILE. You define your Real columns just like in Relational. Then because Pick defines by iterating through rows, you define your Virtual columns with a function like Excel vlookup. You don't have this horrendous 100-line (if you're lucky enough for it to be that small) abhomination of a SQL view. So your virtual field is defined as part of the FILE that is the object of interest, and if you need it anywhere else, you just create a virtual field that references the master definition and pulls it across.

Can't remember whether I said, but seeing as I'm going to be integrating ScarletDME with Google Sheets and BigQuery, it comes with an Excel connector, there's a whole bunch of JSON, web, whatever connections available if you want it ... if you've got time to play download it, reach out to me if you want, go onto the relevant google groups, and I'll do everything I can to help ...

Cheers,
Wol

Packaging Rust for Fedora

Posted Nov 1, 2022 15:41 UTC (Tue) by kleptog (subscriber, #1183) [Link] (2 responses)

IMHO if you're doing anything significant in SQL, an ORM with a generative query API is a must. Then you have a single source of truth for your database structures. The environment can validate the queries at compile time. Migrations can be autogenerated. And you can write your queries in a way that makes sense. If this results in 20KB queries being sent to the SQL server, that's Not My Problem.

Packaging Rust for Fedora

Posted Nov 1, 2022 15:58 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

ORM?

(Oh, and 20kb queries aren't the problem. It's queries that are hand-written and huge. And by the way, the guys expected to write the queries are NOT database bods, they're end users who need the information. (And without whom, there's no point having a database!!!))

Oh - and if ORM means "Object Relational Model", then just ditch the relational and use Pick instead :-)

Cheers,
Wol

Packaging Rust for Fedora

Posted Nov 1, 2022 23:24 UTC (Tue) by ssokolow (guest, #94568) [Link]

They're probably lumping anything with a query builder API under "ORM", even though things like Python's SQLAlchemy make the Object-Relational Mapper layer an optional higher-level wrapper around the lower-level query builder system.

Packaging Rust for Fedora

Posted Dec 3, 2022 6:38 UTC (Sat) by ghane (guest, #1805) [Link]

> “Standard business programmer” just does motions which were designed to achieve things he doesn't know about and then tries to fix issue which not need fixing while simultaneously missing things that do need fixing.

This. Precisely this.

Developers keep opening issues that make no business impact, and keep trying to solve every issue with a combination of StackOverflow and copy/paste.

This week, I have been trying to explain to a team why they should remove an "UpdatedAt" database field from a table that is an immutable append-only log. "But every other table has a CreatedAt and UpdatedAt field!"

Meanwhile, users see "Oops an Error happened" popups. "OK" button.

Packaging Rust for Fedora

Posted Oct 31, 2022 13:55 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

> Mocking a class permits testing just that exact class in isolation.

I think you mean "things using that class".

However, I have managed to get away with zero "class mocks" in my Rust code as well. The `#[test]` attribute allowing tests to be written right beside the code under test helps a lot as it also means that Rust's visibility rules allow the tests to reach "behind the curtain" to do things as necessary (for setup or validation). About the only thing I've mocked is remote services which…are easy to mock without mucking with the actual code since the codebase is abstracted over remote implementations already (GitLab or Github), so adding one that just uses in-memory storage and a tempdir for all of its bits is easy enough.

Packaging Rust for Fedora

Posted Oct 31, 2022 4:25 UTC (Mon) by ssmith32 (subscriber, #72404) [Link]

"And many people online ask if every class should have an interface; among other things, that makes the object mockable."

And the correct answer is always "no".

Because:
- It pointlessly doubles the number of files you have to deal with.
- It pointlessly increases how long it takes to navigate from a call site to an implementation when debugging.
- It doesn't "make an object mockable". Because you can mock objects just fine without them. Mockito, et. al.

This is from working at several different companies on largely Java codebases. New Java folks come in and having read some book by Crazy Uncle Bob or some such nonsense, and they learn quickly that it's not a good idea.

Interfaces should only be used when you *need* multiple implementations. And, no, your hand-rolled mock does not count. You should have used Mockito.

Packaging Rust for Fedora

Posted Nov 1, 2022 10:30 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

> This is where a global cache, like it was proposed for Fedora, would be appreciated.

Ooh, that's a good idea. And the cached compiled modules would of course need versions attached, cause at any given point you may have some binaries that are recompiled against the number modules and some that are not yet. At some point, particularly for energy efficiency on lower-powered devices and efficiency where you have a multitude of machines (large DC), you may wish to just provide these cached, compiled modules as a ready-made package. So you need a globally co-ordinated versions numbers (or at least, sufficiently wide in scope for your situation). Then you can provide foo-modules RPM package, that other packages can rely upon.

Yes, this sounds like a radical improvement on libraries. ;)

Packaging Rust for Fedora

Posted Nov 1, 2022 13:14 UTC (Tue) by Wol (subscriber, #4433) [Link]

Is this the Windows XSX thing?

As I understand it, this keeps multiple copies of dlls, so that you DON'T overwrite an old version with a new one and screw up programs that rely on the old version. So what we want is shared, versioned liberaries, that are happy to share if they're using the same version, but don't trample over someone else's *different* version of the same libary.

Bit like a curated crates setup - if they can make that do that, you'll only have one copy of each crate, and hopefully they'll have some mechanism whereby you can update the crates manifest/linkage without having to relink the program, so if it all goes pear-shaped with a new version, you can revert the link-update.

Cheers,
Wol

Packaging Rust for Fedora

Posted Oct 28, 2022 15:44 UTC (Fri) by smoogen (subscriber, #97) [Link] (37 responses)

My main concern for Rust will be how it acts in 4-6 years when it is no longer the 'golden child' language which titillates a lot of minds towards wanting to play with it and thus make lots of cargo items. At some point some new language paradigm will hit the zeitgeist of developer community (take Functional languages 10 years ago, various object oriented languages 25 years ago, etc) and you end up with a large pile of poorly maintained infrastructure being held up by a dwindling number of diehard Rust developers. [The same happened then and lots of used libraries died off.. the problem now is that the size of the problems and amount of infrastructure relying on what may be very dead code will be a lot greater.]

Packaging Rust for Fedora

Posted Oct 28, 2022 17:59 UTC (Fri) by khim (subscriber, #9252) [Link]

> At some point some new language paradigm will hit the zeitgeist of developer community (take Functional languages 10 years ago, various object oriented languages 25 years ago, etc) and you end up with a large pile of poorly maintained infrastructure being held up by a dwindling number of diehard Rust developers.

Have that happened to any language which was kinda-sorta-remotely popular at any point?

Haskell, PHP, Perl, Ruby or, heck, even Object Pascal or TeX… they all have pretty robust infrastructure, enough for their needs.

They may not see lots of development and advances, but it doesn't take a lot to support infrastructure for the given language.

They mostly just use free offerings from AWS, GitHub and other such companies… it doesn't take that much to support them.

> The same happened then and lots of used libraries died off.. the problem now is that the size of the problems and amount of infrastructure relying on what may be very dead code will be a lot greater.

You say that happened already… which language and how can we even observe that trouble?

Packaging Rust for Fedora

Posted Oct 28, 2022 19:53 UTC (Fri) by walters (subscriber, #7396) [Link] (20 responses)

The very large organizations that are investing in Rust are doing it for things they expect to last much longer than 4-6 years. I don't think your concern has merit.

It doesn't make sense to lump Rust in with higher level (non-systems) functional languages - "application languages" (usually GC'd) indeed have some churn. Rust is going into the core of operating systems (all of Windows, Linux (kernel and userspace) and Android, etc. precisely because it has the right tradeoffs to last equally as long as C/C++ have done)

Packaging Rust for Fedora

Posted Oct 28, 2022 20:39 UTC (Fri) by smoogen (subscriber, #97) [Link] (19 responses)

I saw the same corporate investment in Java in the mid-1990s to mid 2000's... and then dealt with a lot of 'where did that class come from? which of the identical versions was it? Oh wait this worked for Java-1.5 and we need 1.9?' once the zeitgest seemed to have moved on. [I also remember a strong push in the 1990's when Java was going to replace all other languages and we had to rewrite everything into Java. Heck I worked with a project that even made a CPU which was Java based so it's OS was written in Java..] In the webspace a similar issue with PHP and NodeJS exploded where lots of people jumped in, had a lot of fun, made some needed parts and disappeared... and then the parts were left sitting there to rot.

At some point, in each of these someone says 'could we work out how to maintain and standardize this for the long term' and pretty much everytime it gets replied with some sort of 'that isn't needed.. you are going to kill all the fun if you do that.' or 'surely all these large organizations who are investing won't let that happen...' And then at some point either it goes through a horrible crash and then a standardization OR it dies off in other ways.

Look I like RUST. I like what it offers and solves a lot of issues. I am tired of how people keep repeating this ecosystem building and crashing.

Packaging Rust for Fedora

Posted Oct 28, 2022 21:13 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

> I am tired of how people keep repeating this ecosystem building and crashing.

What's the alternative? Doing without any ecosystem at all like in Ada world?

Hardly a better option.

Packaging Rust for Fedora

Posted Oct 29, 2022 12:52 UTC (Sat) by smoogen (subscriber, #97) [Link]

No the answer is to build an ecosystem which realizes that realizes humans are crap at dealing with complexity and is able to regularly collect, cull, and reseed when things get out of control in various areas. Many of the technical lessons from previous problems are already built into cargo, but some require active maintainership by a community like any distribution.

Packaging Rust for Fedora

Posted Oct 28, 2022 22:05 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (15 responses)

To be fair Java and C# did replace a lot of COBOL, as well as kill Visual Basic and the various dBase derivatives (anyone remembers Visual FoxPro?), and more. Any website where there is some "business logic" (package tracking, taxes, train and airline ticketing, etc.) is likely to have Java or C# running in the backend.

Packaging Rust for Fedora

Posted Oct 29, 2022 13:33 UTC (Sat) by khim (subscriber, #9252) [Link] (14 responses)

> anyone remembers Visual FoxPro?

I'm pretty sure guys which explain how to use Visual FoxPro 9 and even Visual FoxPro 2.1.36 on Windows 11 remember it very well indeed. Plus, according to TIOBE (flaved as it is) it's more popular than Haskell and almost as popular as Ada!

That shows why Linux distributions wouldn't replace Windows any time soon: it's hard to image binary package whose support ended seven years ago to be that popular in Linux world… but in a world outside of IT 5 or 10 years is not a large time at all.

And that, it turn, explains why Rust developers feel so disinclined to help with supporting Linux distros. For the majority of them these are funny guys who produce some intermediate software which helps them to server real users (Android/iOS/macOS/Windows ones) yet try to demand way too much when they are not really all that important.

Packaging Rust for Fedora

Posted Oct 30, 2022 20:09 UTC (Sun) by ceplm (subscriber, #41334) [Link] (13 responses)

> And that, it turn, explains why Rust developers feel so disinclined to help with supporting Linux distros. For the majority of them these are funny guys who produce some intermediate software which helps them to server real users (Android/iOS/macOS/Windows ones) yet try to demand way too much when they are not really all that important.

If they really believe it, then they are really really silly (and BTW, I don’t believe that anybody serious amongst Rust developers believe it). What you are saying about the dominance of Android/iOS/macOS/Windows might be true with regards to the desktop/end-user operating systems, but it is absolutely not true with regards to the server installations (which is where databases are most relevant). There I think the situation is quite reverse. The first three obviously nobody even tried to use for server installations and I believe that even Microsoft gave up trying to push Windows as the server operating system for serious business/web applications (what’s the share of Linux on Azure? Is it over 50 % already?). And yes, they may ignore long-term maintenance issues with statically linked applications, which are long term very hard to maintain (especially because of lack of maintenance of APIs), but in the end, unless something drastic happens the situation is in few years will be exactly as the previous poster described: graveyard of broken, unmaintainable, buggy applications. And it doesn’t matter if it is Rust, GoLang, NodeJS, or any other language which ignores the lessons C programmers acquired after years of very painful experiences.

Packaging Rust for Fedora

Posted Oct 30, 2022 21:21 UTC (Sun) by khim (subscriber, #9252) [Link] (4 responses)

> In the end, unless something drastic happens the situation is in few years will be exactly as the previous poster described: graveyard of broken, unmaintainable, buggy applications.

The exact same thing that happens in distro-land, too. One simple example: cvf was small, simple utility which can verify checksums (md5, sha1 and other types of files were supported, including, importantly, .torrent files). It was in Debian and EPEL. Now… it's gone.

I can still download and use it on Windows (and I know how and why) yet I can not use it in that “perfect world” of Linux distributions (and yes, I know why, too).

Linux distros can not do anything if upstream is not supporting their creation and if upstream does support something… why would I need someone to make both their and mine life miserable?

> And it doesn’t matter if it is Rust, GoLang, NodeJS, or any other language which ignores the lessons C programmers acquired after years of very painful experiences.

If you want to say “freeze your language and don't ever change anything” then yeah, I may agree, it's one way of achieving stability. Another, better one, would be to provide stable base for development.

> What you are saying about the dominance of Android/iOS/macOS/Windows might be true with regards to the desktop/end-user operating systems, but it is absolutely not true with regards to the server installations (which is where databases are most relevant).

Servers are almost entirely useless without clients. Just a tiny part of the whole. And distributions are a tiny part of that small part. People are using Linux distros because it's just cheaper to run docker images on Linux distros than on Windows. Even on Azure. It's not because distros are better in any other way.

Packaging Rust for Fedora

Posted Oct 31, 2022 0:05 UTC (Mon) by ceplm (subscriber, #41334) [Link]

I won’t respond to the rest of the reply (because it is mostly “And you are lynching Negroes” type of one), but this bit deserves a comment:

> Servers are almost entirely useless without clients.

https://www.joelonsoftware.com/2004/06/13/how-microsoft-l...

When was the last time you were using desktop client for anything? It is all (or 97.6 %) HTML and Internet browsers these days, so the desktop OS is as a result mostly irrelevant.

Packaging Rust for Fedora

Posted Oct 31, 2022 9:13 UTC (Mon) by ballombe (subscriber, #9523) [Link] (2 responses)

> The exact same thing that happens in distro-land, too. One simple example: cvf was small, simple utility which can verify checksums (md5, sha1 and other types of files were supported, including, importantly, .torrent files). It was in Debian and EPEL. Now… it's gone.

Because it was written in python 2, and python developers decided to phase out python 2.

Packaging Rust for Fedora

Posted Oct 31, 2022 13:35 UTC (Mon) by khim (subscriber, #9252) [Link]

> Because it was written in python 2, and python developers decided to phase out python 2.

Sure. I know that distros can not do anything if upstream doesn't like to support what they wrote and rewrite perfectly working tool for no good reason when something changes in the foundations.

But if they can not do that… then what value do they give to the users? I can not rely on software which distros give me unless said software have active and cooperative maintainer… but if there are active and cooperative maintainer then why would I need a middleman between said maintainer and me?

Packaging Rust for Fedora

Posted Nov 1, 2022 13:20 UTC (Tue) by ceplm (subscriber, #41334) [Link]

> Because it was written in python 2, and python developers decided to phase out python 2.

Sure, so it has absolutely nothing to do with the argument which was taken here against Linux distributions and packaging.

Packaging Rust for Fedora

Posted Oct 31, 2022 1:11 UTC (Mon) by ssokolow (guest, #94568) [Link] (7 responses)

I find arguing in the direction of "but servers" kind of odd, given that languages like Go and Rust which statically link everything are effectively a codification of the "I can't wait for your perfect solution anymore. I'll fix it." reaction that is containerizing everything using Docker.

It's "We can't wait for Hurd. Let's put GNU userland on top of this thing called Linux instead" all over again.

Packaging Rust for Fedora

Posted Oct 31, 2022 6:12 UTC (Mon) by ceplm (subscriber, #41334) [Link] (5 responses)

And enterprise distributors of Linux (and I do NOT speak for my employer and this is not the information about our customers, just what I’ve heard) now have number of large clients who are quite nervous that they have tons of Docker (et al.) containers full of unmaintained stuff they have not complete control over. And yes, of course, whatever I said about Rust applies to Go/NodeJS and other unmaintained stuff in containers as well.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:43 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> And enterprise distributors of Linux (and I do NOT speak for my employer and this is not the information about our customers, just what I’ve heard) now have number of large clients who are quite nervous that they have tons of Docker (et al.) containers full of unmaintained stuff they have not complete control over.

And before it was a ton of unmaintained crap smeared over RHEL servers with who-knows-what custom configs hidden inside /var/share/somewhere/etc/why/thefuck/its/here, all placed there by a "user-friendly" installer script. I wish I were exaggerating.

At least Docker images are a step up because the Docker files list the commands needed to build the image.

There's no solution to unmaintained old software except making maintenance easier.

Packaging Rust for Fedora

Posted Oct 31, 2022 18:15 UTC (Mon) by Conan_Kudo (subscriber, #103240) [Link] (3 responses)

You assume that the Dockerfiles exist. I have many-an-unmaintained image with no Dockerfile. There's no rebuilding or fixing this. You have to do the same kind of reverse engineering you did for unmaintained VMs or physical servers.

Packaging Rust for Fedora

Posted Oct 31, 2022 18:35 UTC (Mon) by khim (subscriber, #9252) [Link] (1 responses)

How is that different from what any other industry faces?

Packaging Rust for Fedora

Posted Oct 31, 2022 18:41 UTC (Mon) by Conan_Kudo (subscriber, #103240) [Link]

It isn't. I'm saying that container appliances are just as screwed as VM appliances.

Packaging Rust for Fedora

Posted Oct 31, 2022 19:53 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Even having a layered image is better than having one huge VM. Having a Dockerfile also does make it easier for vendors to rebuild the software. Also, Docker makes it easier to sandbox such applications and manage them centrally.

It's not perfect, of course, but it's strictly better than the status quo ante. Even if not by a lot.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:46 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

go is made by google, which doesn't really care about good practices and has no problem to rebuild and maintain all the dependencies. Others might be using at their own risk.

Packaging Rust for Fedora

Posted Oct 29, 2022 17:23 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

> I also remember a strong push in the 1990's when Java was going to replace all other languages and we had to rewrite everything into Java.

This has arguably happened, with Android.

Packaging Rust for Fedora

Posted Oct 29, 2022 2:49 UTC (Sat) by pabs (subscriber, #43278) [Link] (1 responses)

Whats the next big thing in languages? My bet is on dependent typing like in Idris (or a future Haskell version).

Packaging Rust for Fedora

Posted Oct 29, 2022 13:37 UTC (Sat) by khim (subscriber, #9252) [Link]

Most likely, but we are, probably, decades away from that. We have to become accustomed to Rust first.

Rust is really exciting not only because of “safety without GC”, but because it's ownership and borrow system made many things which previously were only possible in functional languages (which, for various reasons would never become all that popular) to the realm of imperative programming.

“Safety without GC” is huge bonus, of course, but I think what is happening here is more exciting than that.

Packaging Rust for Fedora

Posted Oct 29, 2022 5:29 UTC (Sat) by IanKelling (subscriber, #89418) [Link]

This is a good point. It certainly will happen.

Packaging Rust for Fedora

Posted Oct 29, 2022 6:45 UTC (Sat) by zdzichu (subscriber, #17118) [Link] (7 responses)

Rust is 12 years old. Version 1.0 was released in 2015. It's not a new language anymore.

Packaging Rust for Fedora

Posted Oct 29, 2022 10:24 UTC (Sat) by sbakker (subscriber, #58443) [Link] (3 responses)

> Rust is 12 years old. Version 1.0 was released in 2015.

Er, welcome to 2027? (You might want to check your clocks.)

Packaging Rust for Fedora

Posted Oct 29, 2022 10:41 UTC (Sat) by excors (subscriber, #95769) [Link] (2 responses)

Rust existed before version 1.0. The public announcement by Mozilla was 12 years ago (2010), after several years of private work by Graydon Hoare. Version 1.0 in 2015 was the first stable release, with a guarantee that code written for 1.0 will remain compatible with any future release, so it was considered reasonably mature by that point.

Packaging Rust for Fedora

Posted Oct 30, 2022 12:49 UTC (Sun) by emk (subscriber, #1128) [Link] (1 responses)

Ironically, I can still compile some of my Rust 1.0 code from 2015 using modern compilers, just by running "cargo build". When I can't compile it, 90% of the trouble is typically caused by a C dependency, OpenSSL. OpenSSL breaks source and binary compatibility relatively often, and it's had plenty of urgent security updates. So when I rebuild old code, I often need to update the networking libraries to something modern.

It's actually difficult to maintain 100% source compatibility over most of a decade. If you download sufficiently old C and C++ sources, they often require a bunch of tweaking to build.

But overall, Rust has done a fantastic job of maintaining source compatibility over time. Thanks to the "editions" system, it's possible to mix libraries using Rust 2015 with libraries using Rust 2021.

My general strategy for deploying Rust binaries on Linux is to use musl-libc, and depend on nothing except the kernel APIs and the system cert store. This works especially well in mixed environments, where I need to support both Ubuntu and Alpine-based Linux containers, which don't have glibc.

Packaging Rust for Fedora

Posted Nov 1, 2022 9:19 UTC (Tue) by roc (subscriber, #30627) [Link]

Years ago we switched Pernosco to use rustls for everything. We have "cargo deny" rules to make sure we don't accidentally depend on OpenSSL ever again. Never regretted that move.

Packaging Rust for Fedora

Posted Oct 30, 2022 10:28 UTC (Sun) by ssokolow (guest, #94568) [Link] (2 responses)

It's a bit misleading to point to pre-1.0 versions of Rust, given some of the massive changes that took place in the lead-up to promising stability.

For example, in 2013, they switched from compiler-privileged sigils to standard library types for things like Arc<T> and, in 2014, they removed the green threading runtime that had been helping to encourage people to draw comparisons with Go.

Rust as the "great interop with C and a higher-level systems programming language" language we know today didn't truly exist until late 2014 or early 2015.

Packaging Rust for Fedora

Posted Oct 30, 2022 17:41 UTC (Sun) by nevyn (guest, #33129) [Link] (1 responses)

If you dare to evaluate the language on the documentation and tutorials/examples ... it's even newer than that. I remember looking at rust seriously just after 1.0, and the documentation was challenging ... quickly looking much more recently the situation had improved a lot (maybe even completely fixed).

Packaging Rust for Fedora

Posted Oct 30, 2022 23:29 UTC (Sun) by ssokolow (guest, #94568) [Link]

I've been using Rust since 1.0 (I lurked in /r/rust until the stability promise came into play, then started writing code once I could trust it would remain compilable), so my perspective on that could be a bit skewed.

Yes, Rust 1.0 was quite spartan... but you can still compile what you wrote for it in modern Rust assuming you don't fall into one of the exceptions like "this code only compiled because of a compiler bug".

Packaging Rust for Fedora

Posted Oct 29, 2022 10:07 UTC (Sat) by moltonel (subscriber, #45207) [Link] (3 responses)

AFAICT Rust is already past this "golden child with an uncertain future" stage. It's being adopted by projects and industries that value long-term commitment. It didn't have the meteoric climb of Go or Typescript, it's adoption curve is much more linear and steady [1]. It's not as domain-specific as those two, it appeals to a wider range of programmer backgrounds and project types than arguably even C/C++. A decade from now, the "poorly maintained infrastructure and dwindling number of diehard developers" is more likely to describe C than Rust.

[1]: https://tjpalmer.github.io/languish/#y=mean&weights=i...

Packaging Rust for Fedora

Posted Oct 29, 2022 12:47 UTC (Sat) by smoogen (subscriber, #97) [Link] (2 responses)

I believe that 'A decade from now, the "poorly maintained infrastructure and dwindling number of diehard developers" ' has been true for C for at least 25 years. Most C libraries have been written and rewritten by taking up various 'oh we use this many times already but in 4 different versions which didn't work the way I need it to now. This generic version should cover all the use cases.. oh wait'. Eventually the smaller number of people try getting a standards committee or some formal specification written to try and tie down the variants.

Look I am not saying Rust isn't going to take off. I am just asking the people building this ecosystem to really look around at the previous ecosystems which ran into scaling issues.

Packaging Rust for Fedora

Posted Oct 29, 2022 13:44 UTC (Sat) by khim (subscriber, #9252) [Link] (1 responses)

> I am just asking the people building this ecosystem to really look around at the previous ecosystems which ran into scaling issues.

They did. The conclusion was: there are no scaling issues. There are popularity issues.

Human population on this planet is small enough and computers are fast enough that “one central repo” works just fine if you have enough developers to support what you have built.

And if you don't have enough developers because people have left you… nothing works, anyway.

Packaging Rust for Fedora

Posted Oct 29, 2022 23:22 UTC (Sat) by rvolgers (guest, #63218) [Link]

Right. It doesn't help that the C and C++ world is pretty balkanized, largely due to historic boundaries to collaboration.

I'm not saying there isn't duplication of effort in Rust-land, but it is generally easier to share code. Most crates use a common and very permissive license, use the cargo build system, use the same coding style, are cross platform, rely on the stdlib and the language for expressiveness instead of having to first NIH a ton of data structures and macros etc.

Packaging Rust for Fedora

Posted Oct 28, 2022 15:56 UTC (Fri) by andyp (subscriber, #48701) [Link] (30 responses)

I feel like the dependencies headaches with languages like Node.js and Rust are largely caused by the modules/crates being small and numerous, and their authors not forming communities. For example, it seems odd to me that a whole new crate dependency might be required just to add a single hash function for better hash map performance. That feels more like a code dump than a library. One solution would be to collect popular, small Rust crates into larger, fewer collections of related modules, either curated by the authors themselves or a community of separate curators. To stay on-brand, they could be called freighters or barges :)

Packaging Rust for Fedora

Posted Oct 28, 2022 16:50 UTC (Fri) by mb (subscriber, #50428) [Link] (1 responses)

> One solution would be to collect popular, small Rust crates into larger, fewer collections of
> related modules, either curated by the authors themselves or a community of separate
> curators.

That just results in something similar to https://xkcd.com/927/

Having many smaller crates is an advantage in general.
The code bases are smaller for each crate.
They are better to review and audit.
Testing is easier.
And last but not least, they are easier to replace, if needed.

Having more and smaller crates results in *less* unnecessary and unused code in your application.

Yes, the rebuild-everything-all-the-time approach is not really nice.
But I don't see why we couldn't have a crate compile cache that caches crate compile results for subsequent applications (if they use the same crate version and features).
Does such a thing exist?

Packaging Rust for Fedora

Posted Oct 29, 2022 8:52 UTC (Sat) by moltonel (subscriber, #45207) [Link]

There's sccache, very common in CIs and dev setups.

Packaging Rust for Fedora

Posted Oct 28, 2022 16:59 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I think there's more than just a problem with the individual modules/crates being small, but also the culture of lots of small updates. The result is a set of dependencies that are constantly changing. Individual projects can reduce the impact on their code by locking to a specific version of the dependency, but that doesn't help a distribution trying to package many different projects, each of which might have locked to a different version.

Packaging Rust for Fedora

Posted Oct 30, 2022 14:30 UTC (Sun) by emk (subscriber, #1128) [Link]

Honestly, the Rust community is very good about semver. (Not perfect, but very good.) I have a complex project with probably 200 transitive dependencies, and I hadn't touched it since last winter. When I went to update the dependencies, all but one of the supposedly semver-compatible dependencies upgraded without a hitch. (The exception changed how it formatted dates, breaking a specific third party service.) There were also about 6 dependencies with new major versions, corresponding to supposedly breaking changes. Maybe one of these actually required a few lines of changes to compile.

One nice thing about the Rust ecosystem are tools like "cargo deny", which enforces license policy and checks for CVEs and unmaintained dependencies. I can add rules like "no copyleft licenses for this project" or "no indirect dependencies on OpenSSL", and be notified if these rules are broken. This helps maintain a larger list of dependencies with a bit less stress.

Anyway, as an open source software author, I gave up on worrying about distro packaging years ago. For Linux, I just ship source code plus 100% static binaries built using musl-libc. There are too many distros, each with their own politics, maintainer process and packaging tools. I used to work hard to get my stuff packaged, 15 or 20 years ago, but doing so contributed heavily to maintainer burnout.

Packaging Rust for Fedora

Posted Oct 28, 2022 18:09 UTC (Fri) by khim (subscriber, #9252) [Link] (25 responses)

> I feel like the dependencies headaches with languages like Node.js and Rust are largely caused by the modules/crates being small and numerous, and their authors not forming communities.

Not even close. They do form communities and they do support each other. They just don't view reduction in the number of crates as worthwhile goal.

They have toons which work just fine with hundreds and even thousands of crates, why would they want to reduce that number?

They do understand the issue of having to deal with code with unclear origins and try to solve that issue, but if you are doing GUI and use millions of lines of code… how is it better to have tens of millions lines of code in one place (as when you are using Qt) or split over hundreds (or even thousands) of crates?

> One solution would be to collect popular, small Rust crates into larger, fewer collections of related modules, either curated by the authors themselves or a community of separate curators.

What advantage would that have over list of “kosher”, “tested” crates?

Packaging Rust for Fedora

Posted Oct 28, 2022 18:52 UTC (Fri) by darmengod (subscriber, #130659) [Link] (24 responses)

>> One solution would be to collect popular, small Rust crates into larger, fewer collections of related modules, either curated by the authors themselves or a community of separate curators.
>What advantage would that have over list of “kosher”, “tested” crates?

The keyword here is 'curated'. There are many common idioms for which there are eight competing crates all reinventing the wheel, with no clear "winner" among them.

For some use-cases you're lucky (for argument parsing 'clap' is an outstanding solution and covers basically all reasonable scenarios) while for others you waste more time evaluating every competing solution (most of which turn out to be well below production quality) than actually using them.

Packaging Rust for Fedora

Posted Oct 28, 2022 18:58 UTC (Fri) by khim (subscriber, #9252) [Link] (19 responses)

> for others you waste more time evaluating every competing solution (most of which turn out to be well below production quality) than actually using them.

So now you have to jump over hoops to picks some subpar solution instead of the one showed to your throat against you will? Thanks, but no, thanks.

You were supposed to show advantages, not problems.

The fact that there are no good crate to pick for some need is not a reason to pick one of the immature ones and try to push it instead of others.

And if clear winner exist you don't need any curation to pick that obvious winner.

Packaging Rust for Fedora

Posted Oct 28, 2022 23:58 UTC (Fri) by roc (subscriber, #30627) [Link] (18 responses)

You do need some kind of curation to let people who aren't already part of the community know what the winner is.

Packaging Rust for Fedora

Posted Oct 29, 2022 14:04 UTC (Sat) by khim (subscriber, #9252) [Link] (17 responses)

Sure, but look on the question which started the whole thing:

> What advantage would that have over list of “kosher”, “tested” crates?

Heck, it's obvious that curated approach (Linux distros) have quite thoroughly lost to uncurated one (AppStores for Android, iOS or even Windows): all the “good things” are not in curated Linux distros, they are elsewhere.

Sure, you need some way to find best things in the uncurated pile of things… but you have many ways to accomplish that. Reviews, “editor's choice”, heck, even tutorials like Practical Rust help you there.

Heck, the first community which went the cargo way, CTAN predates Linux distros! CPAN quickly followed and Perl community never felt the need to do what Linux distro guys demand, either.

If anything it's Linux distros who try to bend the whole world to their will, not the other way around. They have to accept that the problem they are trying to exist is very strange, unusual and yet narrow one: it just so happened that C and C++ have become standartized enough that it became a good idea to support many OSes and compilers (with autotools initially and with other tools later) and yet it never become unified enough to get analogue of CTAN/CPAN/CRAN/etc.

Most other communities are in one of two camps: either they are splintered enough for the distribution not to make any sense (like Forth or Scheme) or they coalesced around one, single, repo which makes “distro help” useless and pointless.

Packaging Rust for Fedora

Posted Oct 29, 2022 15:56 UTC (Sat) by amacater (subscriber, #790) [Link] (12 responses)

CTAN and TeXlive work very closely indeed with distros. CPAN doesn't predate Linux distros - Debian has only been here for 29 years.

All the good things are not necessarily in IOS/Windows - and the world is running almost entirely on Linux servers, Linux-based cloud ...

There are a few projects which seem to have tied themselves to one particular Linux distro (or worse, a particular version) - the one
that strikes me is gnuradio and that causes them problems.

Rust really does seem to want to do things it's own cargo way in spite of Linux distros and doesn't seem very curated at all by its own community. Fedora and Debian are trying to do the right thing to provide a basis for everything to work.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:26 UTC (Sat) by khim (subscriber, #9252) [Link] (11 responses)

> All the good things are not necessarily in IOS/Windows

Yes, some things are not available there. But for every example of avesome Linux-only app there are dozen of Windows-only or iOS-only apps.

> - and the world is running almost entirely on Linux servers, Linux-based cloud ...

Which run docker images and Kubernetes pods.

Google may solve the distributions problem by just offering specialized version of Android suitable for use on servers with full support for these.

It's not, really interested, because when you containerise everything distros are not a big problem — but they are not a big help, either.

> There are a few projects which seem to have tied themselves to one particular Linux distro (or worse, a particular version) - the one that strikes me is gnuradio and that causes them problems.

Very similar problems to what the projects which don't support Windows and macOS (in that order) experience.

> Rust really does seem to want to do things it's own cargo way in spite of Linux distros and doesn't seem very curated at all by its own community.

It's not it's own cargo way. It's how everyone except C/C++ are doing things. Python and Ruby, Go and Java, Javascript and, yes, Rust… they are doing the same thing.

Rust is not special. C and C++ are special. It's the only language (well, two languages, but very intervined) which is very popular, have tons of libraries, yet without it's own package manager (there are few, but most developers don't use them… yet) and without sane way of managing dependencies.

> Fedora and Debian are trying to do the right thing to provide a basis for everything to work.

Except neither of them even gives me an SDK suitable for use with, at least, few different versions of these distros. Well… Fedora gives me Flatpack SDK. Which is better than nothings, I guess.

RHEL does, true, but it's entirely separate effort from Fedora.

Fedora and Debian may pretend they are providing basis for everyone, but that's only true for C and C++, to some degree. Everyone else does not need or want that basis.

Sure, distro developers may do whatever they want with Rust, Python or Java. It's open source software after all. But they shouldn't expect special treatment. They are not special. Well… maybe they are but that's only because they cause more pain that other choices of foundation.

Packaging Rust for Fedora

Posted Oct 29, 2022 23:19 UTC (Sat) by ilammy (subscriber, #145312) [Link]

Linux distro is the de facto dependency manager for C, because it’s the only one that’s known to work on your system. In a way, that’s your only “sane” dependency management—for C, where basically everything is considered “insane” by the distros.

It causes an unfortunate side effect where application developers think that disto is their missing package manager for C libraries, for the lack of a better one. Then developers—and distro maintainers—start having thoughts that The Distro should be the package manager for every other language as well, because it is The Right Thing.

Well it’s not. The distro is for the system. Not for the applications developed on that system.

Sure, some applications are developed for the distro, but unless you’re a distro maintainer, you should not ever think that the distro is somehow obliged to provide you with packages for all dependencies under the sun that you might need.

Packaging Rust for Fedora

Posted Oct 30, 2022 1:09 UTC (Sun) by jkingweb (subscriber, #113039) [Link] (9 responses)

> But for every example of avesome Linux-only app there are dozen of Windows-only or iOS-only apps.

Can you name me some awesome Windows-only applications from the last twenty years? Other than games, I had precisely zero by the time I left Windows (one of the reasons I jumped ship).

I'm genuinely curious to hear of some.

Packaging Rust for Fedora

Posted Oct 30, 2022 10:43 UTC (Sun) by ssokolow (guest, #94568) [Link]

I keep a copy of Exact Audio Copy under Wine because, every time I need to back up some music CDs, I find that whipper, the closest Linux runner-up, still has an "if we can't do this perfectly, barf rather than supporting an option to accept a best effort" approach to aging music CDs and it's just flat-out incompatible with track 1 being a data track.

Project64 has broader compatibility with the ROMs I dumped from my childhood N64 carts (thank you, Retrode) than Mupen64Plus or the Mupen64Plus cores for Retroatch. (Notably, Donkey Kong 64.)

There's never been a Linux tool capable of creating Stuffit archives for test fodder (and Smith Micro end-of-lifed it), so I had to eBay some copies of Stuffit and run them in Wine to produce legally redistributable test archives.

The open-source asset extractors for things like Unity lag way behind the Windows freeware tools in supporting modern versions of Unity whenever I find myself needing that to format-shift soundtracks I've already paid for once.

...and that's not including hardware-specific stuff where the protocols have never been reverse-engineered like the Musicsoft Downloader for loading MIDI files onto Yamaha keyboards via SysEx messages.

Packaging Rust for Fedora

Posted Oct 30, 2022 11:58 UTC (Sun) by khim (subscriber, #9252) [Link] (7 responses)

It's funny that you have quoted the sentence without reading it. Not that I haven't said that for every awesome Linux-only app there are dozen of awesome Windows-only or iOS-only apps.

Most of the apps are not, really, all that awesome. They are just available for Windows and/or Android/iOS only… and you need them.

Want to backup documents your children created on their calculator in college? Here's your app, pick whether you need EXE, MSI or DMG. Need to deal with bank? Here's your Windows-only app. Need to check in on your plane? Oh, sorry, web site is broken for last month, but here's you Android/iOS app. Taxes? You know what to do, right?

It's not that there are too many amazing programs for Windows or iOS. It's that lots of apps which you need to be member of the society are not available under Linux.

Sure, you can deal with it. You can ask your friends to help install Linux on your TI Nspire CX II CAS… but, frankly, that phrase even sounds silly.

If you are hermit and actively try to avoid everything not available on Linux it may work… till you would try to find a job and would get instructions in mail which would include link to Windows-only special “safe” browser… Windows-only, of course.

This all comes down to the fact that for decades it was possible to develop and deliver app for Windows and macOS, but before flatpack and snap it wasn't possible to do for Linux.

Today… it's possible, sure, but… development process for these apps is entirely separated from development of apps and libraries for the linux distros, why would you need to bother about that one at all?

It would be as silly as if Android guys would have demanded that Rust developers would have abandoned rustup and cargo and switched to blueprints. They, as developers of platform, do their own thing, Rust developers offer tools for app developers)… and nobody tries to swallow the whole world.

Why can not Linux distros do that?

Packaging Rust for Fedora

Posted Oct 30, 2022 17:35 UTC (Sun) by pizza (subscriber, #46) [Link] (5 responses)

> Want to backup documents your children created on their calculator in college? Here's your app, pick whether you need EXE, MSI or DMG. Need to deal with bank? Here's your Windows-only app. Need to check in on your plane? Oh, sorry, web site is broken for last month, but here's you Android/iOS app. Taxes? You know what to do, right?

> It's not that there are too many amazing programs for Windows or iOS. It's that lots of apps which you need to be member of the society are not available under Linux.

This is ... quite depressing.

It also really drives home the points that the FSF has been trying to make about the problems that arise from depending upon proprietary software and, more recently, proprietary services.

I consider myself fortunate that I've still been able to opt out of all of the dreck. I don't know how much longer that will be the case, but I try hard to put my money where my mouth is, and do what I can to ensure a couple of niche corners still can be met with software that respects its users' freedom.

Packaging Rust for Fedora

Posted Oct 30, 2022 21:46 UTC (Sun) by khim (subscriber, #9252) [Link]

> It also really drives home the points that the FSF has been trying to make about the problems that arise from depending upon proprietary software and, more recently, proprietary services.

But many services were always proproetary! Your bank always belonged to someone and water comes into your house by pipes owned by someone.

That's the thing: world have changed and very often software is used as part of something bigger.

Think about that damn TI Nspire CX II CAS: I only need to have program which can talk to it while I use it. Maybe 4 or 5 years while it's needed for college. After that I have no need neither for it nor for the program which can talk to it.

And I'm pretty sure in 2 or 3 years community would crack it's protocol (like it did with previous models) and would support it. When I would no longer need it.

And the same with everything. Most programs which have no analogues under Linux don't have analogues because they are only usable for a limited time. They are part of some offer which was always proprietary — only in XX century that some offer included the need to go somewhere and today you can use program to do the same thing from home.

Why would they support more than bare minimum? And bare minimum naturally leads to proprietary programs and then only for popular platforms where ongoing support is not too onerous.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:15 UTC (Mon) by linuxrocks123 (subscriber, #34648) [Link] (3 responses)

> This is ... quite depressing.

It's also not true. The specific calculator model he's complaining about has a Linux file manager that works with it. Every bank I've ever encountered has a website that works in a Linux web browser. No bank I've ever encountered has ever tried to push a Windows-only app on me. Airline web sites don't go down for months at a time, because that would be very expensive for the airline as it would lead to fewer ticket sales. From every airline website I've ever used, you can print your boarding pass from the website and take the printed page with you to board, meaning all you need to buy the ticket and get on the plane is a web browser, and it can be a Linux web browser if you want. Regarding taxes, the IRS has an official online filing portal called "Free Fillable Forms" which I can attest works just fine from a Linux browser.

Now, that website is actually run not by the IRS but by some proprietary shitware company, I think Intuit, for stupid political reasons, and it's not the most user friendly tax filing option out there. I do believe there is an unfilled need for open source tax filing software, and I am actually working on filling that need. While I don't have anything remotely usable yet, but this team does: https://ustaxes.org And you don't need either of our products because Free Fillable Forms does work just fine, even if it is clunky.

The Linux desktop world is not perfect -- nothing ever is -- but, contrary to what khim claims, it's in very good condition.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:54 UTC (Mon) by pizza (subscriber, #46) [Link]

> It's also not true.

We can quibble over the specifics of what he said, but it's clearly been trending in that direction for some time, with mobile-optional becoming mobile-first and trending towards mobile-only, and locked-down-mobile-only at that. Even before smartphones became common, there were examples of banking sites in some countries (South Korea?) and plenty of governmental sites elsewhere requiring specific ActiveX plugins, which effectively locked everyone into IE on Windows.

> The Linux desktop world is not perfect -- nothing ever is -- but, contrary to what khim claims, it's in very good condition.

Oh, I completely agree -- I've been using Linux as my only desktop environment since at least 1998 -- but I still have a Windows VM I need to fire up once in a while to deal with some stuff that WINE probably won't ever handle [1].

[1] Anything involving device drivers or badly-written [2] vendor firmware updating tools, for example
[2] That's probably redundant

Packaging Rust for Fedora

Posted Oct 31, 2022 13:31 UTC (Mon) by khim (subscriber, #9252) [Link] (1 responses)

> The specific calculator model he's complaining about has a Linux file manager that works with it.

Where is it and how can I find it? I only know that tilp still doesn't support it, but I admit, it was easier for me to just use Windows that to look around on unofficial forums for the version which may or may not work.

> Every bank I've ever encountered has a website that works in a Linux web browser.

Except you are a small business, then they would give you Windows-only bank client and if you are lucky key for that bank client would be on USB and not on floppy.

I remember how even LWN itself struggled with that and while maybe, just maybe they solved that problem, it took years.

Packaging Rust for Fedora

Posted Sep 14, 2023 2:12 UTC (Thu) by linuxrocks123 (subscriber, #34648) [Link]

Sorry for the late reply; I don't get email notifications. I believe from my browser history that this is the one I found advertised as working with that model: https://github.com/ErnyTech/nspire-tools/

Packaging Rust for Fedora

Posted Oct 31, 2022 6:52 UTC (Mon) by linuxrocks123 (subscriber, #34648) [Link]

Here's some help for you:

Calculator: https://www.reddit.com/r/nspire/comments/k5v7vy/cxcx2linu...

Taxes: It's not tax season but previous tax season you had a multitude of online options for filing taxes, some free.

Online Free: Credit Karma Tax, Free Fillable Forms
Online Pay: TurboTax, H&R Block

I am actually working on a libre tax software solution, but haven't had much time for it recently due to my day job: https://github.com/linuxrocks123/taxfloss

If you are at all interested, I'll push my 2021 forms to GitHub. They're not right now. The program has no user interface, but the domain-specific language works so if you declare variables with your input you can actually use it. (I did.) You still have to fill in the forms by hand by copying the output, though.

These people are taking a different approach and have a UI and something more usable currently: https://ustaxes.org/start

Packaging Rust for Fedora

Posted Oct 30, 2022 2:23 UTC (Sun) by dvdeug (subscriber, #10998) [Link] (3 responses)

> Heck, it's obvious that curated approach (Linux distros) have quite thoroughly lost to uncurated one (AppStores for Android, iOS or even Windows): all the “good things” are not in curated Linux distros, they are elsewhere.

Uncurated ones? Once upon a time, all programs were installed on Windows or MacOS by going to their website and downloading them or buying a physical copy. Now iPhones and some versions of Windows only install from the vetted app store, and Android can get pretty pushy about it. Yes, there's a lot less control over what goes into those AppStores than what goes into a Linux distro, but there's a lot more control by Microsoft and co. than there was previously. So not obvious at all.

Also note that the appstores carry binary apps that carry all of the non-system libraries with them. Flatpak is one thing, but a pile of source code that downloads a bunch of other source code and may or may not compile is quite another. Downloaded C code, in my experience, loves to fail in the middle of compilation with some obscure error from the compiler that the random user has no idea how to fix. Rust is probably better in some ways, but code is not going to beat someone compiling it on your arch on your distro and hopefully doing a smoke test and you downloading that for ease of use.

Packaging Rust for Fedora

Posted Oct 30, 2022 11:22 UTC (Sun) by khim (subscriber, #9252) [Link] (2 responses)

> Uncurated ones?

Yes. Maybe we have different definition of word “curated”? I go with normal one: carefully chosen and thoughtfully organized or presented. And chosen is critical part.

> Yes, there's a lot less control over what goes into those AppStores than what goes into a Linux distro, but there's a lot more control by Microsoft and co. than there was previously.

But there are, really, no curation. There are rules, sure. But they are uniform. There are no selection process which may decide whether fork of some product is original enough to be included. Programs are not renamed simply because some obscure program took their name first.

You can find curation in these stores, too, BTW. The list of libraries included in Android, iOS, macOS, Windows… it's curated by any definition. But distros manage to combine worst sides of curated and uncurated process! You have to convince them to include your app or library by doing certain, quite atrocious, things… and you you can not be certain that it would be available in that distribution! It must be installed separately and you can not even easily direct user to the store where he needs to just click “Install” to get another app!

The rest of your story talks about how good curated distro is for the user who is also a developer… and it's not incorrect. For the user who is also a developer distros are really cool. The only problem: these are extreme rarity novadays.

And for everyone else… distros are awful. They only offer curated software thus you can not get best software (it's not included for various reasons, most often because it just doesn't exist in version for GNU/Linux) and they only offer curated set of libraries thus you often find out that library you want or need is not included, or included in wrong version or can not be included because distro decided that it's duplicating functionality of some other library…

> Rust is probably better in some ways, but code is not going to beat someone compiling it on your arch on your distro and hopefully doing a smoke test and you downloading that for ease of use.

Looking on calendar. Is it 2022 or 2002? That issue is solved in the entirely different way today. Linux distros were never a good way to do that because you have to compile something on your arch and your distro (and you forgot your version of that distro, BTW) — and that never happens. There are just too many distros, too many versions to support all that zoo.

Instead you just use docker image to have predictable and reproducible builds.

That's how everyone except people who have already decided to spend insane amount of time to deal with bazillion linux distributions for no good reason are doing it today.

Packaging Rust for Fedora

Posted Oct 30, 2022 20:50 UTC (Sun) by dvdeug (subscriber, #10998) [Link]

> Yes. Maybe we have different definition of word “curated”? I go with normal one: carefully chosen and thoughtfully organized or presented. And chosen is critical part.

Given that Apple advertises "the apps we offer are held to the highest standards for privacy, security, and content. Because we offer nearly two million apps — and we want you to feel good about using every single one of them.", "Over 215K submissions rejected last year for violating privacy guidelines." and "Over 1M submissions rejected for objectionable, harmful, unsafe, or illegal content.", all on the front page of the app store, I think Apple is calling themselves curated.

It is fairly easy to become a Debian developer, and they can upload just about anything. Neither the Debian archive or the Apple app store presume to offer only the best word processor or best puzzle game; just all such programs that fit the basic rules and someone with right credentials packaged. I think you're making a lot more distinction then actually exists.

>Programs are not renamed simply because some obscure program took their name first.

Note that apt install snap doesn't install the appstore, it installs the general purpose gene finding program by the same name. You can make a trademark claim on the appstores, as well; note that a search for Tetris on Debian comes up with a lot more stuff calling itself Tetris then the Google Store.

>You have to convince them to include your app or library by doing certain, quite atrocious, things… and you you can not be certain that it would be available in that distribution! It must be installed separately and you can not even easily direct user to the store where he needs to just click “Install” to get another app!

I don't understand what this means.

> The rest of your story talks about how good curated distro is for the user who is also a developer

No. I run the mencoder contained in Debian despite the fact the mplayer developers fume about it. I could download the version from mplayerhq.hu and compile, and have in previous days, but don't feel like jumping through the extra hoops. Non-developer users would find compiling it much harder. Distros mean that you can install a program and it will work on your architecture, with your libraries, and not overwrite anything. (Note that snapd installs /usr/bin/snap, and snap doesn't. a compromise when snapd was added to the archive.) This is the same thing that the other Appstores offer, that any system that delivers source code or generic Linux binaries doesn't. This is what users need whether or not they're developers.

> (it's not included for various reasons, most often because it just doesn't exist in version for GNU/Linux)

? Are you complaining because Linux distributions don't package software that doesn't work on Linux?

> they only offer curated set of libraries thus you often find out that library you want or need is not included, or included in wrong version or can not be included because distro decided that it's duplicating functionality of some other library…

Non-developers never want to install a library. That's part of the advantage of distributions, that they never have to worry about installing libraries, they're automatically handled for them. They also keep the libraries up-to-date. so you aren't running an version of the libraries with known issues.

Again, with Debian, I don't recall them rejecting any library because it's duplicating functionality of some other library. If it wants to use the same file names, yes.

> Linux distros were never a good way to do that

To do what? Distros offer code compiled on your arch on your distro on the version of the distro with hopefully at least a smoke test. They do that job quite well.

> Instead you just use docker image to have predictable and reproducible builds.

From what I understand, docker images let you put down a turnkey database or web server on a server. They don't do anything for running various programs on the same X or Wayland screen. There are systems that run a program in its own box with its own libraries, but I have 1906 non-library Debian packages, and don't want to host all those libraries for each one.

Packaging Rust for Fedora

Posted Nov 2, 2022 15:11 UTC (Wed) by nim-nim (subscriber, #34454) [Link]

> There are no selection process which may decide whether fork of some product is original enough to be included.

There is no such selection process distro side either, the sole selection process is finding enough people motivated to build an n-th version of the same thing, and the only way to fail this process is to make building the n-th version too hard compared to the benefits it provided over the (n-1) other versions.

Which, is really easy to do when you are pulling hundreds of other modules which are all needing review individually.

Packaging Rust for Fedora

Posted Oct 29, 2022 9:17 UTC (Sat) by moltonel (subscriber, #45207) [Link] (3 responses)

> The keyword here is 'curated'.

This is what the current system already provides. When you pull in a crate with lots of deps like bevy, it has acted as a curator selecting those deps. When you look up a crate's ecosystem-wide reverse dependencies, you're asking the community curator. When you're importing cargo crev/vet reviews, you're trusting an individual curator.

With proper tooling, smaller more numerous deps are actually easier to review than traditional C-style deps.

Packaging Rust for Fedora

Posted Oct 29, 2022 15:22 UTC (Sat) by darmengod (subscriber, #130659) [Link] (2 responses)

I'm heavily jaded against using "popularity" metrics as an indicator of a crate's (or any software package's) actual value.

Yes, there is oftentimes some correlation, but in Spanish we have a beautifully-fitting saying for this kind of thinking:

¡Coma mierda! Un millón de moscas no pueden estar equivocadas.

Which looks something like this when rendered into English:

Eat dung! There's no way a million flies are wrong!

The fact that crate A has a million (or any suitably impressive number) downloads doesn't tell me much. It can be the case that crate A is, in fact, great, or it can be the case that there is simply no truly good crate for this use-case, and crate A just happened to be the least-worse one that had a relevant enough name.

A million downloads also won't tell me anything about the developer effort behind the crate, and therefore whether I can expect reasonable fixes to be published when bugs/security issues are discovered, brilliant pull requests reviewed and merged, etc. Maybe the crate *is* great, but the one-guy effort behind it has burnt out and vanished into the ether without leaving a clear successor in his place. That's the sort of stuff that "community metrics" won't realiably tell you anything about.

This is just a rant, don't get me wrong. There's no real solution to those problems; it's our job as engineers to deal with the world as it is and make the best we can of it.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:05 UTC (Sat) by khim (subscriber, #9252) [Link]

> Maybe the crate *is* great, but the one-guy effort behind it has burnt out and vanished into the ether without leaving a clear successor in his place.

That's state of Rocket between august 2021 and february 2022.

> This is just a rant, don't get me wrong. There's no real solution to those problems; it's our job as engineers to deal with the world as it is and make the best we can of it.

Curated garden is not a solution either. Think boost: some of it's libraries are so unstable that many developers either refuse to use it entirely or only allow support some of it.

> Eat dung! There's no way a million flies are wrong!

Yeah, that's how we end up with production software written in a language that thinks that strings “1e3” and “1000” are identical. But even then… it took just a few years for axum to beat rocket. actix-web is still ahead despite the well-known fiasco, but the trend is obvious.

It's true that sometimes the most popular thing is not the best one… but it's still good approximation and one which distro developers use, too.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:09 UTC (Sat) by moltonel (subscriber, #45207) [Link]

Popularity is an important metric with pros and cons. It shouldn't be your most important metric but it shouldn't be ignored either.

Packaging Rust for Fedora

Posted Oct 29, 2022 2:47 UTC (Sat) by pabs (subscriber, #43278) [Link]

My biggest problem with Rust packaging by distros is that (in Debian at least), packagers use Rust crates as input. I personally consider Rust Crates as downstream packaging and think that distros should be based on upstream VCS repos instead of downstream packaging. I've seen situations where the contents of the crate were different to the upstream repo. For example missing test files, or extra generated files etc. In one particularly bad case (rand) there were generated Rust files created by a Python script using some other source file in the crate and the Python and the true source were not in the crate at all. This issue isn't specific to the Rust ecosystem though, Python sdists often are missing test/other files too.

I also note that there are some movements towards dynamic linking for Rust, which might one day be usable for distros. The standard name mangling work is almost concluded. Not sure what other relevant changes would be needed though.

https://wiki.debian.org/StaticLinking#Rust
https://lwn.net/Articles/797616/
https://github.com/rust-lang/rfcs/pull/2603
https://github.com/rust-lang/rust/issues/60705

Packaging Rust for Fedora

Posted Oct 29, 2022 8:32 UTC (Sat) by wtarreau (subscriber, #51152) [Link] (8 responses)

For me it illustrates once again something we've seen many times over the last decades. Some newbies with zero experience of in-field system maintenance and/or zero consideration for end-users expectations that "stuff that works must simply continue to work" tend to selfishly reinvent amazingly funny packaging systems that are solely meant to make developers' lives easier. But if there are 1000 users for one developer, the developer has great responsibilities and ought to accept something a bit more rigid that makes the users' lives easier. Sadly that's not what we're seeing in field and everybody has already met an application break after a tiny update of a language-specific package management system that works on its own and concurrently to the distro, which tries hard to maintain consistency for the user.

Distros have a very hard work trying to match users' expectations with some developers' whims and extravagant "maintenance" process, particularly when users want those developers' projects. It's a very difficult task and we must never forget that the only reason Linux is so popular is that there are people making this effort of aggregating all such software to make it work seamlessly. This at least deserves a bit of effort from developers of whatever language to make their packaging systems less eccentric and more respectful of what distros really need to bring the software to end users.

Here in my opinion the Rust community must try to be a bit more professional and open to those who are distributing their work, by learning from their long experience, listening to their constraints and adapt to them. It's not difficult, everyone had done this for 3 decades. If they don't it's not that they can't, it's that they just don't want because they think they know better, and usually such attitude tells a lot about what to expect in the long term.

Packaging Rust for Fedora

Posted Oct 29, 2022 14:41 UTC (Sat) by khim (subscriber, #9252) [Link] (6 responses)

> It's a very difficult task and we must never forget that the only reason Linux is so popular is that there are people making this effort of aggregating all such software to make it work seamlessly.

Linux is popular because it's free. It's used on so many devices where Linux distros are not used it's not funny.

And distros are popular because of FSF's failed to build a viable kernel yet refused to use Linux. This left a viable combo which needed someone who would turn it into a full product. Distros have solved that task and they have also solved another problem which no longer exist: high cost of over-the-network distribution. It was much cheaper to buy set of CDs or DVDs that to download stuff about 20-25 years ago.

> Some newbies with zero experience of in-field system maintenance and/or zero consideration for end-users expectations that "stuff that works must simply continue to work" tend to selfishly reinvent amazingly funny packaging systems that are solely meant to make developers' lives easier.

Isn't that description of Linux distros? They acted till very recently against that desire. It wasn't possible to make something which you can use in that fashion till docker, flatpack and snap solved that problem. And docket wasn't even developed by distros, it became popular against their will.

> Sadly that's not what we're seeing in field and everybody has already met an application break after a tiny update of a language-specific package management system that works on its own and concurrently to the distro, which tries hard to maintain consistency for the user.

Never have seen that in my work in last 10 years, at least. The fact that we don't use anything provided by the distribution in our apps (except for the kernel) was the key. Others have solved the issue with docker packages. They work fine for server deployment and that's where GNU/Linux matters. Flatpack and Snap are supposed to solve the issue on client, but because Linux client is much less important they are much less popular.

> Distros have a very hard work trying to match users' expectations with some developers' whims and extravagant "maintenance" process, particularly when users want those developers' projects.

Why is it a problem for users of Linux distros, but not a problem for everyone else? Android, iOS, macOS, Windows… all OSes break apps for time to time, but in these OSes breakage is rare exception. In Linux world it's the norm (except when flatpack or snap is used). Why is that? Why should developers of Rust care about fringe case more than about the norm?

> It's not difficult, everyone had done this for 3 decades.

Indeed. Everyone except for C and C++ developers ignored distros. They would, happily, continue to do that in the future.

Even C++ developers try to do that, but there are just too many competing options: Buckaroo, Conan, Vcpkg… none have become popular enough to replace distros… but they are working on that!

Packaging Rust for Fedora

Posted Oct 29, 2022 15:15 UTC (Sat) by pizza (subscriber, #46) [Link] (5 responses)

> Why is it a problem for users of Linux distros, but not a problem for everyone else? Android, iOS, macOS, Windows… all OSes break apps for time to time, but in these OSes breakage is rare exception. In Linux world it's the norm (except when flatpack or snap is used). Why is that?

The main reason for this is that Linux is not "owned" by anyone and is not a complete platform in of itself, whereas the others are, and perhaps more importantly are strictly controlled by a single corporation, either by the carrot (eg Google pouring billions of dollars of R&D into Android every year, out-investing everyone else combined) or the stick (eg Apple suing anyone who tries to color outside the lines).

Oh, and MacOS breaks existing stuff with nearly every single release, even putting aside their once-a-decade CPU architecture switches. Apple also forces developers to keep updating their iOS apps for nearly every major(==yearly) release lest they get booted from the App Store -- In no small part because their underling platform changes all the damn time. So Apple is a doubly-poor example of breakage; honestly Linux fares objectively _better_ in comparison.

Packaging Rust for Fedora

Posted Oct 29, 2022 15:42 UTC (Sat) by khim (subscriber, #9252) [Link] (4 responses)

> So Apple is a doubly-poor example of breakage; honestly Linux fares objectively _better_ in comparison.

Apple is still better. Yes, it gives developers new hoops to jump through from time to time, but you still can build one, single, package to give to users. Which usually covers at least 5 years of macOS or iOS development (today you can give x86_64 package and it'll work fine on M1/M2 devices, too).

Linux distros refused, for many years, to even entertain that possibility and when they, finally, made it possible it became a separate tool which they, themserves, are not using.

Compare that to Google approach where many Google first-party apps come in a form of something you download from Google Play. Not everything, sure, but many things are done “like developers would do that”.

For Linux distros flatpack and/or snap are some addons they may not care about at all (and when some distros do a baby steps are done in that direction others throw a temper tantrums).

Thus no… while Linux distros are, slowly, becoming more developer and user friendly and Apple becomes more developer and user hostile… they haven't swapped places yet.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:00 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> Linux distros refused, for many years, to even entertain that possibility and when they, finally, made it possible it became a separate tool which they, themserves, are not using.

If you are referring to Flatpaks, Fedora is increasingly using it. GNOME Software supports it and Fedora has Flathub (with a filter) enabled by default in the workstation edition. There is also ongoing work to expand the scope of that as well including https://pagure.io/fedora-workstation/issue/300.

Ubuntu has several snaps used by default as well.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:29 UTC (Sat) by khim (subscriber, #9252) [Link]

Well… that really cool. Thanks for correcting me. With how pushy Apple is lately and how badly it's treating developers novadays there's hope that Linux may, eventually, become easier to deliver for than at least macOS. That would be a good thing.

Packaging Rust for Fedora

Posted Oct 29, 2022 16:43 UTC (Sat) by pizza (subscriber, #46) [Link] (1 responses)

You're moving the goalposts -- the point was about forwards compatibility; ie taking an existing binary and the expectation of it working on a newer release of the platform/operating system. You know, the thing you were decrying Linux for completely lacking.

Apple requires developers to constantly stay on the bleeding edge with tools and platform releases, at the risk of having their stuff summarily booted from the App Store. Sure the latest release lets you target all (supported) prior releases with a single binary, but the point is that developers still have to generate and then submit those new binaries (including any necessary adjustments due to the latest platform changes that are often incompatible with prior releases, plus comply with whatever new requirements Apple has decided to impose since their last submission. (I'm talking iOS here, though MacOS has been pushing steadily in this direction too)

One can make a reasonable argument that this is actually a good thing for the overall platform health, but forward compatible it is not, and it's rather specious to decry Linux for this but praise Apple for the same.

Packaging Rust for Fedora

Posted Oct 29, 2022 17:51 UTC (Sat) by khim (subscriber, #9252) [Link]

> Apple requires developers to constantly stay on the bleeding edge with tools and platform releases, at the risk of having their stuff summarily booted from the App Store.

Now you are moving the goalposts. Have you ever see that letter which Apple sends if you don't rush to use newest and greatest features? I'll include it here, it's not long:

> This app has not been updated for a significant amount of time and is scheduled to be removed from sale in 30 days. No action is required for the app to remain available to users who have already downloaded the app.

You can keep this app available for new users to discover and download from the App Storey submitting an update for review within 30 days.

If no update is submitted within 30 days, the app will be removed from sale.

Yes, it's a problem for developers, too. But that doesn't happen with macOS (since you can “sideload” apps there) and even on iOS it very explicitly tells that no action is required for the app to remain available to users who have already downloaded the app.

Yes, very annoying, but… it's policy, not technical issue. And it very explicitly doesn't affect your ability to taking an existing binary with the expectation of it working on a newer release of the platform/operating system.

> One can make a reasonable argument that this is actually a good thing for the overall platform health, but forward compatible it is not, and it's rather specious to decry Linux for this but praise Apple for the same.

That's very strange reading of what I wrote:

> Thus no… while Linux distros are, slowly, becoming more developer and user friendly and Apple becomes more developer and user hostile… they haven't swapped places yet.

It's quite obvious that Apple becomes steadily worse (and risks very real lawsuits and other legal actions because of what they are doing). But it's still better than what Linux distributions doing, objectively speaking.

Packaging Rust for Fedora

Posted Oct 30, 2022 8:17 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

If you want developers to work a certain way, you can pay them to work that way. The practical reality is, the people who actually do pay their developers don't seem all that bothered by this sort of thing (probably because they're mostly doing bespoke containers and such, and not bothering with distros at all, if we ignore RHEL for a moment).

Packaging Rust for Fedora

Posted Oct 29, 2022 21:53 UTC (Sat) by seanyoung (subscriber, #28711) [Link] (35 responses)

Arch linux has a simple sensible approach to rust packaging.

https://wiki.archlinux.org/title/Rust_package_guidelines

Packaging Rust for Fedora

Posted Oct 30, 2022 8:57 UTC (Sun) by cyperpunks (subscriber, #39406) [Link] (34 responses)

Well, seems Arch allows builders to download stuff during build, that not the case for any other distros such as Debian, Ubuntu, RHEL family, Fedora and openSUSE.

Packaging Rust for Fedora

Posted Oct 30, 2022 9:32 UTC (Sun) by seanyoung (subscriber, #28711) [Link] (30 responses)

Why that rule of not downloading anything during build? If the concern is reproducible builds,
then the arch way of using cargo --locked works fine. If the concern is that they don't trust
crates.io, then Fedora can easily run their own crates registry which mirrors rust crates.io.

The problem with all these rules like "builds cannot download" is that it is simply not how development is done any more.

Fedora packages are just becoming more and more useless, and folks like me will just not use Fedora packages any more (just use cargo install to install binaries) or move to a sensible distribution like arch.

Packaging Rust for Fedora

Posted Oct 30, 2022 12:26 UTC (Sun) by ssokolow (guest, #94568) [Link] (5 responses)

As I understand it, being someone who maintains a flatpak and helped get another one prepared:

1. Unified auditing. All dependencies have their sources and hashes pinned down in a single, unified record.

2. Separation of sandbox rules between the download phase (only tools like flatpak-builder can run code, network access) and the build phase (arbitrary package code runs, no network access)

Part #2 helps to prevent things like using package code to co-opt your build machine/farm as a vector to skirt around your border firewall and attack softer targets within your LAN. (Assuming, of course, that this is a "this LAN builds distributables for package X but doesn't run them" situation.)

Packaging Rust for Fedora

Posted Oct 30, 2022 15:25 UTC (Sun) by gilnaa (guest, #161422) [Link] (4 responses)

You can achieve this by hosting your own registry, which is basically just another http server.

Packaging Rust for Fedora

Posted Oct 30, 2022 16:16 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

Note that my idea (that I've posted about before) is that installing a Rust package puts its crate into a machine-local registry. This would allow for automatic resolution to distro-provided versions based on `BuildRequires:` and the like.

Note that just rehosting `crates.io` might not be preferrable since they can be post-codegen steps (of various kinds) and instead `cargo publish` into this registry that is then pulled out during some `rpm-cargo-*` macro magic would probably be a better mechanism.

Packaging Rust for Fedora

Posted Oct 30, 2022 16:18 UTC (Sun) by ssokolow (guest, #94568) [Link] (2 responses)

How does that solve the "unified" requirement? A flatpak manifest puts all the hashes and URLs in the same JSON or YAML file, regardless of what language each dependency is written in.

Packaging Rust for Fedora

Posted Oct 30, 2022 16:43 UTC (Sun) by mathstuf (subscriber, #69389) [Link] (1 responses)

Just because you have the hash doesn't mean you have the source available when you need/want it. If some registry goes down due to a DNS renewal failure, "hostile" acquisition, or other fun things, that hash list is quite useless.

Packaging Rust for Fedora

Posted Oct 30, 2022 17:16 UTC (Sun) by ssokolow (guest, #94568) [Link]

Of course. I'm saying that, just because you have a local Cargo registry, and a Conan registry, and a pip registry, and so on doesn't mean you have a clean, unified, easy-to-use way to check your bill of materials.

Conversely, if you have something like a flatpak manifest, you can use flatpak-builder arguments like --download-only, --disable-download, and --bundle-sources to easily cache and manage access to dependency sources.

Packaging Rust for Fedora

Posted Oct 30, 2022 14:49 UTC (Sun) by kleptog (subscriber, #1183) [Link] (6 responses)

I believe the original issue was GPL: for a given binary you must be able to deliver the source code it was built from. If the build process downloads files randomly from the internet, how can you produce "the original source"?

Nowadays we have the idea that a hash uniquely defines a given release of code and can be used to actually find the corresponding source. And that you can sort of rely on the idea that it will not randomly disappear off the internet. So it might be kind of feasible to carve out an exception there, where the builder has a proxy that registers which creates were downloaded and includes the relevant hashes in the package metadata.

But if a package just downloaded other assets off the internet it could become very difficult to reproduce. Maybe hook the builder into the internet archive and cross your fingers? Easier just to forbid the builder access to the internet and expect the uploader to predeclare dependencies.

People building closed source software are generally less concerned about reproducibility.

Packaging Rust for Fedora

Posted Oct 30, 2022 17:10 UTC (Sun) by khim (subscriber, #9252) [Link]

> And that you can sort of rely on the idea that it will not randomly disappear off the internet.

Please don't. In 10 years it would disappear.

Just put all the crates you need on the server you own. Then you can use hashes to your heart's content.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:22 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (4 responses)

It's not necessarily an issue, because GPL 3 says this:

> The "Corresponding Source" for a work in object code form means all
> the source code needed to generate, install, and (for an executable
> work) run the object code and to modify the work, including scripts to
> control those activities. However, it does not include the work's
> System Libraries, or general-purpose tools or generally available free
> programs which are used unmodified in performing those activities but
> which are not part of the work. For example, Corresponding Source
> includes interface definition files associated with source files for
> the work, and the source code for shared libraries and dynamically
> linked subprograms that the work is specifically designed to require,
> such as by intimate data communication or control flow between those
> subprograms and other parts of the work.

So if the thing you're downloading is a "generally available free program," then it does not need to be included. Since the material in question is *possible* to download from the internet (presumably, from an automated script with no human intervention), that strongly suggests that it is generally available at least.

While the preamble of the license does talk about free software in general terms, neither the preamble nor the definitions section give an explicit description of what exactly qualifies as a "free" program (for example, the four freedoms are nowhere to be found). Courts will use the dictionary definition, and (probably) conclude that a "free" program is one which costs $0. I'm sure this would annoy the hell out of RMS, but that is how courts read contracts and licenses - only the material within the four corners of the document is part of the agreement. The fact that the Free Software Foundation chooses to use the word "free" in a way that is different from the general public is, legally, their problem, and they need to clarify that in the license if they want courts to respect it.

Packaging Rust for Fedora

Posted Oct 31, 2022 8:29 UTC (Mon) by Wol (subscriber, #4433) [Link] (3 responses)

> While the preamble of the license does talk about free software in general terms, neither the preamble nor the definitions section give an explicit description of what exactly qualifies as a "free" program (for example, the four freedoms are nowhere to be found). Courts will use the dictionary definition, and (probably) conclude that a "free" program is one which costs $0.

WHICH dictionary definition? Courts are (supposed to be) intelligent, and given that Free has a clearly defined meaning in the FLOSS world, I'm sure a UK court would conclude this passes the "knew or should have known" test.

Incidentally, that's why I always use the abbreviation FLOSS, not FOSS.

Cheers,
Wol

Packaging Rust for Fedora

Posted Oct 31, 2022 8:55 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (2 responses)

> WHICH dictionary definition? Courts are (supposed to be) intelligent, and given that Free has a clearly defined meaning in the FLOSS world, I'm sure a UK court would conclude this passes the "knew or should have known" test.

There are several different problems with that argument:

0. As I mentioned, the meaning of a legal document is generally determined by its contents, not by other stuff, no matter how relevant that other stuff might seem. If the four freedoms are not in the license, then they are Not In The License. This is already the usual legal rule, but it is reinforced by section 7's ban on "further restrictions."
1. Anyone could be a licensee, not just people "in the FLOSS world."The GPL is routinely presented in installer wizards (on Windows) where an EULA would normally be displayed, and the end user might not have any idea that there is a such thing as "the FLOSS world." So you can't necessarily expect them to be familiar with nonstandard terminology.
2. The GPL is probably[1] a contract. So contra proferentem applies, and common law courts will (usually) prefer to interpret ambiguity in favor of the non-drafting party, meaning you get the more permissive definition if more than one would make sense.
3. Let's use [2] as an example dictionary, because it's the sort of dictionary a court might use (i.e. a regular everyday dictionary, not a specialized law dictionary or something). It gives these senses of the word "free," if we cut out all of the irrelevant senses (of which there are quite a few):

> 1: not costing or charging anything
> a free school
> a free ticket
> 2a: having the legal and political rights of a citizen
> For many African Americans, celebrating the Fourth of July as the day Americans became free from British rule feels inapplicable since our ancestors were not free.
> — Christen A. Johnson
> b: enjoying civil and political liberty
> free citizens
> c: enjoying political independence or freedom from outside domination
> This is a free country.
> d: enjoying personal freedom : not subject to the control or domination of another
> You are free to do whatever you want.
> 3a: not determined by anything beyond its own nature or being : choosing or capable of choosing for itself
> a player free to negotiate a contract with any team
> b: determined by the choice of the actor or performer
> free actions
> c: made, done, or given voluntarily or spontaneously
> gave his free consent

The basic problem here is that (1) is the only sense (of those three) that can obviously be applied to inanimate objects like software, just based on the literal wording of these definitions. It would not make sense, for example, to give software "the legal and political rights of a citizen" or to describe it as "capable of choosing for itself," unless we're talking about some kind of general AI system that doesn't (yet) exist. You can argue that the FSF's use of the term is a perfectly reasonable extension of (2) and/or (3), but it is not literally identical to (2) or (3), so the average court probably isn't going to go out on that limb.

[1]: See e.g. https://lwn.net/Articles/747563/
[2]: https://www.merriam-webster.com/dictionary/free

Packaging Rust for Fedora

Posted Nov 1, 2022 13:30 UTC (Tue) by kleptog (subscriber, #1183) [Link] (1 responses)

> 0. As I mentioned, the meaning of a legal document is generally determined by its contents, not by other stuff, no matter how relevant that other stuff might seem.

AIUI this is a Common Law thing. In much of the world the legal framework takes into consideration what was intended, how the parties understood it and how a reasonable person would understand it. That's why you don't need to add "don't be an asshole" to contracts, it's implied.

Yes, this trips up US/UK companies occasionally when they come up to a European court with the defence: "we adhered to the letter of the contract". Whether courts will understand the correct meaning of "free" is not something I worry about.

Packaging Rust for Fedora

Posted Nov 1, 2022 23:37 UTC (Tue) by ssokolow (guest, #94568) [Link]

It also trips up people from commonwealth and former commonwealth countries when they learn that saying "I'm putting this into the public domain" isn't enough in jurisdictions like Germany.

See this legal review of the CC0 for more on that.

(TL;DR: The CC0 has a license fallback because German law doesn't have a concept of prematurely giving up your copyright, and it has a legally binding promise not to sue, because German law doesn't let you pre-emptively sign away rights and protections that will be written into the law at some future date.)

Packaging Rust for Fedora

Posted Oct 30, 2022 15:14 UTC (Sun) by smoogen (subscriber, #97) [Link]

There were several reasons for not downloading anything during the build
1. For many volunteers, networks are expensive in monetary costs to latency costs. Having the source code local means that recompiles and rebuilds 'cheaper'. Putting the builds in a 'must already be local' says that the environment building the final artifact matched what the developer expected as best as possible. Again it is a tradeoff. [Even today we have volunteers who have low bandwidth networks. Instead of pulling down similar 'cargo' or other packages 1000 times if they are building a lot of things, they would prefer to get it once and use it 1000 times.]
2. security. having the source code local means that someone checked it in at some point. having it automatically pulled in during the build tends to end up getting sidelined with 'X took over Y account and uploaded crapware'. the goal is to have a person making 'informed' choices before it is shipped to others. It isn't 100% foolproof but it is a tradeoff.
3. reproducible and auditable builds. not in the way of you rebuild it again and you get the same binary but that you can find and use every tool that was used to build the artifact. This is a second item towards various attacks which has shown up in Nodejs, PHP, Perl, Python, Ruby, and some other tooling where you find that the version you thought was built was replaced one way or another. [It has been a rare case, but for various developers and 'consumers' bitten once it has been enough to say never again.]
4. Ability to meet specific consumer environment demands. Eventually some code ends up getting used by something which has a high regulatory requirement. Those tend towards capture all the bits, lock them down, and show how it was build 20 to 50 years down the road. [I don't know if it is possible or not but legally it is required to be tried to make possible for various places (medical, automobile, energy production, various government agencies, international organizations, etc)].
5. -> N. the above is not comprehensive and people will point out other reasons

Again this is all about tradeoffs. The method used by older distros doesn't solve all the problems, but it solves the ones that various people wanted to be solved at the time they were important to them. Once built it becomes very hard to change without burning the ocean which is why other distros do show up over time and 'take' over specific developer and user needs. I expect that if someone had to meet the needs in 4 with 'arch' they would build a specific artifact storage that their builders would store every artifact downloaded so it could be reviewed and checked later if needed.

Packaging Rust for Fedora

Posted Oct 30, 2022 15:23 UTC (Sun) by cesarb (subscriber, #6266) [Link]

> Why that rule of not downloading anything during build? If the concern is reproducible builds, then the arch way of using cargo --locked works fine.

The concern is not reproducible builds, but being able to build in the first place. All major distributions consider failure to build from source (FTBFS) a critical bug in a package. Depending on a third-party server for the build means a package could suddenly not be able to be built anymore, if anything happens to that third-party server.

> If the concern is that they don't trust crates.io, then Fedora can easily run their own crates registry which mirrors rust crates.io.

That would put Fedora itself in that "third-party server" position whenever someone else tried to build a Fedora package from source.

Packaging Rust for Fedora

Posted Oct 30, 2022 21:24 UTC (Sun) by Wol (subscriber, #4433) [Link] (10 responses)

> The problem with all these rules like "builds cannot download" is that it is simply not how development is done any more.

So what do you do when you want to do a build, and your internet is shitty or non-existent? (Oh - and I've just spent a week with download speeds measured in kb - that is, if I could get a working connection ...)

Cheers,
Wol

Packaging Rust for Fedora

Posted Oct 31, 2022 4:37 UTC (Mon) by pizza (subscriber, #46) [Link] (9 responses)

> So what do you do when you want to do a build, and your internet is shitty or non-existent? (Oh - and I've just spent a week with download speeds measured in kb - that is, if I could get a working connection ...)

My last gig was in the medical field, where it could take 5+ years to get something in front of the first customer, and needs to be supported for a minimum of 10 years beyond that. The project I was working on involved a lot of 3rd-party Python code, and I am not exaggerating when I say that keeping up with and resolving python module incompatibilities consumed the majority of my time. We would routinely have one build to the next fail because a non-pinned three-layers-deep sub-dependency changed underneath us. On top of different sub components trying to pin different major versions of the same underling library. One of the last things I ran into, just before the whole group was shut down, was a dependency started requiring a newer version of python than our underlying platform provided.

Combine that with researchers just pulling in whatever python libraries they wanted, often (by necessity) on the bleeding edge. So each build would routinely pull down hundreds of python packages from the internet, on top of a custom debbootstrap'd Ubuntu build for our custom boards using bleeding-edge Xilinx SoCs -- It took over three hours to put together the bootable flash image, when everything worked.

I warned the powers that be that this was at best a prototype and that in order for it to be remotely supportable for the 15-year product lifecycle, we'd have to invest a _lot_ more into it, including (at minimum) a complete private mirror of Ubuntu, the python/pip and nodejs ecosystems, and a couple dozen github repos. But really, if we wanted this to be anything other than a toy/tech demo it would need to be rewritten, probably in C++ as that was what most of the rest of the company's long-term-maintained software was written in -- easily tens of millions of lines.

But anyway. In order to pass certification, we'd have to be able to prove the provenance of every bit of code running and guarantee the ability to recreate any given deployed build for fifteen years -- which means we can't rely on _anything_ downloaded from the public internet at build-time.

Packaging Rust for Fedora

Posted Oct 31, 2022 7:53 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> But anyway. In order to pass certification, we'd have to be able to prove the provenance of every bit of code running and guarantee the ability to recreate any given deployed build for fifteen years -- which means we can't rely on _anything_ downloaded from the public internet at build-time.

A practical way to solve it is to just mirror the heck out of artifacts. Since everything is hash-locked, it should be straightforward to just use these hashes for something like IPFS.

This is actually something that FSF should be doing. Just set up a huge mirror that pulls all the artifacts referenced from lockfiles of released projects.

Packaging Rust for Fedora

Posted Nov 2, 2022 16:29 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (1 responses)

The problem is not the mirroring, it's the checking (and maintaining).

If you freeze down a dependency for 15 years, and have any form of liability, you need to check every single fix released during those 15 years upstream to see if it applies to your configuration.

To a layman (judge) not applying a “free, public” fix will make you guilty as hell in case of a problem.

Packaging Rust for Fedora

Posted Nov 2, 2022 18:41 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

If you're working in an environment where you're legally liable for the quality, then you'll have to store all the previous versions and lock dependencies even more. You sure as hell won't just apply random updates from distros without testing them.

Packaging Rust for Fedora

Posted Oct 31, 2022 13:39 UTC (Mon) by khim (subscriber, #9252) [Link] (4 responses)

> On top of different sub components trying to pin different major versions of the same underling library.

Note that in Rust world it's supported use-case. I am not saying it's good idea to have that in your project, but it works. Rust and cargo support that combo just fine.

Except when you need to use two versions in one component, but that's next level of crazyness.

Packaging Rust for Fedora

Posted Nov 3, 2022 19:22 UTC (Thu) by mb (subscriber, #50428) [Link] (3 responses)

> Except when you need to use two versions in one component, but that's next level of crazyness.

You can also use two different versions of the same dep-crate in one crate simultaneously, as long as you map at least one of the deps to a different name (under which it is then accessible in the Rust code). That's easily possible with Cargo.toml:

[dependencies]
thedep02 = { version = "0.2", package = "thedep" }
thedep = { version = "1.0" }

Packaging Rust for Fedora

Posted Nov 3, 2022 19:50 UTC (Thu) by khim (subscriber, #9252) [Link] (2 responses)

Touché. This is minor improvement, though: if you are already at the point where you have different versions of the same component as immediate dependency if your code then you are half-step from trying to use types defined in one version for the other version.

This doesn't work in Rust and I don't think it's worth introducing meta-object protocol to allow that (although some languages [actually do that](https://www.gnu.org/software/guile/manual/html_node/The-Metaobject-Protocol.html)) but I just don't think it's what Rust needs.

Packaging Rust for Fedora

Posted Nov 3, 2022 21:11 UTC (Thu) by mb (subscriber, #50428) [Link]

> but I just don't think it's what Rust needs.

It's a feature that types and traits from incompatible versions of the same crate are in fact different types and traits.
That enables you to implement multiple versions of a trait on one single struct simultaneously.
That is sometimes useful, if you want to implement a library crate that must provide multiple different versions of an API (a foreign trait) to its users.

Packaging Rust for Fedora

Posted Nov 4, 2022 13:31 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> if your code then you are half-step from trying to use types defined in one version for the other version.

There are ways around this. For example, log 0.3.9 depends on 0.4 to serve as a bridge between the two nominally semver-incompat versions and not force an ecosystem-wide upgrade.

https://crates.io/crates/log/0.3.9/dependencies

Packaging Rust for Fedora

Posted Oct 31, 2022 14:57 UTC (Mon) by taladar (subscriber, #68407) [Link]

To me that seems very much like a problem with your company promising things that are practically impossible while using third party open source code unless you pay each maintainer of each of those pieces of code.

Packaging Rust for Fedora

Posted Nov 2, 2022 15:04 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (3 responses)

> Why that rule of not downloading anything during build?

Distribution build systems do download things before build but from the pile of things already packaged and audited not a blind internet pull. The magmas do exactly the same their stuff is not pulling from the internet but from internal audited repositories (giant curated monorepos).

The reason the stupid gratuitous inflation of artifact numbers and versions works for the magmas but not for distributions is that the magmas have more (paid) people to hurl at checking their repositories than distributions. And, more managers to hurl at devs that try to use non-blessed artifacts. And, more lawyers to hurl at people seeking damages if stuff fails. And, finally, a software world where they are the only ones able to provide trusted binaries suits them just fine.

The reason the stupid gratuitous inflation of artifact numbers and versions works for Joe nobody developer is that his liability if his stuff does not work is limited, so he does not feel the need to check the things his build systems pulls from everywhere, and he does not understand nor supports or helps the idiots that try to pull out this Sisyphean task distro-side.

Packaging Rust for Fedora

Posted Nov 2, 2022 15:40 UTC (Wed) by nim-nim (subscriber, #34454) [Link]

Also, finally, not a problem to develop with whatever artifact version you want as long as you are able to clamp down at end-release time and rebase on the version the release team blessed (be it the one Google chose to release its latest kubernetes version with or the one Debian of Fedora chose to base their distro on).

It’s this clamping down devs object to, even though it happens the same way distro and magma-side, because it is materially impossible to check every possible artifact version under the sun, and even the magmas are not rich enough yet to attempt it.

Packaging Rust for Fedora

Posted Nov 2, 2022 16:39 UTC (Wed) by seanyoung (subscriber, #28711) [Link] (1 responses)

> > Why that rule of not downloading anything during build?
>
> Distribution build systems do download things before build but from the pile of things already packaged and audited not a blind internet pull. The magmas do exactly the same their stuff is not pulling from the internet but from internal audited repositories (giant curated monorepos).

So my proposal was to use cargo --locked with a distribution-owned copy of crates.io. That means exactly the opposite of a "blind internet pull". I means neatly a curated listed of packages that distribution can audit and lock down precisely.

> The reason the stupid gratuitous inflation of artifact numbers and versions works for the magmas but not for distributions is that the magmas have more (paid) people to hurl at checking their repositories than distributions.

Not at all. Modern tooling like rust is finally getting the reusability of software components right. Many rust crates do one thing, and do one thing very well. Some crates are very sophisticated and some serve a simple purpose; either way, you don't want to open-code all these components yourself. Do you want to open-code how to get the terminal size, for all the unixes and Windows etc or use a simple crate? I do, I don't want to waste time getting some icky ioctl right on some little-used OS, and even if I did, my code will almost certainly have bugs in it because it is not as widely tested.

The tooling provided by distributions is no good for software development. I want to be able to select versions of libraries, not just the one provided by the distribution, possibly point it towards a git repo where I have some of my own patches. I want automatic (recursive) dependency downloading, based on semver. I want to be able to enable features on libraries, not just the features the distribution provides (i.e. ./configure --enable-foo etc). All of this simply by having a simple config file which lists by dependencies, which I can update at any time without changing my distribution. This is what rust/cargo provides.

In my 30 years of software development it has been a constant that the distribution does not provide the version/fix/etc I need, so I have to work around it. The distribution package tooling is utterly useless for software development so very few software developers use it.

I get that the task for distributions is so much harder because of it. How about working upstream with cargo/rust? It's a very warm and welcoming community. Surely there something we can fix together. The result would be amazing.

Packaging Rust for Fedora

Posted Nov 2, 2022 17:22 UTC (Wed) by Wol (subscriber, #4433) [Link]

That pretty much sums up why I run gentoo ...

I was running the latest most up-to-date SUSE, I needed the latest, most up-to-date lilypond, and never the twain shall meet ...

Cheers,
Wol

Packaging Rust for Fedora

Posted Oct 30, 2022 18:48 UTC (Sun) by mmstick (guest, #161882) [Link] (1 responses)

Actually, packaging systems for each of the Linux distributions mentioned supports downloading files before the source tarball is generated. Using Debian as an example, the standard tool for building packages is sbuild. A source tarball is created after executing `debian/rules clean` on the build server with access to the Internet, and then it creates a chroot from that tarball for compiling it in an offline environment. The clean rule is invoked a second time within the chrooted build environment, and you can check for this with the `ischroot` command.

Pop!_OS has been creating Debian packages from Rust software successfully for the last 5 years, and this is the template that we use for generating the source tarballs and building packages with sbuild. The first run of the clean rule outside of the chroot vendors the dependencies into a tarball, and the second run of the clean rule inside of the chroot ignores that step. The build rule extracts the vendored tarball


CLEAN ?= 1
VENDOR ?= 1

override_dh_auto_clean:
	if test "${CLEAN}" = "1"; then \
		cargo clean; \
	fi

	if ! ischroot && test "${VENDOR}" = "1"; then \
		mkdir -p .cargo; \
		cargo vendor | head -n -1 > .cargo/config; \
		echo 'directory = "vendor"' >> .cargo/config; \
		tar pcf vendor.tar vendor; \
		rm -rf vendor; \
	fi

override_dh_auto_build:
	if test "${VENDOR}" = "1"; then \
		rm -rf vendor; \
		tar pxf vendor.tar; \
	fi
	dh_auto_build

Packaging Rust for Fedora

Posted Oct 30, 2022 20:07 UTC (Sun) by walters (subscriber, #7396) [Link]

Instead of stock `cargo vendor`, you probably want https://github.com/coreos/cargo-vendor-filterer
xref https://github.com/coreos/fedora-coreos-tracker/issues/1240 etc.

Packaging Rust for Fedora

Posted Nov 3, 2022 2:38 UTC (Thu) by firstyear (subscriber, #89081) [Link]

OpenSUSE does not allow network access during builds but we do allow vendoring in rust packages. We also have supporting tooling to work with our product security to scan for security issues, update them, and respond.

We have a flourishing rust ecosystem with many new package maintainers, and I even presented at Rust Conf about the approach and how successful it has been.

The primary driver to go "vendoring only" was that there will always be "an exception" where vendoring is the only choice and you can't use distro packages. So why split the problem - just choose a single correct work flow, and make sure it's robust.

(Note - I am the maintainer of Rust in SUSE and OpenSUSE).

Packaging Rust for Fedora

Posted Oct 31, 2022 17:19 UTC (Mon) by rjones (subscriber, #159862) [Link] (24 responses)

Share libraries is a blight, not a feature.

The original point of shared libraries was because computer resources were extremely limited and having shared libraries was suppose to reduce resource usage somewhat. This, very obviously, didn't work out.

Huge amounts of resources are poured just in making shared libraries work. It dictates the design of software, it dictates the design of the OS (for the worse), and it causes monumental efforts being spent just to make shared libraries work, yet is a endless source of headaches for users and developers a like. Just think about all the pain and suffering that has gone into managing Debian's spaghetti nightmares of circular dependencies, docker swarms, kubernetes, snap files, appimages, and all the rest of it just to make software easier to install and manage in a world were managing shared libraries and other dependencies is a major stumbling block to getting anything done at all.

And still most people using Linux seriously spend a lot of their time just figuring out the best way to avoid having OS changes make invasive changes in on the software they use.

When most of it can be replaced with something as simple as:

curl -o /usr/local/bin/libreoffice-7.4.2 https://libreoffice.org/download/libreoffice-7.4.2.bin
chmod +x /usr/local/bin/libreoffice-7.4.2

...If we used static binaries. We can have a package manager that only needs to know how to check signatures and were to download the latest version of software. It doesn't even need to know how to uninstall the old version since 'rm' is more then sufficient. Being able to juggle multiple versions and roll back changes become free. etc etc. Slackware's old tar.gz packaging format is peak software packaging technology so far.

So while I am sure that there can be a huge amount of improvement in managing compile-time dependencies for Rust programs and reducing hundreds of crates to just a few... trying to figure out how to get them to use shared libraries at runtime would represent a huge step backwards technology-wise.

Static binaries should be were we are all heading.

Packaging Rust for Fedora

Posted Oct 31, 2022 17:37 UTC (Mon) by smoogen (subscriber, #97) [Link] (4 responses)

Having also gone through the birth of shared libraries in various OS's in the late 1980's... moving to shared libraries wasn't also because computer resources were extremely limited. It was also that you could have one version of libgzip on the system and when you updated it.. you didn't need to worry about the 600 applications which had it compiled statically in. [Which was what I spent 1 to 2 years dealing with even into the early 2000's when you would find that some subset system still had static libraries compiled in which were vulnerable to various 'crap'.]

I agree that shared libraries are not great and having to deal with dependency hell has been a side effect of it. However static binaries have not been great either. They both have tradeoffs which tend to get overlooked. I guess in the end all software is crap..

Packaging Rust for Fedora

Posted Nov 1, 2022 9:31 UTC (Tue) by roc (subscriber, #30627) [Link] (3 responses)

For Rust, at least, it's easy to capture the list of dependencies for each statically linked binary and query those lists to see what needs to be updated.

Packaging Rust for Fedora

Posted Nov 1, 2022 13:10 UTC (Tue) by pizza (subscriber, #46) [Link] (2 responses)

Great, now what do you do when you don't have the full corresponding source code for that application and all of its statically-linked dependencies?

Packaging Rust for Fedora

Posted Nov 1, 2022 23:40 UTC (Tue) by ssokolow (guest, #94568) [Link] (1 responses)

Only crates.io admins can truly pull something from the site once it's published and they reserve that ability for things like legal compliance.

cargo yank keeps the package version available for download by any downstream package that has it in its Cargo.lock file.

Packaging Rust for Fedora

Posted Nov 2, 2022 5:00 UTC (Wed) by pabs (subscriber, #43278) [Link]

Crates don't necessarily contain source code, for eg the rand crate contains files generated by a Python program from another file. The Python script and the real source isn't in the crate, just in the VCS. Of course, the VCS could have the same issue, but that is a bit less likely these days.

Packaging Rust for Fedora

Posted Oct 31, 2022 19:29 UTC (Mon) by sfeam (subscriber, #2841) [Link] (18 responses)

"Static binaries should be were we are all heading." 100% disagree on that one, at least if we're talking about desktop machines or servers that I have to manage myself. The primary benefit of share libraries was, and remains today, that you can replace a problematic library once, in one place, and fix all the programs that use it.

Packaging Rust for Fedora

Posted Oct 31, 2022 19:51 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (17 responses)

> The primary benefit of share libraries was, and remains today, that you can replace a problematic library once, in one place, and fix all the programs that use it.

How often do you actually do this in practice? Pretty much all the updates come through distributions, and since they have source code for everything, they can just rebuild static binaries anyway.

I develop pretty complicated systems and I don't think I have EVER wanted to replace a shared library with a custom version for ALL applications. On the other hand, replacing a library just for one application was required quite often.

Packaging Rust for Fedora

Posted Oct 31, 2022 20:11 UTC (Mon) by sfeam (subscriber, #2841) [Link] (11 responses)

Yeah, but the updates that come from the distribution might or might not fix the glitch I care about, and might actually introduce new problems. It's not that infrequent that I either have to "downgrade" a distro library to get back to a working version, or bypass the distro to get newer upstream source to build a version the distro hasn't packaged yet. I'm not talking about security fixes - I'm content to leave that for the distro. I'm thinking of things like bad font handling by multiple versions of libpango that percolate up and glitch text display by many programs, including ones I'm fielding bug reports for. Or audio glitching in bad codec library versions.

Packaging Rust for Fedora

Posted Oct 31, 2022 20:13 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

> Yeah, but the updates that come from the distribution might or might not fix the glitch I care about, and might actually introduce new problems

Can you provide concrete examples of such changes?

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Oct 31, 2022 21:31 UTC (Mon) by sfeam (subscriber, #2841) [Link] (9 responses)

Two examples I am still tracking because up- or down- grading from their current distro version is the best advice I can give to users who report problems.

font handling:

libpango 1.43 - good
libpango 1.44 - bad
libpango 1.45-47 - ? (did not see any reports for these versions)
libpango 1.48 - good

numerical library:

libcerf 1.7 - good
libcerf 1.8 - 1.10 - (bad convergence)
libcerf 1.11 - good

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 1, 2022 9:12 UTC (Tue) by farnz (subscriber, #17727) [Link] (8 responses)

This is, however, working around a failure of the distros. If they're providing a carefully curated, ready-to-use version of the software, then they should not be shipping versions that cause issues, and they should be reverting updates that do cause issues or fixing forwards rapidly, so that end users (the people you're giving advice to) don't have to upgrade and downgrade individual dependencies themselves. End users (including developers like me) only want to think in terms of the final complete application, not its full dependency graph.

Distros, in turn, make this part of their job harder on themselves by requiring a single version of each dependency distro-wide (and yes, this has other benefits); that means that if (say) Scribus needs features from libpango-1.44, while Inkscape has problems with libpango-1.44 but not libpango-1.43, they're stuck until the pango dependency is fixed by the distro or by upstream. Static linking inherently avoids this, since they could then ship just Scribus with libpango-1.44, while keeping Inkscape locked to libpango-1.43 until the bug is resolved. In this setup, you wouldn't need to advise people on which version of libpango to have installed, since they'd have the "right" version for the application they were running.

As with so much else in life, this is a set of tradeoffs. The distros have found a set of tradeoffs that acts as a local minimum for C and C++ applications (which is great!), works well enough for interpreted languages, and doesn't work well for Java, C# and Rust. They got away with it not working well for C# and Java because those languages run in their own VM anyway, and the VM is a C application, but they're now having to look for the next local minimum if they are to ship Rust applications.

My hope is that the result of hunting for a local minimum for Rust applications will result in the distros finding a better overall state - not just for Rust, but also for C and C++ applications.

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 1, 2022 17:36 UTC (Tue) by sfeam (subscriber, #2841) [Link] (7 responses)

In the case of pango, my understanding is that the breakage was deliberate, in the sense that someone made a choice between adding support for new font types on very high resolution displays was more important and keeping in place a framework that was designed for older font types and displays with a mere 95dpi (see e.g. https://github.com/harfbuzz/harfbuzz/issues/2394#issuecom...). On the one hand I have a bit of sympathy for pango having to choose between a new framework that is supportable going forward and an old framework whose support has languished. On the other hand, that means people downstream who were perfectly happy with desktop text rendering on their existing systems saw it as breakage. It's a whole lot easier to advise "replace this one library and your desktop will work again" than to say "you'll have to rebuild your whole desktop infrastructure to fix this, or stick with an old version of the whole system, or buy new monitors or change the fonts in your existing apps and documents, etc". The distros were caught in the middle, and may well not have been aware of the forced choice between backward and forward compatibility.

This particular problem is not one of "program X requires version a.1, while program Y requires version a.2"; it's more "this user needs version a.1 for both X and Y to work, while that system over there needs version a.2 for both X and Y to work".

I'm not really following your comment about distros requiring a single distro-wide version dependency. Maybe some do? but I haven't encountered that. My experience has been that if you upgrade in place over a period of years, you gradually accumulate multiple versions of support libraries because local and 3rd party apps don't necessary rebuild against the latest library versions in sync with the distro release schedule. That's minorly annoying, but so what? OK, you have to be careful not be overly enthusiastic about deleting old library versions, but you have them backed up, right? And for that matter the older version is probably still downloadable from the distro if you know what you are looking for.

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 1, 2022 23:43 UTC (Tue) by ssokolow (guest, #94568) [Link]

I'm not really following your comment about distros requiring a single distro-wide version dependency. Maybe some do? but I haven't encountered that. My experience has been that if you upgrade in place over a period of years, you gradually accumulate multiple versions of support libraries because local and 3rd party apps don't necessary rebuild against the latest library versions in sync with the distro release schedule. That's minorly annoying, but so what? OK, you have to be careful not be overly enthusiastic about deleting old library versions, but you have them backed up, right? And for that matter the older version is probably still downloadable from the distro if you know what you are looking for.

I think they're talking about how distro maintainers do things like maintaining their own patchsets to un-vendor libraries, even if that puts things into a configuration upstream does not approve of and refuses to support.

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 2, 2022 9:28 UTC (Wed) by farnz (subscriber, #17727) [Link] (4 responses)

Ah. In the case you're describing, you're building an unsupported configuration - it's a very short term workaround, since sooner or later, the packages users depend upon will require the newer pango's ABI and not work with the older pango's ABI. You're trading bugs - you've got users to choose an unsupported configuration (since the distro maintainers will tell you to update to latest releases first) that works for now, in preference to a supported configuration where bugs can be fixed.

Unless the distro explicitly supports this action, you're creating more trouble down the road - the user will at some point have to choose between a pango that works but doesn't display nicely on their display, or not running a current version of their application. In a statically linked world, they'd have to choose older application version that displays nicely, or newer one that doesn't - and that's a tradeoff the users understand - complete with "I won't upgrade from Debian 13 to Debian 14 because it breaks my Scribus" attitudes. What you've got users to build is a system where things are unreliable (because sometimes they will run things that accidentally or deliberately depend on the ABI of newer pangos), and where they cannot see their way clear to a fix (because ABIs aren't something the typical user understands - they just have two bits of information, "program X crashes" and "I can't upgrade pango").

The long term solution is to work with upstream - either deliberately fork pango, and have two alternatives you can install as a deliberate act, one with the old engine, one with the new, or get the new engine to the point where it's as good as the old on low-DPI displays - because anything else gives users the illusion that they're running a supported system, while actually having a system that'll break at a moment's notice.

And in terms of distros requiring a single version - the default in all distros I've looked at is to say that you can't install both pango-1.43 and pango-1.44 at the same time. You have to choose one or the other; eventually, if there's a pango-1.46 that adds new ABI that an application depends upon, users using your "fix" have to upgrade and accept the problem. Further, most distros explicitly don't support you running a mix of older and newer packages; if you want distro support for anything, you need to run the distro update mechanism and run the latest versions of all packages. In the case of users who've pinned an old version of pango, they're out of luck if they want distro support, because the distro attitude will be "well, you're running outdated code. You need to upgrade pango to fix it".

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 2, 2022 12:36 UTC (Wed) by Wol (subscriber, #4433) [Link]

> And in terms of distros requiring a single version - the default in all distros I've looked at is to say that you can't install both pango-1.43 and pango-1.44 at the same time. You have to choose one or the other; eventually, if there's a pango-1.46 that adds new ABI that an application depends upon, users using your "fix" have to upgrade and accept the problem.

Dunno whether it's the "default" or not (and I haven't had the need to use it myself), but gentoo's slot mechanism allegedly makes running multiple versions easy. Maybe other distros should have something similar?

Cheers,
Wol

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 2, 2022 12:54 UTC (Wed) by anselm (subscriber, #2796) [Link]

And in terms of distros requiring a single version - the default in all distros I've looked at is to say that you can't install both pango-1.43 and pango-1.44 at the same time.

Most distros, however, will let you install pango-1.x and pango-2.x at the same time. The general idea behind this is that pango-1.44 should support everything that pango-1.43 supported, plus perhaps some extra stuff, so that it doesn't matter if you swap pango-1.43 out for pango-1.44 because applications that have so far worked with pango-1.43 will just ignore the new extra stuff, and upgrading pango for applications that do require the new extra stuff in pango-1.44 shouldn't be a problem as far as other pango-using applications are concerned.

If stuff from pango-1.x actually goes away then the new pango should be 2.0, and applications that depend on pango-1.x should just be able to keep using the older version because the two versions can be installed side-by-side. How well the authors of individual software packages adhere to this philosophy is, of course, anyone's guess (and of course there's the edge case of an application relying on a bug in pango-1.43 and breaking when the bug is fixed in pango-1.44, which arguably is not the pango package's fault).

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 2, 2022 16:40 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (1 responses)

> And in terms of distros requiring a single version - the default in all distros I've looked at is to say that you can't install both pango-1.43 and pango-1.44 at the same time

You *could* if pango-1.43 and pango-1.44 where both packaged and maintained (and for a lot of components this is technically possible, just rename the pango-1.43 to oldpango-1.43 to allow two version streams in parallel).

However *no* *one* wants to expand the effort to package multiple versions of the same thing for you. The amount of work snowballs quickly.

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 2, 2022 17:15 UTC (Wed) by farnz (subscriber, #17727) [Link]

Indeed - and the ability to pin an older version than the distro supports is not a real fix for that very reason. Without someone stepping up to maintain a fork of pango-1.43 suited for use on systems that don't gain from the changes in pango-1.44, you're trading off a known bug (the rendering issues in pango-1.44 on "normal" screens) for future problems (unknown bugs in pango-1.43 fixed in a later pango-1.x release including security-relevant bugs, ABI changes in later pango-1.x that increase the blast radius of trying to pin 1.43 etc).

Ultimately, while this is nice for debugging whether or not it's a pango change that's breaking things, it's a dangerous route for fixing things - it results in an unsupportable system because you can't rely on users having the versions of libraries you expect them to have (because they're "randomly" pinning versions that happen to not have bugs that annoy them, and then complaining because there are bugs they don't like).

shared libraries [ no longer specific to either Rust or Fedora ]

Posted Nov 3, 2022 9:16 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

The full move to Harfbuzz is required to properly support modern opentype (TTF and OTF) features, be it latin legatures, non latin scripts, emojis, color, etc. Harfbuzz has two decades of enhancements, fixes and rewriting over the old freetype code it was originally forked from.

High resolution (aka banal 4K screens) is not directly involved, except inasmuch the very old pre-opentype fonts that render worse with the new system, mostly because they were designed around the old system bugs, are useless on such screens (way too small). And also, they look like crap compared to banal vector fonts rendered on such screens: there is a point at which no amount of pixel hand-tuning can compete with more pixels to render fine details. It’s no longer distorted sharp vs blurry fidelity, it’s sharp fidelity vs blocky pixelated that relies on old bugs.

Packaging Rust for Fedora

Posted Nov 9, 2022 9:52 UTC (Wed) by ras (subscriber, #33059) [Link] (4 responses)

> they can just rebuild static binaries anyway.

I'm not sure they could. Debian has some 23,000 packages depending on libc6. Rebuilding all of them and sending the result to the mirrors ain't going to happen quickly.

Packaging Rust for Fedora

Posted Nov 9, 2022 16:12 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

There are two issues in play:

1. It takes a lot of time to rebuild packages. This can't be helped, but computing is pretty cheap these days.

2. It takes a lot of bandwidth to distribute the updates. This is mostly because the current package update infrastructure is horribly wasteful. It forces users to re-download the whole package if only a handful of bytes are changed. That's why OSTree with delta-encoding would also help a lot for system packages. User-level apps should be sandboxed and can also use delta-updates.

Packaging Rust for Fedora

Posted Nov 9, 2022 21:13 UTC (Wed) by ras (subscriber, #33059) [Link] (1 responses)

> This is mostly because the current package update infrastructure is horribly wasteful. It forces users to re-download the whole package if only a handful of bytes are changed.

Here is an idea. Why don't we just split the common parts of every program binary into independent libraries. Pre-do the delta in other words. It wouldn't just save on distribution bandwidth, it would save on storage space too, and hell we could even share the RAM when it is executed. It would probably be more efficient bandwidth wise than doing the delta later, because while we have to update all of libc6 - we only have to do it once, not 26,000 times.

It would probably give us huge saving on time and effort too. Imagine - lets say the library foo has a CVE. Instead of finding every program that uses library "foo" and backporting the CVE fix to each and every copy of the "foo" source those programs included (which would probably be a different version because programmers seem to just love their .lock files, pinning the version), we would have one instance of "foo" (because it is a now a shared library) and fix just that. To take the libc6 example, instead of fixing 26,000 instances of libc6 source we just fix it once! It would take a bit of upfront effort to ensure every program worked with this one version of "foo", but I bet that effort would be more than repaid by the time saved maintaining it later.

In fact the productivity gain would be so big, it would make things like Linux distributions possible. Without it I doubt they would be workable.

Packaging Rust for Fedora

Posted Nov 9, 2022 21:20 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

> It would probably give us huge saving on time and effort too.

It unfortunately doesn't scale. People have to make sure that multiple applications actually do work with the single version of the library. We tried that, and it failed. Never mind that this approach still doesn't cover everything. E.g. templates in C++, macros and header changes in C, etc.

Thus simply treating updates as blobs, and computing diffs automatically will help to free humans to do actually useful work.

Packaging Rust for Fedora

Posted Nov 11, 2022 12:16 UTC (Fri) by jezuch (subscriber, #52988) [Link]

> Rebuilding all of them and sending the result to the mirrors ain't going to happen quickly.

And yet, I'm pretty sure Debian does it regularly and pretty routinely. Though maybe not with glibc (it's a small nightmare of its own).

Packaging Rust for Fedora

Posted Nov 1, 2022 18:32 UTC (Tue) by salimma (subscriber, #34460) [Link]

Hopefully better automation can address some of the pain points; our company hired an intern this summer to work on a tool to address some of the issues brought up:

https://lists.fedoraproject.org/archives/list/devel@lists...

The tool:
- recursively checking out packages
- updating existing packages while carrying over manual changes
- creating compatibility packages when upgrading packages with dependents
- automate parallel COPR test builds of sets of packages in dependency order
- chain-build all packages in Koji with requested side tag
- merging and chain-building all packages across release branches
- help review a rust update in Bodhi and verify no compatibility issues are introduced

It does not automatically commit anything, of course, and right now is a
bit opinionated in favor of picking the lowest satisfactory version
possible, which is probably not what we want long term -- but it
hopefully provides a nice foundation on which to build on

Packaging Rust for Fedora

Posted Nov 2, 2022 19:34 UTC (Wed) by bluca (subscriber, #118303) [Link]

Language developers: "We don't need no stinking distributions! Curation is useless! Our language-specific stores are the best!"

Language-specific stores: https://lwn.net/Articles/913555/

Shared Libraries Reduce RAM Usage

Posted Nov 11, 2022 19:09 UTC (Fri) by XERC (guest, #14626) [Link]

I just wanted to point out that one of the main benefits of shared libraries is that the same library instance in RAM can be used by multiple processes. May be I'm mistaken here, but that's my current, 2022_11, understanding. From speed point of view, the less RAM is needed, the greater the probability that what the CPU needs, is already at some CPU cache. With modern CPU caches that are multiple MiB in size, that "something" might be a whole small shared library or some part of it. Form speed point of view there's quite a bottle neck between CPU and RAM.

Thank You for reading my comment.