Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Posted Dec 5, 2019 19:21 UTC (Thu) by farnz (subscriber, #17727)In reply to: Soller: Real hardware breakthroughs, and focusing on rustc by nim-nim
Parent article: Soller: Real hardware breakthroughs, and focusing on rustc
The deep problem to face down in Rust is the dependency on monomorphization of generics. This is a deep and difficult problem that nobody has a solution to yet, and arguing for dynamic linking as the solution just reveals ignorance of the problem.
Monomorphization is a really good tool for developers - you write the code once, with generic parameters that have to be filled to make the code useful, and then the compiler duplicates the code for each set of filled in parameters; the compiler's optimizer then kicks in and reduces the set of duplicated functions down to the minimum amount of duplicated machine code given the parameters you've supplied. This is great for correctness - I've only got one version of my code to debug - and for performance - no indirect calls via a vtable, just optimal machine code. The trouble is that when you expose a generic as API, which is a common thing to want to do (e.g. Rust's Vec type is a generic), the result of compiling just your library is not machine code that could go in a dynamic library, but some form of intermediate representation that needs compiling further before you have machine code. However, for obvious reasons, the dynamic linker does not and should not include a full LTO phase.
C++ avoids this by saying that if you have generics in your interface, the only way to use your code is as a header-only library. Rust builds to "rlibs", which are a mix of IR where the optimizer needs more information, and machine code where it's possible to generate that; these are effectively static only, because of the need to have an optimizing compiler around to turn the IR into machine code.
There are people on the Rust side trying to find a solution to this, but it's a deeply difficult problem. At the same time, people who work on Cargo (Rust's build system) are taking lessons from distro people on how best to manage packages given that you need static linking - e.g. "cargo-crev" helps you limit yourself to a trusted base of libraries, not just any random code, and makes it easier to audit what's going into your compile. Cargo makes it impossible (by design) to depend on two semver-compatible versions of the same library; you must pick one, or use semver-incompatible versions (at which point you can't interact between the two incompatible libraries - this permits implementation details of a library you use to depend on incompatible but supported versions of the same thing, for example). Cargo workspaces unify dependency resolution for a set of Rust code into one lump, so that I run my top-level "cargo build", and get warned up-front if I'm working with incompatible libraries in different parts of the same workspace, just as apt warns you if you depend on two incompatible dpkgs. Cargo permits you to yank broken version from the central index, which warns people that they can't use them any more (e.g. because this version is insecure and you must upgrade). And there's work being done to make it possible to run your own partial mirror of the package indexes, limited to the packages that you have vetted already - so you can't even build against something old, because it's not in your own mirror.
This contrasts to Go, where it's common to specify the library you require as a VCS tag from a public VCS hosting service like GitHub. Cargo makes that possible if you really, really want to do it, but it's not easy to do - the goal is very much focused on being able to build up a set of trusted "vendored" packages in your workspace, and then build your code only against the trusted vendored packages, offline, in a reproducable fashion, even if the rest of the world becomes inaccessible.
Posted Dec 5, 2019 21:02 UTC (Thu)
by nim-nim (subscriber, #34454)
[Link] (35 responses)
From a distributor point of view the only things missing to make the build part manageable is an easy way to point builds to a system-wide workspace, and a way to populate the index of this workspace in a granular way. Basically
The new Go module system is a lot less elaborate than your description. However, it will practically allow achieving 1-4 (not always by deliberate upstream design, but the end result will be the same).
I agree that the Rust community has generally nicer and saner behaviors than some I’ve seen Go side. However, natural selection will keep only fit projects alive in both communities. As long as the core tooling enables good practices, I’ll expect the kind of convergent evolution we’ve seen between rpm and apt (for example).
On the deploy front, however, without shared libs, a large number of applications, with correct system-wide build dependency management, probably implies frequent cascading rebuilds, with the associated stress mirror, network, disk and user side. Even more so if the dev community lacks discipline and continually churns code that does not work with older semvers.
We’re not seeing this effect yet because first, there are not so many first class applications written in those new languages, and second, there are no effective system-wide build dependency management systems yet (so we’re missing a large number of rebuilds that would probably be desirable from a security or bugfix perspective)
Users are deeply suspicious of mass updates. The software world as a whole failed them all too often. It’s a lot easier to make them accept a single lib update, than replacing large sets of apps that happen to have the same code built-in (and, I’m talking about technophile users here, not the kind of user targeted by RHEL).
Posted Dec 5, 2019 21:36 UTC (Thu)
by nim-nim (subscriber, #34454)
[Link]
The critical part is managing a system-wide shared layer, either at run (shared libs) or build time.
You can buy new network links and new hardware. Without a system-wide shared layer, you need a lot more human brain-cells, and that’s a lot harder to procure, especially in a community context.
Posted Dec 6, 2019 12:00 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (33 responses)
All of your asks have been possible with Cargo for some years now.
Given that distros still aren't happy with the Rust story, I'm guessing that you've missed a requirement or two that matters to distros here - it would be helpful to know what's missing so that it can be added to Cargo.
Posted Dec 6, 2019 16:30 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (32 responses)
Nevertheless, it is probably automate-able (with lots of pain). But, it is useless, if the crates themselves are constantly churning and invalidating the system workspace.
If the system workspace is too hard to set up and use, no one will try to apply a code baseline, and creating a baseline from scratch from projects that didn’t try to converge on a baseline is hard.
Posted Dec 6, 2019 16:39 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (4 responses)
If all you want is what C/C++ and Golang give you, just set up a workspace, and build in there. Most of the steps I've described only exist because you wanted to not depend on what was present on disk, but wanted to identify the missing packages and build them; C/C++ don't even have dependency tracking for you to copy form.
If all you want is what Golang provides, create a file Cargo.toml in your buildroot, containing a workspace section:
Then, unpack your crate into a subdirectory of your buildroot, and cargo build --offline will do the needful using the system crates.
Posted Dec 6, 2019 16:50 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (3 responses)
However, I suppose that what makes my distributions friends working on rust most unhappy, is the constant dev churn, because devs do not know or use the default system crate locations, and just push change over change without trying to reuse stabilized crate versions.
Posted Dec 6, 2019 17:01 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (2 responses)
In practice, you'd do this setup once, and not repeat it - unlike an env var, it's then persistent forever. There's no need to version the paths of crates; crates include metadata that identifies their version, and you can rename them to include versioning without ill effect in your system crate pile.
And the dev churn really isn't that bad in Rust - not least because Cargo does semver aware versioning to begin with, so most crates ask for things by semver-aware version.
Posted Dec 6, 2019 20:05 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (1 responses)
That’s a dev POW. A dev sets up its environment, it then never changes.
To get distribution synchronization, you need an environment shared by every dev, every packager, and the buildsystem. That means either an hardcoded default location, or a setting in a default system config file, or at least something dirt easy to set, like an environment variable.
Not relying on everyone setting manually the same default in each and every build env.
> There's no need to version the paths of crates;
Unless there will never be two rust apps that need the different versions of the same crate to build, you need to version paths because otherwise the paths will then collide at the share distribution level.
And you can say
> you can rename them to include versioning without ill effect in your system crate pile.
Yes we can do a lot of things at the system level. At some point people tire of working with stuff that need massaging before being used.
> most crates ask for things by semver-aware version
If crates always ask for the latest semver available, because devs have to pull things from the internet (there is no shared system crate store) so why not pull the latest one while you're at it, you *will* get a race to the next version and constant churn.
Posted Dec 6, 2019 20:14 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
Crates do not ask for the latest semver available, because Cargo locks down crate versions on first build of a binary target. So, once I've specified (say) lazy-static = "^1", the first time I do a build with that dependency, Cargo will record the version of lazy-static it found, and use that for subsequent builds; I have to explicitly demand an update from Cargo before it will move onto a new version.
If I'm working in a workspace, then Cargo will resolve deps workspace-wide, and lock me down workspace-wide; this is the recommended way to develop Rust, as it means that you don't bump versions of crates you depend upon, unless you explicitly ask Rust to do that.
And I would note that this is no different to my experiences as a C developer working on the kernel and graphics drivers - when doing that, I used the kernel, libdrm, Mesa3D, intel-gpu-tools and the entire X.org stack from git to avoid duplicating work, and ensure that if I *did* have a fix for a problem, it wouldn't conflict with upstream work. The only difference in Rust is that, right now, there's a lot of churn from Rust being a young language.
Posted Dec 6, 2019 16:41 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (26 responses)
and boom, the systems knows about it as soon as the fontconfig index is rebuilt (and you can force the rebuilt with fc-cache).
That’s how hard it should be to register a crate in the system workspace. Drop in standard place. Optionally, run a reindex command.
And using this store should be either the default behaviour or just a shell variable.
Anything more complex than that will require more motivation to use. Humans are not good at using inconvenient things by default.
Posted Dec 6, 2019 16:44 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (25 responses)
Hang on - you asked how to translate the deps chain from Cargo to the distro packaging system, and now you're talking about how you bypass the distro packaging system.
This is moving the goalposts - an equivalent is given that I need the font Kings Caslon, how do I find and install the Debian package for it? It's a *lot* harder than just copying the file into place - I have to somehow translate the font name "Kings Caslon" to a Debian package name, download and unpack *that* package, and *then* I can drop it into place.
If all you want is to use a system package source, just put the package sources in a known location, and (as per my other comment) set up a workspace to build your new code in. Job's done - all of the rest of the work is about setting up that system package source to begin with.
Posted Dec 6, 2019 18:42 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (24 responses)
# dnf install font(kingscaslon)
Sorry about that:) It is good that we have this conversation.
What you feel is difficult, is not difficult at all distribution-side. Distributions had to invent names for all the things they ship, to make them exist in their index. So defining naming conventions, is part of the distribution bread and butter.
In my example the distribution did not even invent a mapping, the mapping was defined upstream, it’s the output of
$ fc-query --format '%{=pkgkit}' <fontfile>
I see we come from very different perspectives, and what is evident to me is not to you (and vice versa). Therefore, I will expand quite a bit.
Distributions want to make things available to the rest of the distribution, and to their users.
For rust, those things necessarily include the crates rust software was built from, because rust does not use dynamic libs. Therefore base-lining at the source code level is imposed by the language. Base-lining is a requirement to coordinate the work of all the distribution contributors.
Things that distributions can handle best:
1. exist under a standard system-wide root (/usr/share/fonts, /usr/lib, FHS and XDG are distribution keystones)
2. have a standard filename or path under this root, that avoids collisions between different artefacts and versions of the same artefact
3. are discovered and used by whatever needs them, as soon as they are dropped under the default root, with the default filename (eventually after reindexing)
4. have standard artefact ids and versions, and standard version conventions that map to something similar to semver
And then, for convenience (but convenience matters a lot to over-committed distribution contributors):
1. you have a command that takes things to deploy anywhere on the filesystem, and deploys them to their standard root (with $PREFIX handling pretty please, many build systems use fakeroots not advanced filesystem overlays)
2. you have a command that reindexes the whole root, if the language needs it (fc-cache, ldconfig)
4. you have a command, that outputs what a new artefact needs in various contexts (for building, for running, for testing, etc). Need = artefact ID + minimal semver, not artefact ID + locked version.
All of this can eventually be recreated and redefined by distributions, if not provided upstream. However, the more work it is, the more likely distribution people will focus on mature languages, for which this work has been done a long time ago (ie C/C++).
A big difference with IDE and dev tools, is that distributions emphatically do *not* want to frob structured formats to do things. Just give use standard commands. We don’t want to do xml for language A, yaml for language B, json for language C, toml for language D, ini for F, etc. Format wars are dev fun, not integrator fun. Everything is a file under a standard place with a bunch of standard manipulation commands and eventually a set of standard shell variables is plenty enough for us.
When all of this exist, packaging a new upstream artefact is just:
1. take the upstream bunch of files
2. execute the command that tells what other artefacts and artefact versions they need for building (and optionally, testing)
3. translate the artefact ids into distro ids
4. use semver to map the minimal versions asked by the artefacts into whatever is available in the distribution baseline
5. install the result (make it exist in the standard root, reindex the result)
6. run the standard build command
7. run the standard install command (with prefix-ing). In the rust case, that would include the command that deploys the crate under the default root, for reuse
8 run the standard test command
9. run the standard "what are you" command on the resulting files, so the corresponding ids and versions can be added to the distribution index
Forcing a version is possible in 4, as a manual exception in the build process, but it’s not the default behaviour, because doing otherwise is just piling technical debt.
And that’s basically all tooling side. Once this part is done, there is no difficulty in packaging large numbers of artefacts, as long as their code and dependencies are sane, and the churn is not incompatible with the target cadencing
Posted Dec 6, 2019 19:02 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (22 responses)
Conventions and defaults create network effects. Humans strive on habits and routines.
The difference between “plop file here and it will be used” and “just edit this file in <my pet> format, for <every> artefact, and set <whatever shared root you want to use” is technically slight but huge from a human networking point of view.
Posted Dec 6, 2019 19:14 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (21 responses)
The thing is that I'm a Fedora packager - none of what you're explaining to me is news to me, and as someone who *also* develops in Rust, I don't see the difficulties in mapping Rust to Fedora packages (although I'm aware that they exist, they are largely a consequence of Fedora fighting hard to dynamically link instead of statically link, which, due to the monomorphization problem, is a hard problem for Rust).
Can you please stop assuming that I'm clueless, and start focusing on what, exactly, it is about Rust and Cargo that makes it hard for distros to package Rust code, beyond the fact that Rust crates are hard to dynamically link using a linker designed around the needs of 1980s C code?
Posted Dec 6, 2019 19:48 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (20 responses)
Fedora is fighting hard to overcome the churn Rust side.
Some of the most awful and broken things in modularity, were constructed specifically for Rust, not to dynamically link, but to cope with the terrible amount of churn the SIG had to face.
And why do you get this churn in Rust? Because Rust has no notion of a shared code state layer at the system level. There is no baseline to build upon, every dev just bumps its code needs all the time.
The baseline could be constructed with shared system libs (that rust does not have) or with a default shared workspace at the system level (that rust does not have either). Or, maybe someone will have another idea.
Fact being, this baseline does not exist now. Without something to synchronize upon, people do not synchronize. Without synchronization, it’s a race to the latest version bump. It’s no more complex than that.
Posted Dec 6, 2019 20:03 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (19 responses)
Rust does have the notion of a shared state - it's part of the workspace model. And the only reason the churn was an issue for the SIG (I followed this, BTW) is that it becomes impossible to maintain shared library packages for crates when they are being developed at high speed, and you don't have tooling to use Cargo to keep up with the churn.
In other words, this isn't a technical issue - it's that Rust is currently a young language, and developers are releasing early and releasing often. The only reason Fedora doesn't have a churn problem with C++ or C code is that the libraries in use in C and C++ have been around for a long time, and are consequentially relatively stable, as compared to Rust libraries, and thus Fedora being 12 months behind current doesn't actually make much difference.
If Fedora wanted to, it could treat the churn in Rust the way it does in C and C++ code - stick to relatively stale versions, and hope that the features users want are present. The trouble with that from Fedora's point of view is that with the current pace of Rust application development, Fedora would look bad because of all the missing features that upstream advertises, but Fedora does not support.
Posted Dec 6, 2019 20:21 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (11 responses)
That would not work for Rust.
That works for C/C++, because the distribution versions choices are materialized in dynamic libraries. It’s easy for upstreams to check what library versions are used by major distros. It’s easy to make sure they have something that works with those versions, if they want to reach users. Also that pushes upstreams to agree on the next version they’ll consolidate upon because they know distros won’t ship every possible variation under the Sun.
Yes, I know, terrible dev oppression with stale versions.
In the meanwhile, the stuff produced by oppressed devs gets distributed, and the stuff produced by high speed rust devs, does not.
A formula one can be reinvented for every race, and stop at pit stands every lap (high speed car fixing). A normal car better run a long time without changes. Changes in normal cars better be batched R&d side for next year’s model.
Posted Dec 6, 2019 23:36 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (10 responses)
I disagree fundamentally with you, then; the only reason I ever use my distros -devel packages for work C and C++ code is because it's usually a right pain in the backside to go to upstream and build what I want for myself. Cargo makes it easy for me to build my upstream Rust dependencies for myself, so why wouldn't I do that?
This leads to different incentives - for C and C++ libraries, I want to make life easy for distros, not because distros are necessarily good, but because they are considerably easier to deal with than either vendoring all my dependencies or getting my potential users to build them in a way that I can use. In contrast, over in Rust land, I can tell my users to run cargo install $cratename, and my crate is automatically built and installed for them.
So, from where I'm sitting as a dev, the distros fill a gap in the C and C++ ecosystem that's filled by Cargo in the Rust ecosystem. All the distros need to provide is C and C++ packages, plus rustup, and I can get users to install my code without further involvement from the distro. Remember the trifecta: users, operations, developers; because C and C++ have (quite frankly) awful tooling for non-developer installation of code, there's room for distros to get in quite a lot of operations stuff while acting as a link between developers and users, because if distros refuse to provide that link, the users can't get the developers' code to work.
In contrast, because Cargo provides most of what a package manager provides to a user (install, upgrade, uninstall, dependency resolution, rdep handling), if distros try too hard to get in the way, I can bypass them relatively easily. You thus have a problem - the operations leverage over developers in the C world is that without the distro compromise that operations will accept, users can't run developer code. This isn't true in the Rust world - Cargo is good enough as a package manager that I can bypass operations.
So, distros no longer have leverage; how do operations get heard in this world, given that if you push back against good development practices like "releases early, release often", users will bypass distros to get the tooling they want? Remember that Linux is already a minority OS - if you're saying that you're going to hold it back further, users will just bypass distros if it's easy to do so, as they're the people who, today, are already motivated to find solutions to problems beyond following the pack.
Posted Dec 6, 2019 23:56 UTC (Fri)
by pizza (subscriber, #46)
[Link] (5 responses)
Not according to Microsoft -- As of July 2019, Windows Server is now the minority on Azure [1], and that's the cloud provider with the lowest Linux percentage.
[1] https://www.zdnet.com/article/microsoft-developer-reveals...
Posted Dec 6, 2019 23:58 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (3 responses)
That's server-side, not total - and server side is exactly the place where you have developers who can run tools like Cargo instead of being helplessly dependent on distros. In other words, that's where distros are least useful to begin with, beyond the base libraries to get your developers going.
Posted Dec 7, 2019 0:43 UTC (Sat)
by pizza (subscriber, #46)
[Link] (2 responses)
I have no idea what point you're trying to make, beyond "distros are useless, because reasons"
Posted Dec 7, 2019 11:42 UTC (Sat)
by farnz (subscriber, #17727)
[Link] (1 responses)
I'm saying that Linux distros are not a significant driver for distribution of code; server-side, you do whatever the devs want you to, client side is iOS, Android, Windows etc.
This, in turn, pulls down their influence - why should the authors of Krita, or someone writing code to run on Amazon AWS or Google Cloud Engine, care if their users have to run "cargo install", or "yarn install", instead of "apt install"? Unlike C++ land, where dependency management is otherwise manual, modern languages don't need distribution packaging to be easy to install and use code written in those languages - and that means that distros no longer hold the place of power they do in C++ land.
Posted Dec 8, 2019 21:47 UTC (Sun)
by flussence (guest, #85566)
[Link]
Posted Dec 7, 2019 0:37 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Dec 7, 2019 10:19 UTC (Sat)
by nim-nim (subscriber, #34454)
[Link] (3 responses)
And, for Rust, you do not want to make it easy on distributions.
You’re pushing complexity on distributions.
You’re pushing complexity on users (cargo install foo is nice for a limited set of fringe components, do you actually expect users to construct the thousands of components composing a system like that? That requires big company-level of hand-holding).
And, you’re surprised distributions feel C and C++ libraries are better suited for systems works (what distributions do)? And do not want Rust anywhere near their core in its current state?
Really, what did you expect?
A system that does not incentivize, making the life of others easier, will result in little adoption by those others.
And you can feel distributions are “not necessarily good”, but that’s irrelevant, unless you want to pick up the distribution work yourself. That won’t leave you with much time for dev, but such is real life, removing someone that provides a service to you, does not make the need for the service go away.
Posted Dec 7, 2019 11:49 UTC (Sat)
by farnz (subscriber, #17727)
[Link]
You're missing the point - code is written by developers. If developers move en-masse to a new language, distros don't get a choice about wanting C instead of Rust - they have to live with it, because the users of code want the new features of the new versions of code.
In C land, distros have power because there's no dependency management without them. In Rust land, if distros get between users and devs, they're trivial to bypass. We've already seen what happens with C code where the developers and users agree that it's beneficial to move to the new shiny - distros have to move, too, whether they like it or not, because users want the new GNOME etc, and thus distros have to accept GNOME's dependencies.
Stop thinking that distros matter to users - they don't, particularly, and there's a reason that no other OS has a distribution-like model. Users care about applications that make their computers do useful things; if distributions make it harder to get useful things than bypassing distributions, then users will bypass distributions.
If the distributions can't work out how to work with Rust developers, and system software migrates to Rust, then distributions will get reduced to sufficient tooling to run rustup and Cargo, because that's all that users will want from them; if distributions don't want to end up here, then they need to work with Rust now to get to a state where they are still relevant.
And note that Rust does make it as easy as it can on distributions - Cargo even translates its own dependency information to a simple JSON form for you, with or without resolving versions at your discretion. If distributions want Rust to do more than it does for them, then they need to engage with the Rust community to explain what they need and why.
The discussion with nim-nim elsewhere in this thread is typical, FWIW - as they asked for things from Rust, I explained how to get what they want from Cargo, and then they went off into unrelated territory. My favourite part of that was their claim that I just need to run dnf install font(kingscaslon), and dnf would package my font for me, then install it…
Posted Dec 7, 2019 13:11 UTC (Sat)
by pizza (subscriber, #46)
[Link] (1 responses)
This is the bit that I keep coming back to. _someone_ has to do this work, and if it's not the distros [1], and not the language/tool vendors, then it's going to have to be you [2]. It's legit to say that the traditional distro model has its flaws and room for improvement, but one would have to be pretty naive to claim the sorts of problems and concerns distros have traditionally solved (ie "testing, deploying and maintaining configuration managed baseline systems useful for 3rd parties to develop on/for") somehow no longer matter.
[1] Not just distros ala Ubuntu, but whatever k8s and various proprietary cloud vendors come up with to fill the same role.
[2] "you" can mean an individual developer or an tools/platform team that big orgs tend to have. With every one of them reinventing the same wheel. If only there were independent organizations that could take on this work for the benefit of all involved... I know, we'll call them "distributions"...
Posted Dec 7, 2019 14:12 UTC (Sat)
by farnz (subscriber, #17727)
[Link]
And therein lies the rub - part of the problem here is that distros are saying that someone else should do that work - and it looks like, at least for Rust, the language/tool vendors are going to do it well enough for users' and developers' needs, and simply bypass the distros altogether, with things like crater for testing the entire package ecosystem in a single build.
Note that this has already happened for JavaScript - if that's your language of choice, you get Node.js from the distro, and then switch to NPM or yarn from there on in; users are happy to run bundles created by developer tooling, and the distro becomes an irrelevance as soon as it's good enough to get Node.js to run. Maybe that's a good end point for distros - enough runutime etc to let the language tooling take over?
Posted Dec 6, 2019 20:32 UTC (Fri)
by nim-nim (subscriber, #34454)
[Link] (6 responses)
So, if the community wanted it, the technical aspects could be changed.
However, it is too easy to attribute it to the language age. Java never outgrew the churn state, and few would call the language young today. Early choices, made when a community is young, and its members few, easily set in hard habits later.
Posted Dec 7, 2019 4:18 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link] (5 responses)
Then distros' jobs would be to say "hey, there's a security fix in x.y.z, please update your minimum to x.y.z+1 and make a new release" and every stable-tracking distro gets a similar fix available. Of course, if the distros provide an index that cargo can read, just not providing the minimum and giving a minimum-with-fixes available (treating the intermediate versions as "yanked") will likely have the same effect. Investigation into that as a viable alternative would be necessary.
Posted Dec 7, 2019 8:46 UTC (Sat)
by nim-nim (subscriber, #34454)
[Link] (4 responses)
That’s the thundering herd effect that makes it unpractical, to package large amounts of software, if upstream devs do not coordinate their version needs a minimum (and they can coordinate via shared libs, unified master company scm like Google, anything else but the coordination is necessary). semver alone does not work if everyone keeps requiring semver tip versions.
You can delay the effect with containers or static linking, but as soon as a critical fix is needed, it comes back with a vengeance. The dev dream of “any dev uses whatever versions it wants and the rest of the supply chain will cope” is utopic in presence of un-perfect code, that will always require an eventual fix.
> Of course, if the distros provide an index that cargo can read
To find out what is available on my current distribution:
$ dnf repoquery -q --provides $(dnf repoquery --whatprovides 'crate()') | grep 'crate('
crate(abomonation) = 0.7.3
To install one of those
$ sudo dnf install 'crate(abomonation) = 0.7.3'
That could probably be streamlined and plugged into cargo if someone wanted to (it would be much better if cargo provided a standard name and version mapping for rpm and apt at least, that would simplify reading it back cargo-side; the artefact naming is pretty flexible distribution side but the version format less so, as it is used to compute upgrade paths. The original semver format, before non Linux people added rpm and deb-incompatible things in it, was easy to map).
And then the result of the install is useless for the rust dev if there is no system workspace enabled by default. The crates will be installed but not used.
> Of course, if the distros provide an index that cargo can read, just not providing the minimum and giving a minimum-with-fixes available (treating the intermediate versions as "yanked") will likely have the same effect.
Yes, that’s the most likely scenario distro-side. There is a limited set of checked and available versions, anything else is considered yanked. Doing things any other way means drowning under version combinations.
Posted Dec 7, 2019 11:52 UTC (Sat)
by farnz (subscriber, #17727)
[Link]
Every one of those crates is using the Cargo name and versioning - there is a trivial mapping from Cargo to dnf notation, and back again. The trouble is that the distro has not written the code to do that mapping for me, but is expecting that Cargo will magically stop using its own dependency resolution and handling (which is at least as good as dnf's) and start using the distro's code, just because that's what the distro wants.
Posted Dec 7, 2019 13:33 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
If OpenSSL 1.0.2j has some critical fix, why would I want to allow my software to compile against 1.0.2i anymore? Bumping the minimum helps *everyone* get better versions over time, not just those who remember to rebuild their OpenSSL.
The current flurry of changes needed for minvers support is because it is new in Cargo. If it had been there from the beginning, it likely would have been much easier to handle. But, this is the same problem with any bootstrapped initiative be it reproducible builds, new architectures, etc. The initial push is large, but the constant churn should be quite low.
Posted Dec 7, 2019 14:09 UTC (Sat)
by smurf (subscriber, #17840)
[Link] (1 responses)
All of this can happen automagically, given modern tooling (which Debian for one is slowly moving towards – much too damn slowly for my taste) … except for the "fix the inevitable regressions" part of course.
Posted Dec 8, 2019 11:33 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link]
I'm arguing that Debian shouldn't make its own house fine and not let upstream know that there's an issue with one of their deps.
Posted Dec 6, 2019 19:10 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
Yet again, I am completely lost in what you're suggesting - you keep jumping around between what has to be done to package something (which is the core of the discussion), and what can be done when you *have* packaged something.
I have an unpackaged TTF file for Kings Caslon (it's a proprietary font, so not in any distro). When I run dnf install font(kingscaslon) on Fedora 31, it does not find my TTF, package it, and install it in the right place - how do I make that command work for me, since you claim it solves the problem of packaging a font?
Rust has already solved the naming problem to Rust's satisfaction - in practice, you use the name any given crate has on the primary crate index for the community, and you're done. For Rust's needs, because Cargo does semver-aware version resolution, and tries to minimise the number of crates it pulls in subject to dependency resolution, this also keeps the dependency tree under control.
It sounds, from what you saying, like distros can't handle C libraries - they don't meet 2 or 4. Rust meets all four of your requirements, and the only bit missing is the command to translate from Rust's crates.io names to distro names, and use that for dep resolution instead of asking Cargo to download from crates.io.
Posted Dec 9, 2019 1:22 UTC (Mon)
by kvaml (guest, #61841)
[Link]
Soller: Real hardware breakthroughs, and focusing on rustc
1. the build system asks cargo the lists of crates+minimum semver needed for a build,
2 the answer is translated into distribution package dependencies,
3 the build system installs those dependencies,
4. each dependency installs the corresponding crate code + the system workspace index part that tells cargo to use this crate and nothing else
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
[workspace]
members = [
"/path/to/system/crates/*",
"*"
]
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
/usr/share/fonts
~/.local/share/fonts/
or XDG_DATA_HOME/fonts if you like non-standard layouts
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
3. you have a command, that can output the artefact ids and versions, corresponding to a filename (fc-query)
(mostly, adding a namepace like font(), if upstream naming forgets it will be injected in a dependency graph, that includes many different kinds or artefacts)
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
crate(abomonation/default) = 0.7.3
crate(actix) = 0.8.3
crate(actix-codec) = 0.1.2
crate(actix-codec/default) = 0.1.2
crate(actix-connect) = 0.2.3
crate(actix-connect/default) = 0.2.3
crate(actix-connect/http) = 0.2.3
crate(actix-connect/openssl) = 0.2.3
crate(actix-connect/rust-tls) = 0.2.3
…
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc
Soller: Real hardware breakthroughs, and focusing on rustc