LWN: Comments on "Gentoo in binary form"

Incentive to switch from Arch to Gentoo

Wol — Sun, 30 Jun 2024 20:59:49 +0000

> The language that's as far down the dependency tree as practically possible. That would be C. The same language that Python is written in. Not because C is great, but because it is easy to break Python on the system, esp. in the py2->py3 transition era; e.g. you can't just uninstall and install Python freely like any other package due to this dependency. A broken Python would not be a big deal if it would not also break the tool that is needed to fix the issue.

While I agree that Python is too easily broken, I really don't think using a language that is crap at string handling is the best language to use for something that is almost pure string handling ...

Using some form of lisp would be best, but too many people don't like / can't handle lisp. Even BASIC would be better than C.

Yes pick a language as low down the dependency as reasonably possible. But that's not an excuse to pick a language that is simply crap at the job.

Cheers,
Wol

Incentive to switch from Arch to Gentoo

acolin — Sun, 30 Jun 2024 16:55:25 +0000

The language that's as far down the dependency tree as practically possible. That would be C. The same language that Python is written in. Not because C is great, but because it is easy to break Python on the system, esp. in the py2->py3 transition era; e.g. you can't just uninstall and install Python freely like any other package due to this dependency. A broken Python would not be a big deal if it would not also break the tool that is needed to fix the issue.

Also the list of dependencies to fetch and build for bootstrapping a system (stage2) expands with Python that you might not otherwise need at all -- relevant to Gentoo Prefix (which I was using). Rust would suffer from same disadvantage -- having to build the Rust compiler during bootstrap is not appealing.

A shell language would solve this dependency problem just as well, since shell is a hard system dependency no matter what. But presumably it's not practical because of complexity of a package manager and performance required for the dependency resolution computation that is needed on Gentoo in particular (use flags constraints).

This performance aspect is another disadvantage of choosing Python -- optimizing the package manager has a good return on investment, since it is used over and over.

And someone already did all the work, apparently.

Incentive to switch from Arch to Gentoo

Wol — Sun, 30 Jun 2024 07:25:18 +0000

> The only wart was that Portage, the system package manager, depends on Python.. Apparently a native dependency-free implementation exists but remains on the side due to some politics.

What language would you suggest? Perl? Bash?

Some sort of scripting language is surely a necessity, and from my experience with lilypond I wouldn't recommend C++/Scheme. Or you could do it in emacs?

For all its (many) faults, I would think Python is a very good language for writing Portage in.

Cheers,
Wol

Incentive to switch from Arch to Gentoo

acolin — Sun, 30 Jun 2024 05:47:13 +0000

This might be the decisive incentive to make the move from Arch to Gentoo. Until this announcement Arch was more practical than Gentoo. Binary Gentoo, though, might be equally as practical as Arch.

Gentoo have been providing some binary packages, like Firefox, for a long time, but most had to be build from source, which is inconvenient when in the middle of working on something and needing to install a package (even if machine is fast). However, later when more time is available, an easy and first-class capability to build a customized and optimized version from source is an advantage.

In the recent past I've used Gentoo to create a reproducible scientific computing "environment" on a cluster, for tight control over which version of each library to build and how, with the resulting build recipes being commitable and simple to manage. Plus, a starting point (if not a perfectly working) build recipe already existed in the Gentoo repos for nearly all software. The experience was smooth. The only wart was that Portage, the system package manager, depends on Python.. Apparently a native dependency-free implementation exists but remains on the side due to some politics.

Gentoo in binary form

aaronmdjones — Sun, 31 Dec 2023 12:26:39 +0000

> At the time this was not supported and we ran into crazy things like not being able to upgrade the tar package because the binary installer would attempt to overwrite its own binary during the unpacking process.

This is what install(1) is for. Never attempt to overwrite any binaries in-place; if they're running, the kernel simply won't let you.

The way install(1) works is that it unlinks the target ("deleting" it) -- even if it is running, this is allowed; the filesystem will merely hang onto the data blocks (not allowing them to be reused by any other file allocations) until they are not being used anymore -- and then it creates a new file with the same name. The contents of this file will be backed by different data blocks, allowing you to replace the file without overwriting it.

Specifically, the way Gentoo works these days (whether you're using a binary package or not), is that it first "installs" the package files to /var/tmp/portage/package-category/package-name/image/, and then it will sync this over to / (or whatever you set $ROOT to).

Gentoo in binary form

Wol — Sat, 30 Dec 2023 19:23:00 +0000

> We ditched Gentoo for Debian later just because it was so much easier to manage. But it sure was an interesting time. (It did kinda of put me off Gentoo for life though.)

In a sense this is what ATTRACTED me to Gentoo, though. That for the most part, it works well, but you NEED a pretty good understanding of what's going on to make it behave ... I viewed it as a good learming experience.

Cheers,
Wol

Gentoo in binary form

kleptog — Sat, 30 Dec 2023 14:00:18 +0000

This brings back memories using binary Gentoo packages to maintain a fleet of over 100 servers in the field somewhere around 2010 (I inherited the setup). We'd have one buildbot that built the packages and then distributed the binary packages to the servers for installation. That way you didn't need to have a complete build environment to build everything on each machine and you got uniformity.

At the time this was not supported and we ran into crazy things like not being able to upgrade the tar package because the binary installer would attempt to overwrite its own binary during the unpacking process. It's probably gotten much better since then.

We ditched Gentoo for Debian later just because it was so much easier to manage. But it sure was an interesting time. (It did kinda of put me off Gentoo for life though.)

Gentoo in binary form

Duncan — Sat, 30 Dec 2023 07:22:40 +0000

FWIW 2004 was when I switched to Gentoo as well... and I'm still with it. (Another five years and I'll be looking at my Gentoo-silver!...)

The binary package format has matured and been speced out, now, tho the old format's still supported. They only recently decided the new format was stable enough to default to it, thus allowing them to announce this (it has been in the works for awhile, waiting for new-format stabilization).

The new format's basically a tarball of tarballs, with the package content one sub-tarball and most of the metadata another (the old format was a content tarball with uncompressed metadata appended, which worked, but...). There's support for (optional) package signing and verification in the new format, thus allowing secure binary delivery (portage has supported signed srcpkg repos and the main gentoo tree has been signed and default-verified for some time now), with portage allowing end users to do their own signing if so desired.

USE flag metadata, etc, is now included, and there's now multi-build/multi-package support, so your package server can build separate binpkgs for multiple setups and portage will automatically pick the right one if available, building its own if there's no match. (The amd64-arch official packages are available in three variants,gnome/systemd, plasma/systemd, nomultilib/openrc.)

(BTW, there's another gentoo project, prefix, that allows installing portage and packages as a secondary system operating on a non-gentoo primary host, for those who aren't ready to go full-gentoo-native. Among other things it supports gentoo hosted on the new Apple hardware/OS stack.)

For the big-two archs, the official gentoo binpkgs are generic-amd64 (-march=x86_64 -mtune=generic -O2 -pipe) or generic-arm64 (-O2 -pipe), but you can of course mix and match with your own builds if you want specific code and/or optimization targets for particular packages, just as you can for your own set of USE flags, or just build your own as usual if you want to customize everything.

Gentoo in binary form

ferringb — Sat, 30 Dec 2023 06:04:43 +0000

First, the underlying format changed; back in your timeframe of '04 I was just getting EAPI shoved in along w/ the environment support for it. After that landed we could start evolving the format; eapi2 (2010?) is when SLOT deps became possible, and SLOT deps got extended into ABI level deps in later EAPIs. Along the way things like transitive USE deps (if this package has python support, my dep must have it) also landed.

The second thing here is that the binpkg and binhost formats had to evolve to be able to represent binary metadata fully... *and* the binpkg support had to improve. Zmedico (afaik) did a shitton of work on that front; specifically reworking the remote binhost support so it became possible to store multiple builds of a package with differing build linkages + USE configurations.

I've been out of that game for a while, so apologies to anyone who I may have overlooked in regards to their work.

Gentoo in binary form

ejona86 — Sat, 30 Dec 2023 04:00:53 +0000

I used prebuilt packages with Gentoo in 2004-ish, sharing across my own computers and official builds quickening new installs. Did the binary package tooling improve in a fundamental way, or is this mainly more frequent and numerous binary packages?

Gentoo in binary form

WolfWings — Fri, 29 Dec 2023 22:20:09 +0000

Basically, but Gentoo's native package manager has matured built-in binary package handling now so they can roll it out like this instead of using a separate secondary package manager like that overlay/fork did.

Gentoo in binary form

Wol — Fri, 29 Dec 2023 22:00:51 +0000

Is this Sabayon by another name? It's always felt weird to me that the two distros aren't one, just distributing in different media.

Cheers,
Wol

Gentoo in binary form

ErikF — Fri, 29 Dec 2023 21:36:55 +0000

I personally don't use the binaries often because my main computer has enough oomph to compile pretty much anything really quickly (and I like to use `-march=native` on my builds.) I do, however, use the Rust i686 binary on my old Asus eeePC 701 because I really don't feel like waiting until the heat death of the universe to see the compilation finish, even with the use of distcc (when it works.)

Gentoo in binary form

Absolutely_Penguin — Fri, 29 Dec 2023 21:14:43 +0000

the puritans will cry out, but this is a great move

Gentoo in binary form

NightMonkey — Fri, 29 Dec 2023 19:39:51 +0000

Gentoo has so many options for the discerning computing user and system administrator. Flexibility of the kind that Gentoo offers isn't for everyone, but I an glad it is there for those who can take advantage of its features and approaches. Glad to see it become even more flexible. I am also thankful for the contributions the project has made to the greater Linux and software communities.

Gentoo in binary form

dullfire — Fri, 29 Dec 2023 17:05:40 +0000

This could be pretty neat. My pinebook pro (with a gentoo install) has been finding it hard to link gcc 13, so this might make gentoo a bit less tricky on that system.