Stapelberg: distri: a Linux distribution to research fast package management

Just face it: Year is 2019 and pretty much all package/filesystem related inovations were achieved, and people tend to create new features on Linux distros that are based on those.

"This isn't innovation" is such a childish argument today. Get over it.

I'm sure Oracle is crying all the way to the bank at the thought that nobody wants to play with their ball.

This is a proof-of-concept, and FUSE allowed much quicker iteration times than writing my own kernel module :)

In my day-to-day, I don’t actually notice a perceivable slow-down, so the current FUSE implementation quick enough to demonstrate what I wanted to show.

But yeah, moving the file system into the kernel would certainly be a step on the productionization checklist for this approach!

There is also an interesting paper which talks about moving only the hot paths of a FUSE file system into the kernel using eBPF: https://www.usenix.org/system/files/atc19-bijlani.pdf

Correctness is vital of course. We have different priorities regarding speed vs features, it seems :)

Thanks for bringing up ravenports, haven’t heard of it before.

Taking a quick look at its buildsheets, they look like Makefiles and allow arbitrary commands at build time (e.g. https://raw.githubusercontent.com/jrmarino/Ravenports/mas...).

I think this is a mistake when reaching a certain number of ports and/or contributors. Restricting the possibilities allows for easier large-scale changes and other tasks that benefit from machine-readable packages.

In distri, package build instructions are declarative¹. See https://github.com/distr1/distri/blob/master/pkgs/irssi/b... for an example and https://repo.distr1.org/distri/jackherer/docs/building.html for the documentation. I want to publish a blog post going into more detail about this approach, too.

① Of course there is an escape hatch in the form of writing custom build instructions, but very few packages need that. Defaults matter—reducing a change from “audit these 700 packages” to “change the builder and audit the 7 oddball packages” drastically helps :)

It should be pointed out that Gentoo didn't care about correctness until third-party package managers came along that did. Portage itself is an unmaintainable disaster to this day, and the distro's QA relies on people with ad-hoc tinderbox scripts and pkgcore. The whole thing is going to come crashing down one day unless a concerted effort is made to pay down the technical debt.

overlayfs would probably be a good step up from fuse on several axes. I haven't checked if overlayfs allows this, but you wouldn't even need to define an "upper", just chain up the "lowers" for the per-package exchange directory.

Not sure what the performance implications are, exactly, but it'd probably scale with the number of packages participating in the "exchange".

This is without any tests, just a strong assumption.

OverlayFS (or UnionFS) aren't good for this. If any user would "cd /usr/share/man", then you couldn't add "/ro/just-recently-installed_08.15/man" to it. Because you cannot change the union while one is using it. This would mean that for package updates you'd need to go into some single-user mode. In some scenarios this would be okay, but generally this would just add a layer of inconvenience.

> I imagine the squashfs cannot be unmounted until all open filehandles to it have been closed

You can unmount a filesystem while there are open filehandles using the MNT_DETACH umount(2) flag (or umount --lazy from the command line). That disconnects the mount point but keeps the filesystem around until all the handles are closed.

> 1. Have you done any benchmarking?

The most recent thing I have tweaked is how binaries locate their shared libraries, and how they are started. We used to use a big rpath (I think NixOS still does?), but putting symlinks into a lib/ subdirectory per package and pointing rpath to that directory is much faster (thanks to e.g. ld.so’s caching).

Overall, performance is good enough for day-to-day use-cases.

> I imagine it would be relatively straightforward to "flatten out" the filesystem (copy all the files directly into place rather than mounted squashfses and "exchanges") to benchmark the overhead of this system.

Indeed. In fact, this happens when we generate an initrd with dracut.

> 2. I'm kind of curious what the boot process is like - is a flat initrd required in order to prep all the package and exchange mounts on the root filesystem first?

An initrd is not required with distri unless you’re using an encrypted file system.

init is defined in https://github.com/distr1/distri/blob/master/cmd/distri/i.... As you can see, pid 1 mounts the FUSE file system at /ro, then execs systemd and the boot proceeds normally.

> 3. Also, what are the implications of in-place upgrade of a package while the system is running? I imagine the squashfs cannot be unmounted until all open filehandles to it have been closed, which could make things difficult, but maybe that can be circumvented by bind-mounting the updated package *over* the path for the existing one?

Packages aren’t ever updated in that sense, only new versions are being added to the package store. E.g., zsh-amd64-5.6.2-5 and zsh-amd64-5.6.3-6 can be installed at the same time.

The /bin exchange directory will point to the binary contained in the package with the highest distri revision number (6 > 5 in the example above), so new processes will use the new version.

To get rid of old packages, “distri gc” deletes the image from the package store, so at the next reboot it won’t be available anymore.

Agreed: there definitely are low-hanging fruit in the bigger package managers with regards to parallel downloading :)

Regarding hooks/triggers: this depends on the specific semantics of the package manager. I think in Debian you have so many extension points (check out https://wiki.debian.org/MaintainerScripts#Upgrading) that you can’t easily change the order in which things are done.

In distri, I want to demonstrate that even the extreme point of view of getting rid of hooks/triggers altogether is viable, so other distributions should feel encouraged to restrict their offering in order to optimize.

I appreciate your enthusiasm for the project! I definitely agree with all the points you listed :)

> I would like to give it spin and hoping it will play well with other OS in the disk ...

It should play as well as any other Linux distribution: distri uses GRUB, which uses os-prober to find other OS.

That said, please be very careful and be sure to have a backup!

> I am wildly hoping it allows install without much fuss( you know that the tipping point, if it's not working at first go with ease, how claver or good your work, it will be dump in the dust), I have seen real pathetic process in others...they survive because they are backed by big shops...heck

Note that distri does not have an installer right now. The idea is that you try it out as a live system only currently.

I have some ideas around how an installer should work, but we’ll see if I get around to implementing that :)

Thanks for the links! That isn’t really my area of expertise, so I haven’t done anything in that regard, and I don’t see that changing in the foreseeable future :)

> I really wish dpkg would stop fsync-ing each and every file it unpacks…

As BTRFS users (where dpkg si *very* slow on a spinning disk), I suffered of this. Even if I can understanding the reasons. I solved using libeatmydata (which I had to patch otherwise some syncs are not avoided).

However in the few filesystem capable of doing a snapshot (like btrfs, zfs or a filesystem layered on top of dm-snapshot), a better solution would be to make a snapshot before the upgrading; after the upgrading the package manager may delete the snapshot if all was fine or may perform a rollback in case of problem.

I create a script which doing that, but it was far to be perfect.

Anyway most of the BTRFS problems disappeared when I switched to a SSD (even tough BTRFS for these operation is still slow).

> Do it in the background, and if the system crashes, have some
> sort of redo log job that can run on startup to make everything fine again. Or something. :-)

> Having a bar is always good. With some luck, this will spur interest in the more traditional package managers (although most of my upgrades run in nightly cron jobs these days, so installation speed isn't something that really bugs me all that much).

When using VMs I generally just switch out the entire root fs, the process of building that FS could also hugely benefit from speeding up as would building containers, because in this case a mistake during the process usually means starting over.

> I really wish dpkg would stop fsync-ing each and every file it unpacks… Do it in the background, and if the system crashes, have some sort of redo log job that can run on startup to make everything fine again. Or something. :-)

There is the force-unsafe-io option (or eatmydata). I find the idea of just staging the whole change in another file tree and then swapping things out on disk interesting. When I used ZFS as my root FS I just used snapshots so I could revert to the "before upgrade" state.

There is probably also some low-hanging fruit in the area of archive formats and compression, the ubiquitous tar really isn't the most suitable for the task, performance wise.

> Are you sure cutting install times by unpack phase justifies adding level of access indirection which will impact runtime performance? People usually install software once, then use it for a time.

Yes, and that’s what I want to demonstrate with this project. The runtime overhead is minimal, and the installation speed-up significant.

> Sounds like NixOS without deps closure immutability. NixOS remounts store readonly and uses hashes in store paths to guarantee immutability not just for individual packages, but for packages together with all their deps (which are still separate packages and are shared between depending packages when it's safe).

distri provides the same guarantees.

> NixOS does this with symlinks. Its system-wide "exchange directories" are stored in special packages which are on the top of deps hierarchy and are available via /run/current-system/sw.

Yep. Creating these symlink farms can take seconds, hence distri provides them on demand (which is a bit quicker).

> A new version of one package obsoletes a different. How does the obsolete package get removed?

The obsolete package will be removed once it’s no longer referenced by any other installed packages. In terms of declaring the roots (packages which users explicitly want to install), I could imagine having a hand-curated list, but I haven’t thought about this a whole lot from the UX perspective. Perhaps there’s a good solution :)

> Does it still work if there are multiple sources of packages?

Not sure what you’re getting at? Can you rephrase your question?

> Two packages provide the same exchange file. The one with the lower revision numbers gets a bug fix release. Now the winner is switched.

Global exchange directories (as opposed to per-package exchange directories) are meant to be used where there is loose API coupling, so I’m not entirely sure what the problem is with the winner switching? Other packages will still find and use precisely the dependencies that they were built with, so if we’re talking about e.g. LDAP DNS resolution (a glibc plugin located at runtime), picking up the bug fix seems like the correct path of action to me? :)