An alternate path for immutable distributions

LWN has had a number of articles on immutable distributions, such as Bluefin and Bazzite, in recent years. These distributions have taken a variety of approaches, including using rpm-ostree, filesystem snapshots, and bootable container (bootc) images. But those approaches, especially the latter, lead to extra complexity for a user attempting to install new software, instead of just using the existing package manager. AshOS (Any Snapshot Hierarchical OS) is an experimental AGPL-3-licensed "meta-distribution" that tried a different approach more in line with traditional package management. Although the project is no longer updated, it remains usable, and can still shed some light on a potential alternate path for users worried about adopting bootc-based approaches.

There are a few reasons to find immutable distributions appealing. The fact that updates can be applied and rolled back atomically is probably chief among them, but they also make it easier to reproduce a corresponding installation from a small set of configuration files. Immutability means that changes to the configuration are cleanly separated out, so it's easy to see how a long-lived immutable system has been changed, and to reproduce a new system from those changes. The question, as always, is whether these benefits are worth the disk usage of A/B updates, build times of any custom images, complexity, and inconvenience. AshOS was an attempt to change the balance of that tradeoff by making it possible to add immutability onto an existing distribution, while keeping the existing distribution's package manager in control.

To do this, AshOS makes and manages snapshots of the root filesystem (including installed software and configuration files, but excluding users' home directories). These snapshots can be overlaid on top of one another, so that package-management operations can be separated out and named (similar to sysext images). For example, a user might have one snapshot for the base OS installation, another for their graphical user environment, and a third for their mail server. To try out a new piece of software or configuration change, they would create a new snapshot to work in on top of the existing ones. Within that snapshot, they could use the distribution's normal package manager to install the software. If that turned out to have horrible consequences, they could remove the most recent snapshot and return to the previous (working) configuration.

Other immutable distributions do offer similar capabilities. Fedora Silverblue-based distributions can use rpm-ostree to layer packages on top of a base image, for example. But doing so is rife with sharp edges compared to just using dnf — for example, if a base image is updated to include a package that was previously layered in, removing the exising layer is difficult to do with rpm-ostree.

Trying it out

It's a charming idea. I have been using Silverblue for a bit more than a year, and found it just enough of a hassle for AshOS's promised simplification to be tempting. Unfortunately, AshOS is in the "usable but not without struggle" stage of open-source projects. The recommended installation procedure is to obtain a live ISO of one's preferred distribution, use that to download the AshOS source code and to configure disk partitions, and then run AshOS's setup script. AshOS tries to be portable across distributions, but it does still have a small amount of per-distribution code (to facilitate installation and produce smart diffs between snapshots). The currently supported distributions are Alpine, Arch, CachyOS, Debian, Endeavourous, Fedora, Kicksecure, Proxmox, and Ubuntu — although the documentation suggests that Gentoo also works, and other distributions similar to the supported ones might work.

I downloaded Arch's 2026.01.01 live ISO image, and went through the installation process in a virtual machine. It took me a few attempts to do it right: AshOS has no particular enforcement that one has set up the partition table in a sensible way, but it also has to know exactly what each partition is for in order to set up snapshots correctly. My first time through the procedure, I ended up with an unbootable system. Eventually, I was able to match AshOS's understanding of my chosen disk layout to the partition table, and things went more smoothly.

The installer is text-based and somewhat rudimentary, but not terribly complicated. After getting through the difficult partitioning-based questions, it asks for the normal details such as username, password, and timezone. Once complete, it installs a minimal package set from the host OS (Arch, in my case) and instructs one to reboot. In the default installation configuration, it uses Btrfs on the root disk for making snapshots, but the documentation suggests that it can be made to work with other filesystems if one tries.

By default, AshOS doesn't install anything other than the bare minimum needed to log in at a text console. It does have the ability to install a few "profiles": special commands that set up common desktop environments. I made a new snapshot and then installed the GNOME profile. Rebooting produced an extremely minimal, but functional, Arch installation. From there, I experimented with adding, cloning, deleting, and updating snapshots with the provided command-line tool (ash), and found the process much more intuitive than the installation procedure.

AshOS manages snapshots in a tree structure starting from the base OS installation. Any branch of the tree can be selected at the boot menu. So, for example, it is possible to have two different desktop environments installed (for testing, perhaps) on top of a shaed set of software. More commonly, one could make a copy of one's existing configuration (using AshOS's tree-manipulation commands), try out an upgrade or modification, and still be able to easily switch back. Each layer is given an ID number and an (optional) human-meaningful description, both of which show up in the boot menu and when manipulating the tree of snapshots from the command line.

The "ash branch --snap N" command can be used to branch from snapshot N; this prints out the ID of the new snapshot, M. Then "ash chroot M" will spawn a new shell within this snapshot wherein one can make changes using normal tools, such as "pacman -Sy firefox" (pacman's somewhat cryptic installation command) or "vim /etc/resolv.conf". When exiting the shell, ash saves the changes and updates the version of the snapshot in the tree. There is also an ash live-chroot command to open a shell with the currently running snapshot mounted as writable, in case one needs to mess with things without creating a separate snapshot first.

There are also some helper commands for running a command on each snapshot in an entire subtee. For example, one could run their package manager's update command on all of the snapshots below a certain point on the tree.

Configuration

AshOS's snapshot management clearly qualifies it as an immutable meta-distribution, but one weakness of a purely snapshot-based approach compared to creating custom bootable containers is configuration management: once a system is set up, how can you tell exactly what was changed, and how to recreate the same setup elsewhere?

This is where the majority of AshOS's distribution-specific code comes in. AshOS has a set of hooks for each supported package manager that lists installed packages and attributes files to them. The ash diff command uses these hooks to produce diffs between snapshots. The diffs show which packages were added and which configuration files were changed. This isn't quite the same level of reproducibility as building an operating-system image from a container description, but as long as one remembers to assign meaningful descriptions to each snapshot, it seems like a clean-enough way to track operating-system changes.

Overall, AshOS is rough. The command-line tooling, for all that it seems reliable, has slightly inconsistent syntax. The installation procedure is baroque, the documentation is a bit messy, and the last development occurred just over two years ago. The project's creator and only contributor, "i2", appears to have lost interest. Despite that, it works — and arguably does so more smoothly than traditional immutable desktops such as Fedora Silverblue. It serves as a proof of concept that it is possible to have the nice parts of immutability (atomic upgrades and rollbacks, configuration management) without giving up traditional package management or switching to a container-centric approach. Nothing that AshOS does is out of reach for someone with some scripting knowledge and a filesystem that supports snapshots, although it does a good job of packaging those operations into a single tool.

AshOS is currently only suitable for people who don't mind relying on experimental, unmaintained software. But the core principle of organizing and layering filesystem snapshots to turn a traditional Linux system into an immutable one seems entirely sound. Perhaps such an approach could prove a useful middle-ground for people wrestling with the downsides of bootable containers — or inspire the developers of those systems to improve the user-interface for installing additional packages.