LWN: Comments on "The mkosi OS generation tool"

Use cases

GNUtoo — Tue, 17 Aug 2021 00:13:30 +0000

Hi,

I've been looking for some software with the abstraction of mkosi for a long time.

The first thing that came to my mind are things like vagrant[1] and packer[2] to
create system images in a variety of formats, but they seem better adapted at
modifying existing base images than creating them from scratch.

So to build an image with vagrand and/or packer you could either:

- Use one of the images made by hashicorp (the company behind packer and
vagrant), or the community of its users[3][4] as a base. The issue here
is that it might not be possible to trust these images in the same way
than images that are part of official distribution releases. This is
because the vagrant repositories and the distributions have different
policies.

- Use official VM images or cloud images from distributions as a base.
The issue here is that not all the distributions have such images, and
you might also want to build types of images that aren't released by your
distribution (like add integrity checking, or use the latest packages
instead of the ones from the last release).

- Build base images from official distributions installers. The issue here
is that, even if it can work, sending keystrokes once the installer is
booted doesn't seem like the right abstraction to me[5].

- Like Tails, you could also write yourself the code to generate that base
image[6], but the limitation here is that it might not work for all the
images types, and it would probably be better to have some common project
to do something like that instead.

Guix[7] (and probably Nix as well) already solved this issue as you can package
the same software in several way (in a VM[8], in a disk image[8], in a debian
package[9], in a docker container[9], etc). And in some cases where it produces
a booting system (like with the VM, disk image, or docker container) there is
also an API that takes care of the configuration of some applications like Nginx.

The issue is that not everyone wants to use Guix, so having some of these
features available for other distributions is a good idea I think.

And here mkosi has the ability to build base images using the bootstrap tools
like debootstrap, pacstrap, and so on to build images. So that looks useful
even if it's not going to enable users to install any distribution from any
distribution, but work inside distributions and bootstrap tools could be done
to imrpove the situation.

The big limitation in the case of mkosi seems to be that it took the decision
to be legacy free (its manual states "mkosi - Build Legacy-Free OS Images").

Unfortunately the "legacy free" seems to limit a lot the use cases:
- Some distributions are still supporting computers with i686 processors, so
using mkosi to create official installer or VM images won't work here.
- Many ARM computers have system on a chip that can boot on block devices and
that load the bootloader from either an offset on the block device, or in
a FAT MBR partition. The later cannot be supported with mkosi.
- Some modern computers running Libreboot or Coreboot, and some VMs use
SeaBIOS for simplicity. Here again it can't produce images with MBR.

I'll also try to look at alternatives of mkosi that have the same abstraction level as that abstraction level really seem to fit many of the use cases I have.

References:
-----------------
[1]https://www.vagrantup.com
[2]https://www.packer.io
[3]https://www.vagrantup.com/vagrant-cloud/boxes#vagrant-box...
[4]https://www.vagrantup.com/vagrant-cloud/boxes/catalog
[5]https://www.packer.io/docs/builders/virtualbox/ovf#boot-c...
[6]https://gitlab.tails.boum.org/tails/tails/-/raw/master/va...
[7]https://guix.gnu.org/
[8]with guix system image
[9]with guix pack

Denis.

The mkosi OS generation tool

dexit — Sun, 20 Oct 2019 15:16:16 +0000

well I for one did look at KIWI but it supports a rather limited set of hosts and target OS-images. Then again, mkosi fails for all targets save Debian, of course. A clear lose-lose situation.

The mkosi OS generation tool

dexit — Sun, 20 Oct 2019 14:45:54 +0000

On Manjaro in Oct. 2019, only the Debian image gets created successfully.

Arch, Fedora, Centos, ClearLinux, Ubuntu, Mageia, OpenSUSE all fail for one reason or another. However sometimes its just that the packager tools such as zypper for SUSE do not build from AUR or anyhow.

In other cases the python error suggests some mkosi problem. btw: Arch pacstrap is here: https://projects.archlinux.org/arch-install-scripts.git/t...

The mkosi OS generation tool

nix — Wed, 05 Jul 2017 23:23:47 +0000

Yeah. I wish there was a faster way to share existing host filesystems than NFS over localhost, but there doesn't seem to be one... (and NFSv4 is pretty nippy, actually. It's only if you're stuck on v3 or God forbid v2 that it starts to bite hard.)

lxd on Fedora

iainn — Mon, 03 Jul 2017 09:41:58 +0000

Ah. I had been considering a COPR repo (similar to a PPA) to install LXD on Fedora. The LXD package from the COPR doesn't support SELinux unfortunately.

But thanks, that's good to know. I hadn't considered a Snap package. Nor had I seen the recent news that snaps work on Fedora with SELinux.

The mkosi OS generation tool

rahulsundaram — Sun, 02 Jul 2017 21:46:06 +0000

>A full time paid employee that did it?

That is a really weak rationale to think that this is done under some organizational direction. systemd itself was written by Lennart on his own time originally.

The mkosi OS generation tool

jospoortvliet — Sun, 02 Jul 2017 21:15:51 +0000

A full time paid employee that did it? Who is working in and around this stack? I know rh doesn't work that black and white and I know lennart well enough to know he was most likely not told to do this but it isn't a crazy assumption...

The mkosi OS generation tool

rahulsundaram — Sun, 02 Jul 2017 15:02:48 +0000

> Seems to me another NIH here by RH.

What makes you think Red Hat is deciding this?

The mkosi OS generation tool

mrdocs — Sun, 02 Jul 2017 03:39:14 +0000

I'm wondering why no one looked at https://github.com/SUSE/kiwi It also builds for RH and Debian, as well as many output formats, ISO, docker images, cloud images etc.

Seems to me another NIH here by RH.

The mkosi OS generation tool

sbaugh — Sun, 02 Jul 2017 01:33:42 +0000

9P shouldn't be used to boot a container with QEMU/KVM in production, though. It's too slow and quite insecure. So building GPT disk images is still very useful.

The mkosi OS generation tool

stgraber — Fri, 30 Jun 2017 16:57:57 +0000

Hmm, I'm the LXD project leader and we do run automated tests on Fedora 25 using LXD installed through snapd on systems that have SELinux in enforced mode, so you shouldn't have to turn it off.

sudo dnf install snapd
sudo snap install lxd
sudo lxd init

The mkosi OS generation tool

gdamjan — Fri, 30 Jun 2017 08:30:46 +0000

that's only practical now with the support for multi-stage builds (perhaps, I haven't looked too deeply into it)

The mkosi OS generation tool

iainn — Thu, 29 Jun 2017 17:37:43 +0000

It's a useful tool, which coincidentally I discovered this week.

I wanted an easy way of spinning up another distro. My first thought was to check if LXD were suitable, but to use LXD on Fedora you have to disable SELinux. My second thought was Docker, but it wasn't suitable because I wanted to persist state changes.

I found mkosi + systemd-nspawn slightly easier, and hopefully provide better isolation, than setting up a chroot.

The mkosi OS generation tool

zlynx — Thu, 29 Jun 2017 17:05:41 +0000

If it gets complicated you pull in a layer that has an actual script in it. Then call that instead of trying to duplicate Puppet or whatever in Dockerfile.

The mkosi OS generation tool

gdamjan — Thu, 29 Jun 2017 13:16:00 +0000

the fs layers are a good idea in docker (if inefficient)

The docker file itself, is just awful. It's like a worse shell script

The mkosi OS generation tool

rahulsundaram — Thu, 29 Jun 2017 02:11:24 +0000

A lot of tools listed in that page seem pretty Debian specific though.

The mkosi OS generation tool

pabs — Thu, 29 Jun 2017 01:29:30 +0000

The proliferation of tools in this space is incredible:

https://wiki.debian.org/SystemBuildTools

The mkosi OS generation tool

jgg — Wed, 28 Jun 2017 21:05:23 +0000

Sure docker can, I've done it a few times. You start with the default dockerhub image for the OS and the install a few packages (eg grub, a kernel, etc) and a few other small modifications. Trivially embodied in a dockerfile and the result is something that can boot containerised systemd as pid 1 or be poured into a disk image and booted on bare metal. The same kind of basic steps you need to do anyhow if you used debbootstrap.. Docker just caches it and makes the initial image available no matter what the host OS is.

In the mainstream container world the popular way to run a container under kvm is with 9pfs (eg in rkt), this is what I mean by 'unique'. This is actually where I ended up after trying mkosi for systemd. Instead, I built systemd using a building container (with a script similar to the one I linked), then 'make installed' it into a temporary running container and directly ran kvm and 9pfs exporting the containers / as rootfs to run the build. Much faster than building an image, and I get more trivial points to customise things I need, like using my own kernel build or modifying the FS before booting to test the stuff I needed..

For mkosi it was the initial package extracting/install that failed for me, and it didn't cache the download step before that, so the entire thing was a PITA to debug. It also seemed like it wanted to run various loop devices and other things on the host when it was building the image, which I'd honestly rather it do inside a privileged container.. I also gave up trying to build a Fedora boot image from my Ubuntu box (again, needs too many host dependencies).

IMHO, the thing about docker, is it has one really good idea - dockerfile, with the overlayfs based caching layers and downloadable seed - and a bunch of not so good, out dated execution around that good idea. systemd has good execution in systemd-nspawnd, rkt provides most of the orchestration piece on top of that, but that entire world lacks a refined equivalent to dockerfile to make the images in the first place..

You are right, I used something between v1-v2 (dec 16 github TOT) in an attempt to do some work with systemd 232.

The mkosi OS generation tool

mezcalero — Wed, 28 Jun 2017 19:24:02 +0000

hmm, Docker cannot generate images that you can boot on bare-metal, can it?

"unique disk image format"? Do you mean the raw GPT disk images mkosi can generate with that? I don't find that particularly "unique", it's probably one of the most universally understood disk format these days, as any Windows, any ChromeOS and pretty much everything recent uses GPT.

And to clarify, after the initial packages have been extracted (i.e. right after the initial debootstrap or dnf --rootinstall), everything relevant we run runs inside an nspawn container.

mkosi is currently at version number 3, hence I don't think you can have tested mkosi 232 ;-). "232" sounds more like a systemd release version.

Lennart

The mkosi OS generation tool

jgg — Wed, 28 Jun 2017 18:57:11 +0000

To me, the idea is good, but, there is a lot of mature stuff out there that does this, and arguably does the sub-steps better (eg docker). Lots of people have per-project wrapper scripts to make this work right, eg rdma-core has a docker based tool to build the source under various containerised distributions images and produce .rpm/.deb files.

https://github.com/linux-rdma/rdma-core/blob/master/build...

Which was inspired by other tools and customised to run smoothly with the source tree it is embedded in.

mkosi is basically the same idea but for systemd. It doesn't really do anything particularly special, expect that it exports the container into a unique disk image format. Other tools like rkt can already do this, but their images are for running in kvm, not bare metal.

The last time I tried to use mkosi (232) it just failed. The main trouble seems to be that it does a lot of work in the host, so it is quiet sensitive to the host configuration, which to my mind, largely defeats the entire purpose.. It was also bonkers slow, but maybe I didn't get far enough to have the caching turn on, or that version didn't have sane caching yet.. Plus it wanted to run as root on the host and do all sorts of strange things - again not the kind of safety I want from a containerised build tool.