LWN: Comments on "A report from the 2022 Image-Based Linux Summit"

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 11 Nov 2022 15:00:21 +0000

Best solution is probably to use U-Boot on such machines, to get a UEFI interface. Otherwise, there's a patch being prepared for Grub by RH: https://github.com/osteffenrh/grub2/commit/d0c402c9615942...

A report from the 2022 Image-Based Linux Summit

pabs — Fri, 11 Nov 2022 08:50:36 +0000

Will there be a way to launch UKIs on BIOS based machines? Lots of cloud vendors still use it and lots of folks could be stuck on old computers they can't afford to upgrade.

A report from the 2022 Image-Based Linux Summit

bluca — Thu, 10 Nov 2022 14:41:14 +0000

Shim is orthogonal to this - it's the second stage loader that runs the UKI (systemd-boot or grub - yes there's a WIP patch for that). Of course if you enroll your own keys, you can self-sign the UKI and load it directly from the UEFI menu.

A report from the 2022 Image-Based Linux Summit

smitty_one_each — Thu, 10 Nov 2022 14:05:16 +0000

> By ... Lennart Poettering ...

> ... our employer, Microsoft ...

I remember seeing neither the TPS Report, nor hearing the conspiracy theory covering this event.

Nonetheless: cheers. I think your work has been a big win for the world in general, Lennart.

A report from the 2022 Image-Based Linux Summit

mjg59 — Thu, 10 Nov 2022 05:28:29 +0000

The turnaround time for review wouldn't be viable even in the event of a policy change. The point of Shim wasn't primarily to deal with licensing issues, it was to ensure that distros could handle their own updates for the components that were more likely to change quickly. You don't want to wait a month to be able to release a kernel update.

A report from the 2022 Image-Based Linux Summit

pabs — Thu, 10 Nov 2022 04:55:02 +0000

I see no mention of shim in this post but the UKI images are to be secure-boot signed, does that mean Microsoft is now willing to sign GPLed UEFI stuff?

A report from the 2022 Image-Based Linux Summit

amacater — Wed, 09 Nov 2022 17:56:56 +0000

BoF - Birds of a feather (flock together). Often used for a small informal meet up, gathering or lecture.

See, for example Debconf which has large plenary sessions, individual talks and then semi-formal BoFs to get people together on a small topic.

A report from the 2022 Image-Based Linux Summit

Wol — Wed, 09 Nov 2022 17:54:29 +0000

Birds of a Feather, I believe ... (ie a small bunch of like-minded people).

Cheers,
Wol

A report from the 2022 Image-Based Linux Summit

calumapplepie — Wed, 09 Nov 2022 17:46:55 +0000

Word, yo.

A report from the 2022 Image-Based Linux Summit

calumapplepie — Wed, 09 Nov 2022 17:39:33 +0000

> The summit was intentionally kept small, as it was meant to be a series of conversations and brainstorming sessions, with no fixed agenda or presentations — a BoF-style event

What does "BoF" stand for in this context?

A report from the 2022 Image-Based Linux Summit

hsiangkao — Mon, 07 Nov 2022 06:39:26 +0000

IMHO, in short, it's much easier to do vendor signing, data verification, device (e.g. LBA-based) write-protection, reproducible builds, corrupted data online recovery, and more by doing image-based deployment.

A report from the 2022 Image-Based Linux Summit

willy — Sun, 06 Nov 2022 18:10:31 +0000

You make a Powerful Point

A report from the 2022 Image-Based Linux Summit

champtar — Sat, 05 Nov 2022 02:06:28 +0000

> I can maybe see about trying to write some sort of docs for this there also in the uapi group?

I would love to see such doc, so I have something to send to vendors so they might improve their RPMs (even if I don't have high hopes).

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 04 Nov 2022 18:17:13 +0000

What's been produced is a set of features and specifications, anything can provide and implement those. There's no reason something else could not implement them, "just" that someone has to do the work, and it's _a lot_ of work.

A report from the 2022 Image-Based Linux Summit

ale2018 — Fri, 04 Nov 2022 16:38:41 +0000

Much of the discussion appears to assume systemd by default. I'm wondering whether the project is actually focused on that kind of distributions —thereby enlarging the gap between those ones and the ones without systemd— or not.

A report from the 2022 Image-Based Linux Summit

aszs — Fri, 04 Nov 2022 15:57:42 +0000

I've been intrigued for a while about possibilities afforded from combining open-source reproducible builds with remote attestation.

Imagine a certificate authority like Let's Encrypt but the challenge it requires for signing the request isn't only to prove the server answers to the cert's domain but also a remote attestation that the server is running the reproducible workload whose digest is in the cert. Then as a user I can have some assurance that the server I'm connecting to is running the code I'm expecting it to just by checking that digest.

Sounds like we're getting closer but how far away is the state-of-the-art from having the necessary building blocks to enabling something like this? What is missing?

A report from the 2022 Image-Based Linux Summit

walters — Fri, 04 Nov 2022 13:42:57 +0000

> ima/fs-verity do not fulfill the same role due to very important differences: they do not cover the whole filesystem hierarchy

(Yes, I know this - and I'm pretty sure you know I know too ;) But, I know we're also conversing with an audience so some repetition is needed unfortunately)

> it simply means that they sit in a different place on the wide spectrum of image-based Linux, with significant enough differences to be meaningfully distinguishable.

Yes! We're in agreement. There's lots of nuances between different points on that spectrum. And a lot of space to share knowledge and approaches across them.

Security is also (as you know) not a binary thing. Personally, I think one of the biggest problems in our industry in general is people *not updating the operating system at all*. Switching to "the operating system auto-updates by default" was one of the big changes from the original Container Linux - and we've carried that forward in both Flatcar and Fedora CoreOS. It's actually a pretty profound difference in practice - I blogged about this but basically I think there's a strong sense of "responsibility" for updates that shifts from "I typed apt|dnf|whatever update and it broke, so it's my fault" to "the system just fell over, it's $OS's fault".

On this topic, I think "unlocked" image based systems (like pulling btrfs snapshots, snap and ostree, etc.) having the properties of e.g. transactionality and offline updates (not disrupting the running system) provide a very meaningful improvement to security from *this* aspect. Not to mention the "we tested the update image server side and you are bit for bit reproducing it".

Another way to say it is, these properties are also very meaningfully distinguishable from traditional package based systems.

What I'm arguing at the core is: s/image based/dm-verity based/ in the original sentence. I also would tend to use the term "locked" or "sealed" when talking about this because honestly "image" means too many things already.

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 04 Nov 2022 11:42:22 +0000

No, there's an option in that bios to turn the 3rd party UEFI cert back on, and the usual option to wipe the lists of certs and enroll your own.
Of course the user experience given by the default settings sucks, and it is being worked on. But it has nothing to do with this.

A report from the 2022 Image-Based Linux Summit

aragilar — Fri, 04 Nov 2022 11:34:40 +0000

I may have misunderstood something, but is not https://mjg59.dreamwidth.org/59931.html an example?

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 04 Nov 2022 10:26:55 +0000

Yup, trying to get it in xdg-specs as well: https://gitlab.freedesktop.org/xdg/xdg-specs/-/merge_requ...

A report from the 2022 Image-Based Linux Summit

pothos — Fri, 04 Nov 2022 10:23:30 +0000

Actually we discussed how rpm-ostree does a 3-way merge (even though not line by line) when updating /etc.
The outcome of this discussion was that we want to push applications to install their default configs to /usr instead and allow drop in files under /etc, which is written down in https://uapi-group.org/specifications/specs/base_directory_specification/

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 04 Nov 2022 09:53:39 +0000

It was an Excel-lent conference too! Thank you, I'm here all week.

A report from the 2022 Image-Based Linux Summit

bluca — Fri, 04 Nov 2022 09:52:34 +0000

systemd-cryptsetup/cryptenroll support recovery keys (even as a QR code! It's really cool, you should try it out) that can be used as additional slots in LUKS, so offline recovery is possible and as easy as it can be made.

There is no such "legal problem", this scare story has been around for 20 years since UEFI first arrived, and guess what, it never happened, because it does not make any sense. The UEFI spec mandates that the machine owner, with verified physical presence at the keyboard, can swap the keys.

A report from the 2022 Image-Based Linux Summit

smurf — Fri, 04 Nov 2022 07:51:53 +0000

Presumably the home partition is encrypted with LUKS. LUKS can store more than one key, so one is in your TPM, one is a passphrase encrypted by a secret on an USB stick, and one is on a MicroSD card that's hidden in your safe.

Not to mention your encrypted backups.

A report from the 2022 Image-Based Linux Summit

zdzichu — Fri, 04 Nov 2022 06:50:34 +0000

Well, restore from backup, of course!

A report from the 2022 Image-Based Linux Summit

NHO — Fri, 04 Nov 2022 06:06:29 +0000

This solves technical problem of security from third party, in cool way.
What happens if motherboard in my laptop died from coffee-related accident and I need to extract data from my SSD that was automatically encrypted without asking me on installation, with keys stored in dead motherboard?
Proposal also does nothing to even address the fairly important legal problem: what if vendor decides that I am also third party, not permitted to meddle with software on my personal computer, and their ownership of root keys entitles them to extract wealth from me for the right to use hardware I own with software I own, until they decide that my hardware is not worth supporting and disable all software on it by the means of short-lived crypto cert?

A report from the 2022 Image-Based Linux Summit

jake — Fri, 04 Nov 2022 01:29:28 +0000

> "met in the Microsoft Office" --- for one second I thought this was a virtual conference.

heh, that's an unfortunate typo, isn't it? :)

fixed now ...

jake

A report from the 2022 Image-Based Linux Summit

xecycle — Fri, 04 Nov 2022 01:26:42 +0000

"met in the Microsoft Office" --- for one second I thought this was a virtual conference.

A report from the 2022 Image-Based Linux Summit

bluca — Thu, 03 Nov 2022 23:35:36 +0000

Even with sysexts, there is still the single root of trust as it's signed dm-verity all the way down. ima/fs-verity do not fulfill the same role due to very important differences: they do not cover the whole filesystem hierarchy (you can create new files, delete existing files and recreate them, create directories, etc), and most important they do not cover the filesystem itself. The kernel is vulnerable to rogue filesystems, it is not a use case that it is hardened against. Signed dm-verity allows you to verify the integrity and trust of the filesystem before opening it, which is crucial for this threat model. In use cases where these properties are needed, it's very important to be fully covered: close enough is not good enough.
When adding the IPE LSM on top (https://microsoft.github.io/ipe/) this allows a pretty neat and hardened system with full code integrity (well, it's at least getting there: still working on enlightening interpreters for scripts).

Of course this does not mean that other systems like ostree and btrfs are bad or that they should not be used, it simply means that they sit in a different place on the wide spectrum of image-based Linux, with significant enough differences to be meaningfully distinguishable.

A report from the 2022 Image-Based Linux Summit

walters — Thu, 03 Nov 2022 21:56:09 +0000

> The key differentiator is support for full integrity protection at the block layer via dm-verity. Locally mutable systems like os-tree cannot provide that by definition. In many settings this is not a minor thing, but a fundamental design goal if not table stakes.

IMO, the definition of "locally mutable" gets fuzzy here when one adds things like 3rd party sysexts into the mix. There's also the opposite case of using IMA or fs-verity underneath file-based systems (like ostree, but not exclusively - the old OLPC model of "let's use rsync to hardlinked filesystem trees" was another variant). Sure, it's not as strong as dm-verity, but it absolutely will (particularly in combination with other mechanisms like dm-crypt and other LSMs) stop many attacks. My classic example is the runc exploit. https://lwn.net/Articles/842164/

But I think we're in agreement here to say that the cases are "dm-verity", "file based", and "btrfs snapshot" right? The original phrasing of excluding ostree from "image based" seems contradictory from having us under a shared discussion group. Not to mention, while rauc supports dm-verity it also seems to support non-dm-verity (e.g. plain squashfs) and I hope you'd agree that it's still an image based update system, it just doesn't have the dm-verity properties (in that setup) either.

A report from the 2022 Image-Based Linux Summit

bluca — Thu, 03 Nov 2022 21:26:44 +0000

> Hmm, I still think of ostree as an image system. I'd tend to say "block device image" perhaps versus "file based"? I would imagine that the people doing btrfs snapshots also think of them as image systems? In the end, the filesystem tree you're booting into is exactly reproducing the server, plus support for offline updates. To me those are two key differentiators versus the opposite end of the spectrum - traditional package based systems.

The key differentiator is support for full integrity protection at the block layer via dm-verity. Locally mutable systems like os-tree cannot provide that by definition. In many settings this is not a minor thing, but a fundamental design goal if not table stakes.

> Finally, one thing I'd really hoped that could be discussed, but I'm not seeing here is a common push for filesystem compatibility changes across both distribution and 3rd party software. Specifically around things like installing files to /opt and best practices around building image systems and where state is stored - look at all the stuff in e.g. https://documentation.suse.com/sles/15-SP1/html/SLES-all/... and e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1900691
I can maybe see about trying to write some sort of docs for this there also in the uapi group? I think this is relevant for e.g. systemd-sysext too. Ah, though interesting I see sysext merges /opt too. In the ostree model that's /var/opt to be always part of machine-local state.

The specs repo is on Github and contributions are welcome. There is already a fork of the XDG-Base spec for handling of configuration files (that I'm trying to get merged upstream). In general we do not want to deviate excessively from the FHS, but enhancements are good.

A report from the 2022 Image-Based Linux Summit

walters — Thu, 03 Nov 2022 20:18:32 +0000

Thanks so much for starting this and writing it up! Looking forward to collaborating with others in this space. And hopefully I'll be able to make the next one of these in person!

Some comments:

Others, like those based on OSTree, ship defaults in /usr/etc/ and then copy them over to /etc/ on instantiation.

No, actually the configuration is merged across updates too. It means you get new default config files. To me, this is a key aspect that keeps ostree based systems feeling like a Unix system. First, running vi /etc/fstab will continue to Just Work. If you only instantiate /etc on firstboot/install time, then your system has hysteresis - its state depends on the installation version. Such a model I think can work at a small scale for custom, targeted operating systems - but for general purpose scale, people who maintain software that installs files in /etc are just going to say "it's your problem" if it doesn't work for them to add new files in /etc over time.

So far, initrds have always been built locally

I think you mean for "package native" systems - several other image based update systems (including rpm-ostree and I'm pretty sure rauc have been doing it for a long time. You are right about the integrity aspect though.

The projects represented at the summit can be divided in three camps: image-based deployments, ostree-based deployments, and Btrfs-based deployments.

Hmm, I still think of ostree as an image system. I'd tend to say "block device image" perhaps versus "file based"? I would imagine that the people doing btrfs snapshots also think of them as image systems? In the end, the filesystem tree you're booting into is exactly reproducing the server, plus support for offline updates. To me those are two key differentiators versus the opposite end of the spectrum - traditional package based systems.

Finally, one thing I'd really hoped that could be discussed, but I'm not seeing here is a common push for filesystem compatibility changes across both distribution and 3rd party software. Specifically around things like installing files to /opt and best practices around building image systems and where state is stored - look at all the stuff in e.g. https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-transactional-updates.html and e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1900691
I can maybe see about trying to write some sort of docs for this there also in the uapi group? I think this is relevant for e.g. systemd-sysext too. Ah, though interesting I see sysext merges /opt too. In the ostree model that's /var/opt to be always part of machine-local state.