A report from the 2022 Image-Based Linux Summit

Posted Nov 3, 2022 21:26 UTC (Thu) by bluca (subscriber, #118303)
In reply to: A report from the 2022 Image-Based Linux Summit by walters
Parent article: A report from the 2022 Image-Based Linux Summit

> Hmm, I still think of ostree as an image system. I'd tend to say "block device image" perhaps versus "file based"? I would imagine that the people doing btrfs snapshots also think of them as image systems? In the end, the filesystem tree you're booting into is exactly reproducing the server, plus support for offline updates. To me those are two key differentiators versus the opposite end of the spectrum - traditional package based systems.

The key differentiator is support for full integrity protection at the block layer via dm-verity. Locally mutable systems like os-tree cannot provide that by definition. In many settings this is not a minor thing, but a fundamental design goal if not table stakes.

> Finally, one thing I'd really hoped that could be discussed, but I'm not seeing here is a common push for filesystem compatibility changes across both distribution and 3rd party software. Specifically around things like installing files to /opt and best practices around building image systems and where state is stored - look at all the stuff in e.g. https://documentation.suse.com/sles/15-SP1/html/SLES-all/... and e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1900691
I can maybe see about trying to write some sort of docs for this there also in the uapi group? I think this is relevant for e.g. systemd-sysext too. Ah, though interesting I see sysext merges /opt too. In the ostree model that's /var/opt to be always part of machine-local state.

The specs repo is on Github and contributions are welcome. There is already a fork of the XDG-Base spec for handling of configuration files (that I'm trying to get merged upstream). In general we do not want to deviate excessively from the FHS, but enhancements are good.

A report from the 2022 Image-Based Linux Summit

Posted Nov 3, 2022 21:56 UTC (Thu) by walters (subscriber, #7396) [Link] (2 responses)

> The key differentiator is support for full integrity protection at the block layer via dm-verity. Locally mutable systems like os-tree cannot provide that by definition. In many settings this is not a minor thing, but a fundamental design goal if not table stakes.

IMO, the definition of "locally mutable" gets fuzzy here when one adds things like 3rd party sysexts into the mix. There's also the opposite case of using IMA or fs-verity underneath file-based systems (like ostree, but not exclusively - the old OLPC model of "let's use rsync to hardlinked filesystem trees" was another variant). Sure, it's not as strong as dm-verity, but it absolutely will (particularly in combination with other mechanisms like dm-crypt and other LSMs) stop many attacks. My classic example is the runc exploit. https://lwn.net/Articles/842164/

But I think we're in agreement here to say that the cases are "dm-verity", "file based", and "btrfs snapshot" right? The original phrasing of excluding ostree from "image based" seems contradictory from having us under a shared discussion group. Not to mention, while rauc supports dm-verity it also seems to support non-dm-verity (e.g. plain squashfs) and I hope you'd agree that it's still an image based update system, it just doesn't have the dm-verity properties (in that setup) either.

A report from the 2022 Image-Based Linux Summit

Posted Nov 3, 2022 23:35 UTC (Thu) by bluca (subscriber, #118303) [Link] (1 responses)

Even with sysexts, there is still the single root of trust as it's signed dm-verity all the way down. ima/fs-verity do not fulfill the same role due to very important differences: they do not cover the whole filesystem hierarchy (you can create new files, delete existing files and recreate them, create directories, etc), and most important they do not cover the filesystem itself. The kernel is vulnerable to rogue filesystems, it is not a use case that it is hardened against. Signed dm-verity allows you to verify the integrity and trust of the filesystem before opening it, which is crucial for this threat model. In use cases where these properties are needed, it's very important to be fully covered: close enough is not good enough.
When adding the IPE LSM on top (https://microsoft.github.io/ipe/) this allows a pretty neat and hardened system with full code integrity (well, it's at least getting there: still working on enlightening interpreters for scripts).

Of course this does not mean that other systems like ostree and btrfs are bad or that they should not be used, it simply means that they sit in a different place on the wide spectrum of image-based Linux, with significant enough differences to be meaningfully distinguishable.

A report from the 2022 Image-Based Linux Summit

Posted Nov 4, 2022 13:42 UTC (Fri) by walters (subscriber, #7396) [Link]

> ima/fs-verity do not fulfill the same role due to very important differences: they do not cover the whole filesystem hierarchy

(Yes, I know this - and I'm pretty sure you know I know too ;) But, I know we're also conversing with an audience so some repetition is needed unfortunately)

> it simply means that they sit in a different place on the wide spectrum of image-based Linux, with significant enough differences to be meaningfully distinguishable.

Yes! We're in agreement. There's lots of nuances between different points on that spectrum. And a lot of space to share knowledge and approaches across them.

Security is also (as you know) not a binary thing. Personally, I think one of the biggest problems in our industry in general is people *not updating the operating system at all*. Switching to "the operating system auto-updates by default" was one of the big changes from the original Container Linux - and we've carried that forward in both Flatcar and Fedora CoreOS. It's actually a pretty profound difference in practice - I blogged about this but basically I think there's a strong sense of "responsibility" for updates that shifts from "I typed apt|dnf|whatever update and it broke, so it's my fault" to "the system just fell over, it's $OS's fault".

On this topic, I think "unlocked" image based systems (like pulling btrfs snapshots, snap and ostree, etc.) having the properties of e.g. transactionality and offline updates (not disrupting the running system) provide a very meaningful improvement to security from *this* aspect. Not to mention the "we tested the update image server side and you are bit for bit reproducing it".

Another way to say it is, these properties are also very meaningfully distinguishable from traditional package based systems.

What I'm arguing at the core is: s/image based/dm-verity based/ in the original sentence. I also would tend to use the term "locked" or "sealed" when talking about this because honestly "image" means too many things already.