LWN: Comments on "Adding fs-verity support for Fedora 36?"

Adding fs-verity support for Fedora 36?

Klavs — Mon, 03 Jan 2022 12:27:17 +0000

TPM is the "closest attempt" - but again - if its a personal user device - I strongly dislike "protecting" against the user being able to exercise their rights

Adding fs-verity support for Fedora 36?

Klavs — Mon, 03 Jan 2022 12:26:07 +0000

you cannot in fact protect against physical attacks IMHO - and it sounds unlikely anyone would even bother trying for a "personal user"-device - and also - trying to do physical protection, sounds more like you're trying to build a tivo device - ie. one thats protected against the owner actually using his rights to modify the software as he sees fit.. - not a good use case in my book.

Adding fs-verity support for Fedora 36?

zuki — Mon, 20 Dec 2021 12:14:43 +0000

> Adding keys is a much noisier operation than swapping out binaries with modified versions.

The point I was trying to make is that partial protection is very easy to circumvent without anyone noticing. My Fedora installation is 800k *files* under /usr, and 4500 files under /etc. With so many files, it's just too easy to find thousands of files that can be modified to hide something nefarious without anyone ever noticing. Signing keys may be not be the best option for stealth, but they are an obvious example of how you can defeat the system if you can modify arbitrary files.

Adding fs-verity support for Fedora 36?

LtWorf — Fri, 17 Dec 2021 07:07:15 +0000

But this doesn't protect user data?

Isn't it better (actually safe instead of kinda safe in a few selected scenarios) to use a usb stick to boot and then an encrypted partition?

Adding fs-verity support for Fedora 36?

walters — Thu, 16 Dec 2021 23:16:58 +0000

(upstream ostree developer here btw)

One big problem with ostree and fs-verity that I was wrestling with is that by design, the data format is not extensible - extensibility is hard to mix with cryptographic checksums and reproducibility. There isn't a place to put new per-file data in the current design; at least, not without doing some hacks.

That said we have a big initiative going on recently around "container native" ostree where we encapsulate the updates inside a container image - see https://fedoraproject.org/wiki/Changes/OstreeNativeContainer - and the tar format is much more flexible (which is both a benefit and a drawback, see e.g. https://github.com/vbatts/tar-split )

But basically I've been thinking it could make sense to embed fs-verity Merkle trees and signatures in the tar stream, and have the client know how to write those natively.

Adding fs-verity support for Fedora 36?

mebrown — Thu, 16 Dec 2021 17:48:18 +0000

Correct. The merkle trees and signatures would all be computed and signed by the build system in a secure manner.

It looks to me like the RPM mechanism in discussion for Fedora actually 'ships' only the signature with the RPM, and the merkle trees are re-constructed at installation time. We are likely to do something similar, but are still in early design phase so that's not coded up yet.

Personally, I am a big fan of the "ship the signatures, reconstruct the merkle trees on install" method. We did something similar in the past with DM-Verity and even re-construct the verity forward error correction codes on install. Saves a fair amount of download bandwidth.

Adding fs-verity support for Fedora 36?

mebrown — Thu, 16 Dec 2021 17:43:46 +0000

The entire point of the stack (DM-Crypt+DM-Integrity) is to ensure there is no possible way to modify anything while the machine is off. This protects against several threat models: supply chain attacks where equipment is intercepted en-route and malicious firmware installed, attacks involving theft, possibly others.

Most of the government purchasing contracts require robust protection against supply chain attacks.

Adding fs-verity support for Fedora 36?

draco — Thu, 16 Dec 2021 00:34:35 +0000

Can't be turned off even if I'm directly modifying the bits on the device?

Adding fs-verity support for Fedora 36?

walters — Wed, 15 Dec 2021 21:27:00 +0000

Are you doing the ostree+fs-verity stuff out of band? This relates to https://github.com/ostreedev/ostree/pull/2269

Adding fs-verity support for Fedora 36?

bluca — Wed, 15 Dec 2021 18:22:56 +0000

fsverity is a feature flag, you cannot disable it on a filesystem once it's enabled

Adding fs-verity support for Fedora 36?

LtWorf — Wed, 15 Dec 2021 18:05:22 +0000

but… if you change the files while the machine is off… why not also disable the fs check then as well?

It seems the old problem of trying to make a root account less powerful by just adding extra steps to achieve the same things.

Adding fs-verity support for Fedora 36?

mebrown — Wed, 15 Dec 2021 16:48:02 +0000

I ship an embedded linux device. That device has to adhere to various NIST and other governmental standards regarding robustness and reliability. We are very interested in using this stack of technologies to secure our device for our users. We have built in a way for the user to take control of their device and install their own stuff to handle the folks that want to do the open source management themselves, but the default device state is that it is verifiably secure with cryptographic checks up and down the stack. We cant sell into a huge segment of the marketplace without these controls in place.

stack looks like: DM-Crypt + DM-Integrity, EXT or F2FS, FS-Verity, OSTree + Flatpack

With the above plus SELinux and all internal daemons running as non-root, we have good protection against offline attacks (think desoldering EMMC chips) as well as protection against runtime hacks.

The biggest hole I can see is we need a way to sign/seal directories to prevent executable replacement attacks.

This stack enables us to provide more features over the older technologies, so there is distinct benefit to users even if they never crack it open themselves. The old stack was squashfs+dm-verity, so fundamentally everything was read-only. With the new stack, we can enable installation of application-level updates, installation of new applicaitons, and more.

Adding fs-verity support for Fedora 36?

LtWorf — Wed, 15 Dec 2021 16:07:47 +0000

My suspicious nature is that there is a use case for this but it's nothing the community would like.

If there is not a use case I agree that it's a complete waste of time since probably use cases will require different features to be implemented.

Adding fs-verity support for Fedora 36?

gray_-_wolf — Wed, 15 Dec 2021 13:26:01 +0000

Then what *is* the actual use case for this? If someone can temper with the actual blocks on the device, what does this exactly prevent? I know that the article said that it's supposed to be just plumbing with some use case (maybe) comming later, but should the use case be first?

Adding fs-verity support for Fedora 36?

Conan_Kudo — Wed, 15 Dec 2021 11:45:28 +0000

It wouldn't work very well for them, since (if you are the administrator) you can just turn it off and do what you will.

Adding fs-verity support for Fedora 36?

atnot — Wed, 15 Dec 2021 11:26:47 +0000

> If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?

I don't think that's inherently disarming. Adding keys is a much noisier operation than swapping out binaries with modified versions. If attackers are forced to do easily detectable things to gain persistence on a system, I think that is a good thing. If you can't have a lock it's good to at least have a seal.

Adding fs-verity support for Fedora 36?

LtWorf — Wed, 15 Dec 2021 11:15:51 +0000

Seems to me the actual users of the feature would be proprietary softwares that don't want people to replace bad sections of code with NOP, edit scripts, and so on…

Adding fs-verity support for Fedora 36?

gray_-_wolf — Wed, 15 Dec 2021 09:52:10 +0000

> Not all filesystems are ZFS, with their own internal per-block checksums.

Btrfs does that as well, so there is in-kernel support for this. You don't need ZFS.

Adding fs-verity support for Fedora 36?

rahulsundaram — Wed, 15 Dec 2021 09:18:15 +0000

Atleast partly answered by the following in the linked feature proposal

https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_yo...

https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_th...

Adding fs-verity support for Fedora 36?

taladar — Wed, 15 Dec 2021 08:51:48 +0000

So what happens if we need to update the file on purpose, either via RPM, or, such as in the case of the recent log4j issues, through manual workarounds (e.g. adding a property to the shell script starting some Java application)?

Adding fs-verity support for Fedora 36?

Henning — Wed, 15 Dec 2021 02:33:17 +0000

This seems more like a job for dm-integrity and it seems like systemd 250 will support it (at least according to the release notes).

Adding fs-verity support for Fedora 36?

developer122 — Wed, 15 Dec 2021 00:15:42 +0000

The building block for larger security schemes is nice and all, but I'm not really even interested in it from the standpoint of security.

This sounds like it could also be used to ensure the integrity of a file, like system libraries for example, against random corruption. It can be automatically detected if a file has been damaged since it was installed, and I suppose in later work damaged files could be automatically re-aquired.

Not all filesystems are ZFS, with their own internal per-block checksums.