LWN: Comments on "The PuzzleFS container filesystem"

The PuzzleFS container filesystem

calumapplepie — Wed, 27 Sep 2023 19:01:57 +0000

>> You also can't share page cache use between different puzzlefs mounts even if they use shared base files.
> This is a great point. I guess it could be worked around with some core mm+fs fu, but it would definitely not be simple.

Could KSM be extended to work on file-backed pages?

The PuzzleFS container filesystem

rcampos — Tue, 26 Sep 2023 20:28:33 +0000

The minute FUSE is involved, you lost any kind of performance game.

There were some patches to FUSE to allow a passthrough in some cases, that is what I wish is used to do this in a reasonable way.

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 16:57:13 +0000

Ah, that's very nice, I didn't know that, thanks!

The PuzzleFS container filesystem

walters — Tue, 26 Sep 2023 16:11:07 +0000

In theory too, one could have a hybrid system that operates I think a lot like git does with pack files (deduplicated files w/deltas) that can be dynamically materialized the first time they're referenced. Or for files that you *know* will be shared or are "hot", materialize them automatically. This would require FUSE userspace today though. It's an interesting topic because for the in-kernel puzzlefs implementation you'd still probably (?) want a userspace system in control of optimizing things.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 16:08:42 +0000

> I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

In that case, memory of /usr/foo/a is deduplicated according to how overlayfs works since /usr/foo/a is on the same EROFS instance.
That already works without any extra built-in feature.

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 15:57:03 +0000

How would that work in practice? Say I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 15:09:06 +0000

Yes, I know that is not composefs intended use cases.
I'm not sure if some people really rely on layering concept (such as system using raw partitions/devices without real filesystem storage), anyway.

The PuzzleFS container filesystem

alexl — Tue, 26 Sep 2023 15:02:05 +0000

Yes, this is doable, but its kinda a weird way to store the blobs. The main goal of a system like this is to share the blobs between different images, and having the blob store be per-image is contrary to this goal.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 14:52:45 +0000

> you would have to use fs-verity to verity the blob storage layer

I think if some blob storage layer is just an EROFS image, you could directly apply dm-verity on these layers.
And check dm-verity root digests of these layers before mounting. I think that would be in the same effect, anyway.

We can also use fs-verity to verity the blob storage layers if these layers are on a RW fs (the current composefs does.)

The PuzzleFS container filesystem

tych0 — Tue, 26 Sep 2023 14:47:13 +0000

> You also can't share page cache use between different puzzlefs mounts even if they use shared base files.

This is a great point. I guess it could be worked around with some core mm+fs fu, but it would definitely not be simple.

> I guess an optimal system would use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

It was an explicit design goal of PuzzleFS not to have any translation step between the image that is pushed to the container registry and the one that's mounted+run on the host, so any translation step here would be a non-starter, which seems like you need to pick "one or the other" unfortunately.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 14:35:50 +0000

Not quite sure if I got the point. With the current upstream overlayfs, you could actually make
- EROFS + dm-verity block device blobs as data only layers;
- a small overlayfs meta layer (with EROFS + dm-verity) to merge these data storage blobs into a merged rootfs.
Thus all layers are under dm-verity protection, so the whole image won't be tampered.

Alternatively, as an EROFS self-containerd approach, EROFS could share page cache if files with same data across images without relying on overlayfs, anyway.

The PuzzleFS container filesystem

alexl — Tue, 26 Sep 2023 14:30:07 +0000

You can use dm-verity to protect a composefs-style erofs image, but you would have to use fs-verity to verity the blob storage layer.

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 14:17:57 +0000

Congrats on the release!

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 14:17:00 +0000

Yes I mean directly in EROFS, that would be great to have and we'd use it immediately - unless I'm missing something, you can't really do this together with dm-verity in the overlayfs model, as the data storage is not actually in dm-verity, but it's in the blob storage layer?

The PuzzleFS container filesystem

alexl — Tue, 26 Sep 2023 13:52:29 +0000

True, if you don't need the kernel-side to support it you can use arbitrary complex delta approaches to optimize the download.

Also, you can use filesystem level compression (in e.g. btrfs) to compress the files in the backing dir. Or even use reflinks copy create backing files that share some (but not all) blocks.

The sky is the limit, and all these approaches are compatible with using composefs to mount them.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 13:40:58 +0000

> Are there any plans to bring file-based runtime dedup to EROFS? Being able to combine that with dm-verity would be awesome

Hi! Using EROFS + dm-verity + blockdevice + overlayfs (composefs-like model), you could already implement a clean file-based runtime dedup by using overlayfs.

I understand that you mean EROFS self-contained file-based runtime dedupe across images as you mentioned in some previous article. Actually we already have an internal version for internal products by Jingbo, but it's still unclean for now. We will try to clean up and post it to fsdevel mailing list later, but not quite sure it could land smoothly, anyway.

The PuzzleFS container filesystem

alexl — Tue, 26 Sep 2023 13:38:19 +0000

In fact, we just released 1.0 as all the required pieces are merged into the upstream kernel:
https://blogs.gnome.org/alexl/2023/09/26/announcing-compo...

The PuzzleFS container filesystem

walters — Tue, 26 Sep 2023 13:04:24 +0000

> Composefs has run into difficulties getting into the mainline kernel

Not at all - the decision was that the *ingredients* for composefs (mainly overlayfs extensions) would be in the kernel, and "composefs" would be a userspace concept wiring those ingredients together - and this has all happened!

Please see the README.md for https://github.com/containers/composefs

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 12:45:10 +0000

Are there any plans to bring file-based runtime dedup to EROFS? Being able to combine that with dm-verity would be awesome

The PuzzleFS container filesystem

smcv — Tue, 26 Sep 2023 12:45:05 +0000

There is if you download 10 slightly different versions of it - they'll all have the exact same binary for anything that didn't get a security update during the time window covered by those versions (for instance if glibc didn't get updated for a while, they'll all have identical glibc).

The PuzzleFS container filesystem

jezuch — Tue, 26 Sep 2023 12:28:44 +0000

My goodness, is there so much duplication in the Ubuntu image?

The PuzzleFS container filesystem

bluca — Tue, 26 Sep 2023 10:02:12 +0000

No dm-verity, no party!

The PuzzleFS container filesystem

roc — Tue, 26 Sep 2023 08:39:21 +0000

Capnproto would be an interesting thing to have in the kernel. We use it in rr to specify the trace format and manage compatibility as we add things to the format over time. It's worked really well for us.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 08:18:19 +0000

> use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

Yes, yet in that way, there are more effective backup/restore technologies such as delta compression (frequently discussed in several academic conferences such as ATC) to reduce I/Os than just do content-defined chunking. And not necessary in a real filesystem form to keep such delta compression archival format.

The PuzzleFS container filesystem

alexl — Tue, 26 Sep 2023 08:05:13 +0000

In addition to all the competing implementations of this I feel like this article misses one of the main points of composefs.

PuzzleFS (as well as e.g. nydus, etc) does block-chunking de-duplication, whereas composefs does entire-file de-duplication. Yes, the block-chunking approach is more efficient during download (as you may reuse blocks when files are only partially the same), and will use less disk-space.

However the entire-file dedup approach is more efficient at runtime.

This is because the page cache, and things like vm address spaces, are tied to a single inode. You can't get a single inode out of the multiple chunk files. This means the chunked approach doubles the page cache use. Once to cache the chunk, and once to cache the merged file. You also can't share page cache use between different puzzlefs mounts even if they use shared base files. With composefs, the page caching *only* happens on the base file, and is shared between any composefs mounts using the same backing file. This allows better container density, as e.g. all the glibc mmaps shared between any running containers are the same in the page cache.

I guess an optimal system would use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

The PuzzleFS container filesystem

hsiangkao — Tue, 26 Sep 2023 07:31:16 +0000

CDC isn't a new idea and typically used for backup archival programs (such as casync[1]). For kernel filesystems, I guess they are designed more for performance (otherwise FUSE approaches are enough). If data isn't encoded (e.g. compression) and metadata overhead of CDC approaches is sensitive to them (as mentioned in the FastCDC paper [2]), I think they will just use fixed-size block deduplication because of I/O friendly and no extra copy (out of raw block I/Os).

Side note is that EROFS already supports a varient CDC with compression since Linux v6.1. It would be nice to make a comparsion too in some extent so I could know what people care about more for our further enhancements. I know people enjoy new stuffs all the time but it would be nice to get more hints why state-of-art projects don't work well except for Rust adaption.

Similar to Christian said before [3], AFAIK, there were already four container image approaches raised for Linux kernel in addition to Composefs:
1. original Nydus [4], also called Dragonfly Image service in OCIv2 brainstorm [5] --- Actually they already had an in-kernel implementation of their RAFS v5 before I joined Alibaba Cloud, if needed, they could post the original kernel implementation at that time to the mailing list;
2. TarFS [6];
3. Overlaybd [7] --- Another QCOW2-like block driver approach from another team in Alibaba Cloud;
4. Puzzlefs.
I'm not sure if Linux could accept all of them for the same use case, anyway.

[1] https://0pointer.net/blog/casync-a-tool-for-distributing-...
[2] https://www.usenix.org/system/files/conference/atc16/atc1...
[3] https://lore.kernel.org/r/20230609-nachrangig-handwagen-3...
[4] https://github.com/opencontainers/.github/blob/master/mee...
https://github.com/dragonflyoss/image-service/blob/master...
[5] https://hackmd.io/@cyphar/ociv2-brainstorm
[6] https://github.com/kata-containers/kata-containers/pull/7106
https://kangrejos.com/2023/Read-only%20FS%20abstractions%...
[7] https://lore.kernel.org/r/9505927dabc3b6695d62dfe1be371b1...

The PuzzleFS container filesystem

josh — Tue, 26 Sep 2023 05:49:01 +0000

PuzzleFS seems really useful!

I'm wondering how easily PuzzleFS could build atop a remote object store rather than a local one?