The PuzzleFS container filesystem

Posted Sep 26, 2023 8:05 UTC (Tue) by alexl (subscriber, #19068) [Link] (6 responses)

In addition to all the competing implementations of this I feel like this article misses one of the main points of composefs.

PuzzleFS (as well as e.g. nydus, etc) does block-chunking de-duplication, whereas composefs does entire-file de-duplication. Yes, the block-chunking approach is more efficient during download (as you may reuse blocks when files are only partially the same), and will use less disk-space.

However the entire-file dedup approach is more efficient at runtime.

This is because the page cache, and things like vm address spaces, are tied to a single inode. You can't get a single inode out of the multiple chunk files. This means the chunked approach doubles the page cache use. Once to cache the chunk, and once to cache the merged file. You also can't share page cache use between different puzzlefs mounts even if they use shared base files. With composefs, the page caching *only* happens on the base file, and is shared between any composefs mounts using the same backing file. This allows better container density, as e.g. all the glibc mmaps shared between any running containers are the same in the page cache.

I guess an optimal system would use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

The PuzzleFS container filesystem

Posted Sep 26, 2023 8:18 UTC (Tue) by hsiangkao (guest, #123981) [Link] (1 responses)

> use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

Yes, yet in that way, there are more effective backup/restore technologies such as delta compression (frequently discussed in several academic conferences such as ATC) to reduce I/Os than just do content-defined chunking. And not necessary in a real filesystem form to keep such delta compression archival format.

The PuzzleFS container filesystem

Posted Sep 26, 2023 13:52 UTC (Tue) by alexl (subscriber, #19068) [Link]

True, if you don't need the kernel-side to support it you can use arbitrary complex delta approaches to optimize the download.

Also, you can use filesystem level compression (in e.g. btrfs) to compress the files in the backing dir. Or even use reflinks copy create backing files that share some (but not all) blocks.

The sky is the limit, and all these approaches are compatible with using composefs to mount them.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:47 UTC (Tue) by tych0 (subscriber, #105844) [Link] (1 responses)

> You also can't share page cache use between different puzzlefs mounts even if they use shared base files.

This is a great point. I guess it could be worked around with some core mm+fs fu, but it would definitely not be simple.

> I guess an optimal system would use a block-chunked approach for downloads, but expand into full-file dedup in the local storage.

It was an explicit design goal of PuzzleFS not to have any translation step between the image that is pushed to the container registry and the one that's mounted+run on the host, so any translation step here would be a non-starter, which seems like you need to pick "one or the other" unfortunately.

The PuzzleFS container filesystem

Posted Sep 27, 2023 19:01 UTC (Wed) by calumapplepie (guest, #143655) [Link]

>> You also can't share page cache use between different puzzlefs mounts even if they use shared base files.
> This is a great point. I guess it could be worked around with some core mm+fs fu, but it would definitely not be simple.

Could KSM be extended to work on file-backed pages?

The PuzzleFS container filesystem

Posted Sep 26, 2023 16:11 UTC (Tue) by walters (subscriber, #7396) [Link] (1 responses)

In theory too, one could have a hybrid system that operates I think a lot like git does with pack files (deduplicated files w/deltas) that can be dynamically materialized the first time they're referenced. Or for files that you *know* will be shared or are "hot", materialize them automatically. This would require FUSE userspace today though. It's an interesting topic because for the in-kernel puzzlefs implementation you'd still probably (?) want a userspace system in control of optimizing things.

The PuzzleFS container filesystem

Posted Sep 26, 2023 20:28 UTC (Tue) by rcampos (subscriber, #59737) [Link]

The minute FUSE is involved, you lost any kind of performance game.

There were some patches to FUSE to allow a passthrough in some cases, that is what I wish is used to do this in a reasonable way.

The PuzzleFS container filesystem

Posted Sep 26, 2023 12:45 UTC (Tue) by bluca (subscriber, #118303) [Link] (10 responses)

Are there any plans to bring file-based runtime dedup to EROFS? Being able to combine that with dm-verity would be awesome

The PuzzleFS container filesystem

Posted Sep 26, 2023 13:40 UTC (Tue) by hsiangkao (guest, #123981) [Link] (9 responses)

> Are there any plans to bring file-based runtime dedup to EROFS? Being able to combine that with dm-verity would be awesome

Hi! Using EROFS + dm-verity + blockdevice + overlayfs (composefs-like model), you could already implement a clean file-based runtime dedup by using overlayfs.

I understand that you mean EROFS self-contained file-based runtime dedupe across images as you mentioned in some previous article. Actually we already have an internal version for internal products by Jingbo, but it's still unclean for now. We will try to clean up and post it to fsdevel mailing list later, but not quite sure it could land smoothly, anyway.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:17 UTC (Tue) by bluca (subscriber, #118303) [Link] (8 responses)

Yes I mean directly in EROFS, that would be great to have and we'd use it immediately - unless I'm missing something, you can't really do this together with dm-verity in the overlayfs model, as the data storage is not actually in dm-verity, but it's in the blob storage layer?

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:30 UTC (Tue) by alexl (subscriber, #19068) [Link] (3 responses)

You can use dm-verity to protect a composefs-style erofs image, but you would have to use fs-verity to verity the blob storage layer.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:52 UTC (Tue) by hsiangkao (guest, #123981) [Link] (2 responses)

> you would have to use fs-verity to verity the blob storage layer

I think if some blob storage layer is just an EROFS image, you could directly apply dm-verity on these layers.
And check dm-verity root digests of these layers before mounting. I think that would be in the same effect, anyway.

We can also use fs-verity to verity the blob storage layers if these layers are on a RW fs (the current composefs does.)

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:02 UTC (Tue) by alexl (subscriber, #19068) [Link] (1 responses)

Yes, this is doable, but its kinda a weird way to store the blobs. The main goal of a system like this is to share the blobs between different images, and having the blob store be per-image is contrary to this goal.

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:09 UTC (Tue) by hsiangkao (guest, #123981) [Link]

Yes, I know that is not composefs intended use cases.
I'm not sure if some people really rely on layering concept (such as system using raw partitions/devices without real filesystem storage), anyway.

The PuzzleFS container filesystem

Posted Sep 26, 2023 14:35 UTC (Tue) by hsiangkao (guest, #123981) [Link] (3 responses)

Not quite sure if I got the point. With the current upstream overlayfs, you could actually make
- EROFS + dm-verity block device blobs as data only layers;
- a small overlayfs meta layer (with EROFS + dm-verity) to merge these data storage blobs into a merged rootfs.
Thus all layers are under dm-verity protection, so the whole image won't be tampered.

Alternatively, as an EROFS self-containerd approach, EROFS could share page cache if files with same data across images without relying on overlayfs, anyway.

The PuzzleFS container filesystem

Posted Sep 26, 2023 15:57 UTC (Tue) by bluca (subscriber, #118303) [Link] (2 responses)

How would that work in practice? Say I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

The PuzzleFS container filesystem

Posted Sep 26, 2023 16:08 UTC (Tue) by hsiangkao (guest, #123981) [Link] (1 responses)

> I have an erofs image with a rootfs that contains /usr/foo/a, and other two extension erofs images that contain usr/foo/b and usr/foo/c respectively. I create two overlays, each with the base, and one of the extension, so that one has usr/foo/a+usr/foo/b and the other usr/foo/a+usr/foo/c. Is memory being deduplicated, given usr/foo/a is the same?

In that case, memory of /usr/foo/a is deduplicated according to how overlayfs works since /usr/foo/a is on the same EROFS instance.
That already works without any extra built-in feature.

The PuzzleFS container filesystem

Posted Sep 26, 2023 16:57 UTC (Tue) by bluca (subscriber, #118303) [Link]

Ah, that's very nice, I didn't know that, thanks!