OCI is an antiquated format, not fit for modern security requirements

Well, in 2019 and earlier hardly anybody had heard of UKIs, so I very much doubt they were actually in use anywhere :-) It's not that there was a viable alternative, it's simply that there was less security and several threat models were left completely unaddressed. 2019 is when we started working on Az Boost which is where these threat models had to be addressed, and that's why that feature was added.
There is just no version of any userspace solution that fixes those threat models. One can try and imagine any creative setups with LSMs or what not, to try and create some 'super special trustmebro' userspace daemon that is supposed to be unhackable, and then this boi shows up and sends it all tumbling down the drain: PTRACE

Security decisions need to be made by more privileged components than the ones being checked. This is not a matter of implementations or workarounds or solutions, it's a design pattern. If you have the same privilege level as the thing checking if you have privileges, you _will_ find ways to subvert it.

For example, on Windows these days security policies are implemented by a completely different kernel, running at a higher privilege level than your OS's kernel, with hard security boundaries enforced by HyperV. Our org is working to bring this to Linux: https://www.youtube.com/watch?v=vmt4wlf3a1A
The direction of travel is the opposite of "just check it in userspace", and one day (TM) the dm-verity signature will be checked by a higher-privileged kernel, instead of the host kernel, so that an entire new class of threat models can be closed off too.

Another example: it's the entire reason TPMs are separate enclaves, with hardware-enforced boundaries. You don't just have a TPM userspace process that pinky swears never to leak your key, because that would not be a sensible design. Nobody in their right mind would suggest that just running swtpm is a viable alternative for production usage on a secure host, or they'd be laughed out of the room.

That's why I keep saying that Linux is hopelessly behind Windows/OSX. Because it is. And crufty old stuff like OCI, that have cemented in the ecosystem an absolutely terrible image format (tarballs! What is this, 1982?), is a very large part of why this is the case, as projects like yours (for no faults of your own or your colleagues! You have to work with what is there, and I don't envy you one bit :-) ) are forced to do somersaults through flaming hoops to try and somewhat patch the leaky bucket, because god forbid docker switches to a sensible image format that's fit for purpose.

> Because (depending on how it's being implemented to a degree, I'd be curious to a link to the code) it would have the ecosystem splitting problem, and composefs is inherently going to be more efficient by sharing page and disk automatically across images.

But that's again a shortcoming of OCI, being the terrible format that it is. And it doesn't affect Flatpak, because the Flatpak devs made a very clever and sensible decision to separate the runtimes from the apps, and the app developer doesn't supply the runtime, it chooses one. So deduplication happens at the runtime level. OCI doesn't have anything like that, because it's a binfire of an ecosystem. For our use case in Boost we copied this design, and the runtime is shared and developers don't bring their own, and we get the best of both worlds: strong integrity protection that's not currently possible otherwise, and file/page level sharing of DSOs. Once again this is not a problem that composefs created or can solve, it's just inherited from OCI, and has to find ways to work around it.