Merkle trees and build systems [LWN.net]

Merkle trees and build systems

Posted May 31, 2020 22:41 UTC (Sun) by Cyberax (✭ supporter ✭, #52523)
Parent article: Merkle trees and build systems

I'm very intrigued by OSTree. Is there any work to make it possible to build OCI container images using it?

I'm using Docker (just like pretty much everybody these days) and I'm really disgusted by Dockerfiles. It would be nice to replace them with something better. It's already fairly easy to do by simply tar-ing the target image and importing it, but this loses the "layer" structure of Dockerfiles and negates all the caching advantages. It looks like OSTree can be a perfect fit there.

to post comments

Merkle trees and build systems

Posted Jun 6, 2020 5:58 UTC (Sat) by rgh (guest, #13511) [Link] (11 responses)

I don't know if there's anything based on OSTree but as a practical solution have a look at Packer from Hashicorp (https://packer.io). It's sole purpose is to build images and it does it really, really well.

Merkle trees and build systems

Posted Jun 6, 2020 6:08 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

I'm aware of Packer, but it's a bad solution. It doesn't produce "layered" images, so you're stuck with giant tar files.

For Docker caching to properly work, you basically need to do content-based addressing for its layers. I'm actually looking at OSTree and it seems eminently doable, I might actually take a stab at it.

Merkle trees and build systems

Posted Jun 7, 2020 12:10 UTC (Sun) by mathstuf (subscriber, #69389) [Link] (9 responses)

I wonder if anything ever came of this: <https://www.youtube.com/watch?v=bbTxdzbjv7I>. Sorry, I don't have a repo link for it.

Merkle trees and build systems

Posted Jun 8, 2020 3:38 UTC (Mon) by pabs (subscriber, #43278) [Link] (3 responses)

This talk reminds me of how modern backup systems like restic and borg store filesystems; similar to git but without the commit hash chain (just independent snapshots) and with an additional layer of splitting files into chunks using rolling hashes.

Sadly the restic storage design misses out splitting directories into chunks of filenames, which means that there is some inefficiency around directories with many files in them.

I wonder when git is going to adopt the file chunking stuff.

Merkle trees and build systems

Posted Jun 8, 2020 4:07 UTC (Mon) by pabs (subscriber, #43278) [Link]

Ah, I see why it seemed familiar, the speaker mentions (31:40) that he stole most of the design for OCIv2 from restic.

Merkle trees and build systems

Posted Jun 8, 2020 12:19 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

Are they always stored as a single object then I assume? I wonder if statistics on how large directory blobs are in a repository could be made. I doubt they tend to approach normal chunk sizes often which means that, statistically, you're unlikely to find a chunk boundary in a directory blob in the first place.

Merkle trees and build systems

Posted Jun 8, 2020 12:30 UTC (Mon) by pabs (subscriber, #43278) [Link]

Some details are in this bug:

https://github.com/restic/restic/issues/2446

Merkle trees and build systems

Posted Jun 8, 2020 3:44 UTC (Mon) by pabs (subscriber, #43278) [Link]

Website and git for the project:

https://umo.ci/ https://github.com/openSUSE/umoci

Merkle trees and build systems

Posted Jun 17, 2020 1:02 UTC (Wed) by cyphar (subscriber, #110703) [Link] (3 responses)

We are currently going through a more formalised specification process to hopefully get a properly specified version of the scheme I outlined in my talk. While the final scheme might not be the same as the one I outlined (which should be unsurprising given I hacked it together pretty last-minute), the general design should be similar. Unfortunately it will certainly be some time before we can point to production users of such a system.

Merkle trees and build systems

Posted Jun 17, 2020 1:54 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

All the ideas there are great!

But will Docker (or Moby or whatever they'll be called in a week) implement them?

Merkle trees and build systems

Posted Jun 17, 2020 5:28 UTC (Wed) by cyphar (subscriber, #110703) [Link]

Well, Docker has *finally* implemented OCI image support (which just boiled down to supporting the metadata blobs, since the layer blobs were designed to be identical between the Docker v2.2 image format and the OCIv1 image specification). So it's entirely possible they'll support the extensions we're working on, but I wouldn't hold out hope that it would be a quick transition. containerd and podman/cri-o will probably pick them up faster (though I think podman/cri-o will require more extensive changes to their storage internals since they're based on Docker's graphdriver code).

Merkle trees and build systems

Posted Jun 17, 2020 11:36 UTC (Wed) by pabs (subscriber, #43278) [Link]

This seems kind of similar to ostree, have you considered just using that?

Merkle trees and build systems

Posted Jun 9, 2020 3:16 UTC (Tue) by bergwolf (guest, #55931) [Link] (1 responses)

> I'm really disgusted by Dockerfiles

Could you elaborate a bit why you dislike Dockerfiles?

Merkle trees and build systems

Posted Jun 9, 2020 3:27 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

Mostly because they _pretend_ to be declarative descriptions of the resulting image, while introducing subtle non-reproducible bugs.

Just take a typical Dockerfile from Github: https://github.com/wurstmeister/kafka-docker/blob/master/... - this is random example from using their code search function.

You can see that it does: "apk add --no-cache bash curl jq docker" - basically installs the most recent available version of packages, without any notion of "lockfiles".