LWN: Comments on "Docker and the OCI container ecosystem"

Docker and the OCI container ecosystem

jond — Sat, 03 Sep 2022 07:54:05 +0000

Example image sources for cekit: https://GitHub.com/jboss-container-images/openjdk

Docker and the OCI container ecosystem

jond — Sat, 03 Sep 2022 07:30:51 +0000

I use and work on a pre-processor tool cekit - https://cekit.io - which abstracts (somewhat) over dockerfiles. It’s more declarative and lets you break the recipe up into separate modules that can be shared amongst different container sources (like mix-ins, contrast to container image single inheritance )

Docker and the OCI container ecosystem

daenzer — Fri, 19 Aug 2022 07:36:38 +0000

FWIW, the freedesktop.org GitLab CI templates (https://gitlab.freedesktop.org/freedesktop/ci-templates) use buildah, seems pretty nice.

The templates currently offer two modes of operation for generating images: A simple one where one can just specify the base system packages to be installed, and a generic one which allows running a script to set up the filesystem arbitrarily.

The templates produce a single layer, but it can be based on any other image (by default a standard image of one of the supported base distros), which is included as separate layers. This allows efficient sharing of common base contents.

Docker and the OCI container ecosystem

ricab — Thu, 18 Aug 2022 12:41:36 +0000

This is a jewel of an article!

Lack of a lightweight run time

rganesan — Mon, 08 Aug 2022 11:47:23 +0000

One thing the container ecosystem lacks is a lightweight embedded runtime. docker runs into 100s of megs with all it's dependencies and podman is not significantly lighter. Even containerd/nerdctl seems boated.Why would I want "docker build" or "docker compose" on a small IOT device? Something like k3s would me nice but without the whole k8s ecosystem.

Docker and the OCI container ecosystem

hsiangkao — Fri, 05 Aug 2022 06:04:27 +0000

Hi, you could refer to Nydus image service [1] and erofs fscache support for container images [2].
Rolling hash deduplication + fixed-sized output compression is working in progress.

[1] https://github.com/dragonflyoss/image-service/
[2] https://d7y.io/blog/2022/06/06/evolution-of-nydus/

Docker and the OCI container ecosystem

hsiangkao — Fri, 05 Aug 2022 06:01:38 +0000

you could refer to chunk-based content-addressable Nydus image service [1] and erofs filesystem fscache support for container images [2].

[1] https://github.com/dragonflyoss/image-service/
[2] https://d7y.io/blog/2022/06/06/evolution-of-nydus/

Rocket containers

rahulsundaram — Thu, 04 Aug 2022 03:53:20 +0000

> rkt was also very good docker replacement, k8s could even use it, but as as redhat aquired CoreOS, rkt was killed in favor of containerd/runc and later podman ... basically redhat killed coreos and almost all their tools :(

I don't see how. rkt didn't make the cut but plenty of things from CoreOS continues to be developed including CoreOS itself, along with quay, k8s operators, tectonic (integrated with Openshift) and so forth.

Rocket containers

higuita — Thu, 04 Aug 2022 03:37:19 +0000

rkt was also very good docker replacement, k8s could even use it, but as as redhat aquired CoreOS, rkt was killed in favor of containerd/runc and later podman ... basically redhat killed coreos and almost all their tools :(

Docker and the OCI container ecosystem

mathstuf — Mon, 01 Aug 2022 17:06:36 +0000

Headed slightly off-topic here, but I have to ask:

> Docker recently acquired the company that developed Sysbox (Nestybox) to extend containers isolation and workloads.

Why does it always seem to be "acquire" and not "fund"?

Docker and the OCI container ecosystem

ctalledo — Mon, 01 Aug 2022 17:04:10 +0000

Excellent article, thanks! On the runtimes section, one option that may be worth mentioning is Sysbox which is a runc alternative that enhances container isolation and workloads using pure OS virtualization (I am one of the developers). Sysbox enables the user-namespace on all containers, virtualizes /proc and /sys within them, traps sensitive syscalls, and enables containers to run systemd, Docker, K8s, K3s, and more seamlessly. The latter means Docker or Kubernetes can be used to deploy not just app containers, but also system containers. Docker recently acquired the company that developed Sysbox (Nestybox) to extend containers isolation and workloads.

Docker and the OCI container ecosystem

robert_s — Sun, 31 Jul 2022 19:07:21 +0000

> Do they support layering?

Answering for Nix and its dockerTools: short answer yes (https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-docker...), though half of the benefits of using "layers" in docker-land are already solved by Nix's build-caching system. So the advantages gained by "layers" are mostly around image size.

Docker and the OCI container ecosystem

mathstuf — Fri, 29 Jul 2022 13:00:23 +0000

I agree that `git-annex` is a wonderful solution (far better than `git-lfs` IMO). However, it's reliance on symlinks really sucks for Windows developers.

FWIW, my main issues with `git-lfs` include:

- doesn't reuse git's authentication mechanisms
- all data is only looked for at a single location (i.e., no multi-url remotes)
- fork-based workflows get confused when data is added and removed across some history that you rebase on (it will complain about the missing object but there's no nice way to fetch just that object so you can push without `git lfs fetch --all`)

Docker and the OCI container ecosystem

gscrivano — Fri, 29 Jul 2022 12:30:21 +0000

one thing we are trying in Podman/Buildah/CRI-O is to use a new file format (still OCI compatible) to make every file addressable, so a "podman pull" would just fetch the files that are not present locally[1]

The layering model still makes some sense with the overlay file system, since when you mount a container, the underlying file system expects that layout anyway. An attempt we are trying to improve this model and make easier to move away from layers is composefs[2]. It is a new file system that can mount the image directly and at the same time have both storage and memory page deduplication for shared files[2].

[1] https://www.redhat.com/sysadmin/faster-container-image-pulls
[2] https://github.com/containers/composefs

Docker and the OCI container ecosystem

k3ninho — Thu, 28 Jul 2022 22:47:32 +0000

I'm not sure I understand the problem space and whether this is a solution, but Joey Hess' git-annex [1] for managing large files is an overlooked but solid tool.

1: https://git-annex.branchable.com/

K3n.

Quadlet

valberg — Thu, 28 Jul 2022 09:08:44 +0000

There are plans to integrate Quadlet into the containers/podman project such that they can be released simultaneously. Quadlet and Podman can then also share the generate-systemd logic.

Starting with Podman v4.2 (to be released in early August), you can also use a systemd template to run Kubernetes YAML [1]. That simplifies the workload quite a bit as we only need to feed the YAML file to the template and systemd and Podman will take care of the rest.

[1] https://github.com/containers/podman/blob/main/docs/sourc...

Docker and the OCI container ecosystem

Cyberax — Thu, 28 Jul 2022 05:54:27 +0000

Ohhh.... So many reasons.

1. Dumb syntax. You have \
to write \
very long \
lines that blow up logs because they are actually ran as one long line.

2. The way the layering system works. All of it is kinda crappy.

3. COPY command. There's no way to copy multiple source directories in one layer.

And finally, Dockerfiles pretend to be purely functional and reproducible, but most projects have lines like "RUN apt-get blah" that immediately blow that up.

A true content-addressable system with a better build language than Dockerfiles would be great. Earthly seems to be a good incremental improvement on Dockerfiles.

Docker and the OCI container ecosystem

pabs — Thu, 28 Jul 2022 05:49:13 +0000

Where does your disgust stem from? Mine is that they are mixed-language files. I dislike Makefiles and even printf/regex/SQL for this reason too.

Docker and the OCI container ecosystem

bartoc — Thu, 28 Jul 2022 05:36:41 +0000

or git! I would _love_ to see git updated to be able to handle this stuff (support for reflinking into working tree, smarter compression heuristics, etc).

Actually what would be _really_ cool would be to actually support delta compression for some common binary file types, images could delta compress using hevc or AV1 (probably AV1), and I can imagine similar schemes for videos, although making those schemes fast is probably a research project and would be somewhat codec specific.

Ditto for executable files.

Docker and the OCI container ecosystem

bartoc — Thu, 28 Jul 2022 05:32:47 +0000

systemd actually has native support for running "container based" services called "portable services"

I think it might even be able to deal with OCI images. In this scheme all the configuration of what to start (and of cgroups) is done in the normal systemd unit files, but the files are contained in some fs image, and that fs image is also set up with all the needed dependencies and such.

The portable service basically just handles the filesystem bits of containers, the rest is handled by existing systemd mechanisms (systemd already manages cgroups for you, and can create user namespaces and such as well).

Docker and the OCI container ecosystem

bartoc — Thu, 28 Jul 2022 05:28:09 +0000

I don't find dockerfiles all that horrid as long as everyone on the team writing them actually understands how the layering works. They are reasonably simple and good enough.

I do prefer buildah's approach though.

Actually I would prefer a scheme that used a content addressable datastore instead of the layering thing, something like ostree or casync (or git, really that would be most convenient, but git has some missing features when dealing with large binary files)

Docker and the OCI container ecosystem

mathstuf — Wed, 27 Jul 2022 21:29:18 +0000

I've used hand-written `.service` files for podman for some smaller deployments (though `podman-systemd-generator` is likely better long-term) and a larger one uses `podman-compose`[1] to generate its systemd units (since they collaborate, it helps to ensure they all share a common network and set of volumes).

K8s just seemed *way* overkill for the "I have a single machine and just want it to be easy to manage upgrades of containers and the base system".

[1] https://github.com/containers/podman-compose

Docker and the OCI container ecosystem

jordan — Wed, 27 Jul 2022 20:18:41 +0000

This is an area of interest to me - the containers/storage and containers/image libraries used to contain at least some level of ostree support, but I know at least part of that was removed due to nobody using it or being willing to maintain it. I've seen other parts of it that have gotten some work at least semi-recently, though; I haven't yet explored what is currently possible there.

Podman supports <a href="https://www.redhat.com/sysadmin/image-stores-podman">additional image stores</a> - I've experimented with using this feature to store image stores in ostree repositories and then get delta updates instead of having to pull whole layers at once.

I know Balena has some sort of delta update technology for their container runtime but I'm unclear how it works.There are also things like <a href="https://github.com/containerd/stargz-snapshotter">stargz</a> (usable with containerd/nerdctrl) that do interesting things with how images are transferred over the network.

Docker and the OCI container ecosystem

jordan — Wed, 27 Jul 2022 20:11:14 +0000

Thanks! A follow-up about orchestration could be interesting but I'd have to do a lot of research. I used to run Swarm but now I mostly deploy my own stuff with Compose and have thus far managed to avoid learning very much about Kubernetes :)

Docker and the OCI container ecosystem

jordan — Wed, 27 Jul 2022 20:07:46 +0000

> CRI-O does have a cli, crictl, but you'd only use it for low level troubleshooting/debugging/etc.

I was aware of crictl but left it out of the CRI-O section because it's not specific to CRI-O; it can also be used with containerd or another CRI runtime. The article was getting pretty long and it's hard to mention everything.

> There's a whole other container ecosystem that has sprung up around Singularity.

I'm aware of that ecosystem but haven't played with it. I did give a brief shoutout to Apptainer at the end; it appears to me that that is the new name for Singularity, but I could be wrong about that!

Docker and the OCI container ecosystem

mathstuf — Wed, 27 Jul 2022 20:07:09 +0000

I agree. I don't really like Dockerfiles either, but the `COPY` and `RUN` with shell scripts really helps. It also keeps artifacts of what happened in the image itself (no point in deleting since it's in the previous layer anyways).

Other than that, I've found `buildah` to be useful for building minimal images where I can do some logic outside of the container and tell it afterwards. Much better than manual flattening after the fact IMO.

Quadlet

mathstuf — Wed, 27 Jul 2022 20:05:28 +0000

> I toyed with the idea that I could just use the podman command to generate templates I could use via ansible or whatnot, but it didn't really work out.

`quadlet` mentioned elsewhere in the comments seems like it'd work better. But can't you have Ansible run the generator command as your "sync" rather than expecting specific file contents?

Docker and the OCI container ecosystem

mathstuf — Wed, 27 Jul 2022 20:04:39 +0000

Great writeup, thanks. Has there been any movement on updating the container storage to be something smarter that does something like `restic`, `os-tree`, or `casync` that uses a rolling algorithm to break blocks into "chunks" and then allowing sharing at more than just the "layer" level? Is this just being left to different implementations to do as they see fit or is it basically just overwhelmed into irrelevance at this point?

Docker and the OCI container ecosystem

jordan — Wed, 27 Jul 2022 20:03:05 +0000

I also dimly remember it having it in the distant past and support for it being removed. Very cool that it's back!

Docker and the OCI container ecosystem

lobachevsky — Wed, 27 Jul 2022 19:42:41 +0000

Since version 242 systemd-nspawn can also run OCI bundles. I vaguely remember it having that feature in the ancient past before, before they were called OCI, but being removed because back then the format was kind of fast moving, but I might be wrong, because I can't find any note of it anymore.

Quadlet

rjones — Wed, 27 Jul 2022 19:16:52 +0000

One thing that kinda sucks about podman systemd integration, which may not be avoidable, is that the files it generates is can be very specific to the version of systemd you are using. I toyed with the idea that I could just use the podman command to generate templates I could use via ansible or whatnot, but it didn't really work out.

That was a while ago, though. Maybe it's better now.

Docker and the OCI container ecosystem

rjones — Wed, 27 Jul 2022 18:21:56 +0000

Docker files are gross, but there are ways to mitigate the grossness.

The big one I do is to simply have a shell script that does all the work of setting up the docker image. The dockerfile only copies in the script and required assets and the script does all the work of actually setting it up. Of course you can do the same thing with makefiles or scons or whatever build tool you like using. This reduces things to a single COPY and single RUN command.

Along with that it's useful to do multistage builds.

https://docs.docker.com/develop/develop-images/multistage...

This is especially nice if you want to do statically compiled binaries. You end up with a build container and a runtime container. You only then need to distribute the runtime container. This results in a very clean OCI image with all the build-time and setup portions stripped out. It is usually very easy to keep things down well under a 100MB in size.

As far as getting rid of dockerfile itself... It is probably not worth it at this stage. Everybody and their mom supports it and thus it makes it easy to integrate containers into CI/CD workflows, among other things. Sefl-hosting build farms in kubernetes is extremely easy with kaniko. You can use buildah and other associated tools related to podman. Gitlab, github, and most everything else supports building and testing against OCI images.

Docker and the OCI container ecosystem

pj — Wed, 27 Jul 2022 18:07:30 +0000

Just wanted to say thanks for the great overview! I've seen the names of lots of these pieces fly by, it's good to know how they all relate to each other . Any chance you'll do a followup on orchestration systems? there's k8s and some various micro- and mini- self-hosted versions, and then there's docker swarm... are there others? which of those are suitable for production vs toy/exploring ? Maybe there's enough there for a followup ecosystem/survey article.

Docker and the OCI container ecosystem

cortana — Wed, 27 Jul 2022 09:13:13 +0000

Great article! A couple of omissions that occurred to me:

CRI-O does have a cli, crictl, but you'd only use it for low level troubleshooting/debugging/etc.
There's a whole other container ecosystem that has sprung up around Singularity. The idea here is that we _don't_ want to isolate the code inside the container from the user's home directory, shared filesystems, etc; we just want to use containers for distributing software. You'd use this on e.g., a high performance computing cluster. Personally I think it's a bit of a shame that the community didn't coalesce around Podman for this use case, but there we are...

Docker and the OCI container ecosystem

cortana — Wed, 27 Jul 2022 09:00:58 +0000

Buildpacks is higher-level and works kind of like Heroku; it detects what language your application is written in, and then picks appropriate builder and base images. It can rebase application images on top of newer base images without rebuilding the whole application.

For another take on this approach, see Red Hat's source-to-image which invokes an assemble script at a well known location in the base container image; this script embeds all the build logic, so that the developer doesn't need to write a Dockerfile, only comply with the conventions of the base image.

Docker and the OCI container ecosystem

Cyberax — Tue, 26 Jul 2022 23:39:45 +0000

> Nix, bazel, and yocto/openembeded are some mature systems that can build container images without actually using docker or Dockerfiles - they just produce the layer tar files (OCI image) directly.

Do they support layering? This is one feature of Docker that makes it bearable, especially for iterative development.

Docker and the OCI container ecosystem

anguslees — Tue, 26 Jul 2022 23:17:38 +0000

Nix, bazel, and yocto/openembeded are some mature systems that can build container images without actually using docker or Dockerfiles - they just produce the layer tar files (OCI image) directly.

I agree there should be more tools like this - it's not that hard. The thing you lose with these approaches is any easy "bootstrap" step to set up the build system in the first place. If you put the build system in a docker container, then you get back to something like a multistage Dockerfile anyway (for simple projects - complex projects can still benefit from these more advanced systems).

Also be aware of cross compiling. In the FOSS community, we would like to support our non-intel brethren and this requires building images for platforms you don't have by default. Some systems (yocto) are good at this, others (bazel) are still maturing. Docker buildx makes this very easy through transparent emulation, again if you're willing to settle for Dockerfiles.

For me, the combination of cross compiling and easy bootstrap means I use docker buildx for all my simpler projects, and just put up with the awkward Dockerfile syntax :-(

Docker and the OCI container ecosystem

Cyberax — Tue, 26 Jul 2022 21:34:13 +0000

> Sure, but using Debian snapshots would mean that you'd have to take all the updates in the snapshot that you moved to at once

Yeah, but this is usually OK. It also makes it easier to audit dockerfiles to check if they cover all CVEs in the base Debian image.

We also have a script that checks if an image contains packages that are different between two snapshots, this helps to automate "empty" version bumps. Not perfect, but it helps.

We also tried Nix that gives strong reproducibility gurantees, but it wastes way too much time on rebuilding everything.

Docker and the OCI container ecosystem

amarao — Tue, 26 Jul 2022 21:29:08 +0000

Yes. And the main reason people confuse that (assuming that 'stop' is a 'drop') is beause it's not possible to change container (without low-level hacks). If you want to update image or tweak manifest, you need to rebuild image, and new image means new container.

Docker and the OCI container ecosystem

jordan — Tue, 26 Jul 2022 21:24:46 +0000

Sure, but using Debian snapshots would mean that you'd have to take all the updates in the snapshot that you moved to at once, and that you'd have to take updates in the order they were supplied upstream. Having a packrat mirror that holds on to all the versions gives you more flexibility in deciding what you want to update and when.