Containers and license compliance

Containers are, of course, all the rage these days; in fact, during his 2018 Legal and Licensing Workshop (LLW) talk, Dirk Hohndel said with a grin that he hears "containers may take off". But, while containers are easy to set up and use, license compliance for containers is "incredibly hard". He has been spending "way too much time" thinking about container compliance recently and, beyond the standard "let's go shopping" solution to hard problems, has come up with some ideas. Hohndel is a longtime member of the FOSS community who is now the chief open source officer at VMware—a company that ships some container images.

He said that he would be using Docker in his examples, but he is not picking on Docker, it is just a well-known container management system. His talk is targeting those that want to ship an actual container image, rather than simply a Dockerfile that a customer would build into an image. He has heard of some trying to avoid "distributing" free and open-source software that way, but is rather skeptical of that approach.

Docker "hello, world"

So he looked at the Docker equivalent of "hello, world"; he used Debian as the base and had it run the echo command for the string "Hello LLW2018". Running it in Docker gave the string as expected, but digging around under the hood was rather eye-opening. In order to make that run, the image contained 81 separate packages, "just to say 'hi'". It contains Bash, forty different libraries of various kinds including some for C++, and so on, he said. Beyond that, there is support for SELinux and audit, so the container must be "extremely secure in how it prints 'hello world'".

In reality, most containers are far more complex, of course. For example, it is fairly common for Dockerfiles to wget a binary of gosu ("Simple Go-based setuid+setgid+setgroups+exec") to install it. This is bad from a security perspective, but worse from a compliance perspective, Hohndel said.

People do "incredibly dumb stuff" in their Dockerfiles, including adding new repositories with higher priorities than the standard distribution repositories, then doing an update. That means the standard packages might be replaced with others from elsewhere. Once again, that is a security nightmare, but it may also mean that there is no source code available and/or that the license information is missing. This is not something he made up, he said, if you look at the Docker repositories, you will see this kind of thing all over; many will just copy their Dockerfiles from elsewhere.

Even the standard practices are somewhat questionable. Specifying "debian:stable" as the base could change what gets built between two runs. Updating to the latest packages (e.g. using "apt-get update") is good for the security of the system, but it means that you may get different package versions every time you rebuild. Information on versions can be extracted from the package database on most builds, though there are "pico containers" that remove that database in order to save space—making it impossible to know what is present in the image.

It gets worse

But it gets even worse, Hohndel said. Most people start with a Dockerfile they just find somewhere. If you look at the Dockerfile for Elasticsearch, for example, it installs gosu and uses the Dockerfile for OpenJDK 8, which in turn uses other Dockerfiles. One of those is for Debian "stretch", which also updates all of the packages.

There is a "rabbit hole" that you need to follow, Dockerfile to Dockerfile, to figure out what you are actually shipping. He has done a search of official Docker images and did not find a single one that follows compliance best practices. All of the Dockerfiles grab other Dockerfiles—on and on.

No one wants to hear about these problems, Hohndel said; he has tried. He is a big fan of free software, but not really a fan of enforcement; he would rather simply fix the problems. But in order to fix these problems, people have to understand and care about compliance. He has been to KubeCon, and will be again soon, trying to educate folks about these problems. At one of the talks, he asked how many copyleft packages were in a particular Docker image, but he just got blank stares.

In the container image for an uncomplicated three-tier application, he counted 650 packages. The problem is only getting worse, he said. It is "incredibly hard" to get compliance right if it is done at build time, but it is "pretty much impossible" to do after that point. It is important to get people to understand that the complexity of what they are shipping in containers is much greater than what a few simple commands might indicate.

The problems with container images are many. It is hard to figure out which packages are included in the build. The version and which patches are applied are also difficult to determine. Beyond that, the licenses under which those packages are distributed are not obvious. He has seen containers that try to save space by statically linking various pieces that may not be linkable based on their licenses.

The tooling that the industry has developed makes it quick and easy to throw together an image. But it also, "hopefully unintentionally", makes it easy to create a "total compliance nightmare", Hohndel said.

What should be done

Telling people to stop shipping containers is not going to work, so another approach is needed. Containers need to be built starting from a base that has known-good package versions, corresponding source code, and licenses. The anti-pattern of installing stuff from random internet locations needs to be avoided. And software developers need to be trained about the pitfalls of the container build systems, which should not be hard, but is.

Any layers that will be added on top of the base need to be tracked as well. The versions, source location, and licenses should all be stored and a source-code management system should be used to track the information over time. One way to do so is to annotate the Dockerfiles with the meta information about the packages, though creating these annotations is hard, he said.

VMware has started the Tern project to help automate the creation of a bill of materials (BOM) for a container image. It will determine what packages are present in the image from the Dockerfile, but it also understands some of the commands that are used in Dockerfiles to retrieve and install packages, so it can track those too. It is a work in progress, Hohndel said, but may be helpful for container compliance.

[I would like to thank the LLW Platinum sponsors, Intel, the Linux Foundation, and Red Hat, for their travel assistance support to Barcelona for the conference.]