An update on compliance for containers

The inability to determine the contents of container images is a topic that annoys Dirk Hohndel. At last year's Legal and Licensing Workshop (LLW), he gave a presentation that highlighted the problem and some work he had been doing to combat it. At this year's LLW, he updated attendees on the progress that has been made and where he hopes things will go from here.

Hohndel is the chief open source officer at VMware, but his talk has little to do with his company. The topic comes out of his sense of annoyance with the problem; when he gets annoyed, he tends to do something about it, which sums up much of his career, he said. He has given his containers and compliance talk to both compliance and security people, because the problem is basically the same for both. When you ship a container image, what is actually in it and where does that code come from? That is obviously a license-compliance problem, but it is also a security concern; distributors of these images should want to know what is in them and recipients should want to as well.

When he gives his talk, the responses follow a pattern; first there is a "little embarrassed laughter", then surprise, and, finally, attendees start looking around for someone else to do something about it. He took an informal audience poll to see how many had actually created a compliance process specifically for container images. Usually he gets no hands raised, but he got a few tentative ones at LLW, which he said was the first time that had happened. Everyone is talking about containers, of course, and Kubernetes is apparently the most important technology project of this millennium, he said, but he is "deeply surprised" that the topic does not come up more often in our community.

Container images are really nothing particularly new; they are just a packaging format for software. As with Linux distribution packaging formats or those for macOS or Windows, it is matter of ensuring that all of the dependencies for the software of interest are packaged or available with it. What container systems have done is to make it all easy; they have changed the art of packaging software from an engineering thing to a point-and-click exercise. That greatly reduces the barriers to entry, but the "best practices"—he explicitly pointed out that he was using "air quotes" for that term—available online are terrible. In last year's talk, he gave examples of some of what ends up in these images by following those practices, he said.

Progress

So, Hohndel asked, one year later, "have we made any progress at all?" There is a new Linux Foundation (LF) project, Automated Compliance Tooling (ACT), that has two sponsors, one of which is his employer. The focus of the project is to jointly develop tools to help address these problems. The name "ACT" has "so much irony" that he is unable to even come up with a joke about it, he said.

The fact that it is an LF project gives ACT both visibility and credibility, which is great, but there are few organizations that are working on it, which is frustrating. He highlighted three projects that are part of ACT: Tern, which he talked about a bit last year, Quartermaster (or QMSTR), and FOSSology. He spoke with various other people about them at the recently held Open Source Leadership Summit, which is misnamed, he said, since it is really the annual LF membership meeting. For the first time, though, he got traction with other open-source projects in the compliance space; there were discussions of how to integrate the tools in various ways to make them more useful for everyone.

The container-management tooling that creates container images uses layers to build up the image. The layers are essentially tar files with some extra pieces to describe how they fit and they are designed to replace pieces from the lower layers—without users being aware that it is being done. From a usability perspective, "that is fantastic", he said, but for compliance and security, "not so much". People were interested and curious about the container-compliance problem, but no one seems to think it is an urgent problem that they need to immediately solve.

Trolling

Lawyers are risk averse, he said, whereas engineers will happily take risks, especially when they don't understand them—compliance and security fall into that category in his view. The lawyers in the community don't care about these problems because there has not been any enforcement action against container images. And the engineers say that the problem does not matter because the technology works. In order to combat that, what is needed is an enforcement threat, he said.

He suggested creating a "troll company" to start doing some enforcement activity. The "Trolltech" name suggestion from the audience was met with laughter, but shot down since it had already been used. Hohndel said that he was "kind of joking", but that "the line between a bad joke and the true reality is very very thin". He does not own any substantial copyright in the components of today's container images so he cannot simply do this on his own—sadly, no one is making Subsurface container images.

But it would be fairly easy for someone with bad intentions to go after container distributors for not complying with the licenses in their containers. For example, BusyBox and libraries under the GPLv3 with the runtime exception are in most containers. Enforcement of that sort might lead everyone to start using only permissively licensed code, which is not what we want, he said.

Tooling is the answer, he said. It can create an artifact that goes with the image, which contains all of the copyright notices as well as providing information on where to find the correct and corresponding source code for components that need it. That is what Tern is meant to do. It will dig into a container image, tear it apart, and provide an accurate assessment of what is inside it. For images based on APT and RPM components, Tern will create a bill of materials (BOM) that contains all of the different copyright notices, license information, including Software Package Data Exchange (SPDX) headers once that part gets upstream, and pointers to the source files. But it is also an open-source project that needs a lot more help. Unfortunately, "no one cares", he said.

Needs

Hohndel closed his talk with two requests for the assembled lawyers, engineers, and others interested in licensing issues who make up the attendees at LLW. First is help with the engineering projects. It is not necessary to be an LF member to contribute; these are normal open-source projects that are looking for help.

Second, he would like to get attendees' help in spreading the word about these problems. It can't only be him that is going around to conferences and talking about it. He would like to make it a topic of conversation in our community, but he would also like to get various big players in the container world to listen—and help. For example, getting Docker to take the problem seriously and for the Cloud Native Computing Foundation (CNCF), which is shepherding the Kubernetes project among others, to require that the base container images used in its projects have compliance information available.

With that, he switched to Q&A. One attendee asked about creating compliance artifacts as part of the build process, rather than after the fact. Container images should be compliant at the point they are pushed to image repositories, they said. Tern can do both, Hohndel said; it can build up its information during the creation of the image or later. It is significantly harder to do that after the fact, however.

Karen Sandler of the Software Freedom Conservancy (SFC) said that her organization does care about this problem. She recommended that people report non-compliant images to SFC. There is no need to be a rights-holder in the code involved to make a report, she said.

Another attendee said that the information should somehow be made part of the container image or, at least, accompany the image. This is one of the things that really irritates him, Hohndel said. Information about licenses was proposed for Dockerfiles at one point, but it was removed. He has been thinking that the universal identifiers created by Software Heritage would be a great way to link to the source code, but there is no way to get those into the binary image file. He believes that sidecar files are not a viable solution for end users, because those files have a tendency to get lost along the way. He also noted that there are some container-tools vendors that are touting the ability to strip "extra" information (e.g. package information) from images in order to reduce their size.

Container people are interested in scanning images for security vulnerabilities, another attendee said. They wondered if it might make sense to team up with a project like Clair. Hohndel said that container folks are interested in the BOM, but not the copyright and source code information. Tern is modular, though, so it is easy to add scanning target types; once it was able to do some of the scanning, some of the security projects started talking with Tern developers.

Engaging the major cloud players in helping to enforce compliance on the images that their customers run was another question. Hohndel said that he has talked with people from some of the cloud providers (e.g. Amazon and Google) to try to enlist their aid, but there was little enthusiasm there. When you are in the business of selling compute resources to customers, it is not in your financial interest to make that harder, he said.

The final query was about Alpine Linux, which is used as the base distribution for many container images. The attendee noted that Alpine strips all of the license files from its layer. Hohndel said that it is worse than that, in that Alpine strips all information about versions, except those actually stored in the individual binaries, from its layers. It makes it essentially impossible to comply with the licenses of the packages that it uses (e.g. BusyBox). Another attendee said that they have some contacts at Alpine and suggested that might be a path toward resolving the problem. Hohndel closed by saying that making that contact meant that his presentation was a complete success for him.

[I would like to thank the FSFE and the LLW Diamond sponsors, Intel, the Linux Foundation, and Red Hat, for their travel assistance to Barcelona for the conference.]