LWN: Comments on "An update on compliance for containers"

An update on compliance for containers

rlhamil — Sun, 28 Apr 2019 14:03:52 +0000

Maybe I didn't see it, but it seems that in addition to including the licensing info in a retrievable form within a container file, a record of the tools (or a reserved flag for manual entry) used to gather and install the licensing info should also be stored; and finally, the whole container should be signed by its distributor, so that there can be high confidence in its legitimacy. The latter would also encourage a container provider to do a more responsible for doing a reasonably competent AND compliant job of creating the container, since they couldn't so easily claim they didn't do it/didn't know.

And of course, version all this stuff, 'cause it's likely to evolve.

An update on compliance for containers: but they are meant to be abandonware

nilsmeyer — Thu, 25 Apr 2019 08:05:37 +0000

I think pretty much both cases happen. I especially see inexperienced people get it wrong very often, mostly since they don't know any better or pick the path of least resistance, basically conditioned to mostly copy & paste from tutorials, stack overflow, blog articles etc. These are professionals in the sense that they're being paid for their work.

It's very easy to get a base setup for container infrastructure running, you can get all the script and automation to pull up infrastructure very quickly, however there is a very steep learning curve to actually understanding everything you have just deployed. Also, there is a very strong lack of policy and enforcement of policies because that's usually not an area that drives delivery of features. As long as the cost-benefit doesn't shift here many organizations and individuals will continue to wing it and think everything is alright because nothing ever actually happens or if something happens nobody is held accountable.

An update on compliance for containers: but they are meant to be abandonware

rahvin — Tue, 23 Apr 2019 22:48:40 +0000

That is not accurate at all.

Containers are used heavily for services with highly variable loads where they can spin up additional nodes quickly and easily. Most of the major tech companies including AWS use containers extensively behind the scenes. They are literally at the forefront of web deployments these days with all the major providers using them on their cloud platforms. People using these things professionally are just building their own containers (so they control the product) instead of downloading and doing "fire and forget" on some random crap they downloaded from a user built repository.

An update on compliance for containers

rahvin — Tue, 23 Apr 2019 22:43:27 +0000

As mentioned it's far more than ITAR based items. Each country has export restrictions, in the US there is a list of countries that you can't export certain things to, including some that you can't export anything to. If you make it available to those countries it's a federal felony and if they ever come after people the penalty is going to be severe to make an example of the first prosecutions (like they did with the banks and Chinese tech companies that had traded secretly with Iran where the banks got fined several billion apiece and the Chinese tech companies were bared from buying/using US technology).

I looked into containers for some personal stuff a couple years ago but I was boggled at how little information exists about what is in a container. Not only that but it's pretty frequent that containers build on other containers built on other containers, etc (there was a prior article where an example container was built off of something like 8 different containers). In the end there is all kinds of software of unknown versions in the container. There's no real update mechanism for all that software other than a container update and unless you track down the whole build process and duplicate it or do a full audit you often have no idea what software is even inside the container let alone what license, version or security vulnerabilities exist. They are an absolute security nightmare IMO unless you build your own container from scratch and update it yourself for any type of permanent service. Hell even a one off container could create a potential network breach into the internal network.

I think if this project can gain traction and acceptance it will actually move towards solving that problem. A manifest or JSON or something in the header that contained a list of the software and version of everything would do wonders to helping solve the security issue of "what is actually in this container", let alone the tracking issue for license and other compliance.

An update on compliance for containers

marcH — Tue, 23 Apr 2019 04:21:06 +0000

If only it were just containers... "For example, the Node.js dependency manager NPM provides access to over 750,000 packages"
https://lwn.net/Articles/777407/ Cox: Our Software Dependency Problem

> He suggested creating a "troll company" to start doing some enforcement activity.

Thanks, that was my thought exactly. Nothing ever goes beyond the prototype stage unless money's involved one way or the other. Double whammy if patent lawyers from the eastern district of Texas[*] get "re-purposed" for that.

Same approach for security: scan, name and shame. Responsible disclosure for people making some effort, zero-day for the dangerous idiots stripping version information.

(Stupid) problem solved, let's all move back to actual software engineering. Life is short.

[*] a number of them recently put out of work by the Supreme court - and Apple
https://www.theverge.com/circuitbreaker/2019/2/22/1823642...

An update on compliance for containers

jhhaller — Fri, 19 Apr 2019 20:38:27 +0000

It's actually both ITAR and embargos. While I'm a US person, my employer is headquartered elsewhere, and has three different countries which are used as the export source. We have a good handle on our software, but have to pay attention to what might be in a base container, or be added from some site. There's a lot of commonality with FOSS license term compliance, but not completely the same.

An update on compliance for containers

mathstuf — Fri, 19 Apr 2019 14:42:12 +0000

Export control (ITAR and friends) are, I think, not what is being discussed here. This is rather about the embargoes about specific nation states being a thing. ITAR and such are more about all non-US persons. I'm sure other countries have similar classifications of information.

An update on compliance for containers

mageta — Fri, 19 Apr 2019 06:15:21 +0000

That is my understanding as well (IANAL). One takeaway from my annual export training is: If an export can technically happen, it counts as export done.

An update on compliance for containers

cyphar — Thu, 18 Apr 2019 14:15:04 +0000

We have a similar tool in openSUSE (kiwi) which works in a very similar fashion (though it's all configured through XML, a sign of when it was first developed). We use it to build container images as well as other media such as ISOs and VM disk images (it also has support for a bunch of other distros, and integrates into OBS).

I completely agree that the lack of wide-spread use of such tooling really hurts the current state of container images. Distributions figured out how to track what packages are present in ISOs 20 years ago, but now with container images we have to relive the same issues but this time it's tar archives (and don't get me started about the issues with tar).

An update on compliance for containers

civodul — Thu, 18 Apr 2019 13:28:54 +0000

I have often been citing Hohndel's previous talk as an illustration of the lack of transparency that plague container images. It hurts not just licensing, but also security, reproducibility, and user freedom.

In GNU Guix we have this guix pack tool that can build container images (OCI notably). The key difference with things like Dockerfiles is that it's declarative: instead of providing a series of commands to populate your image, you list precisely the packages you want in the image.

Since the vast majority of Guix packages are bit-reproducible, since it can "travel back in time" (you can recreate the Guix of last month or last year, and from there rebuild packages it provides), and since it is now backed by Software Heritage, anyone can recreate a container image and verify that they obtain the same image, bit for bit. The container image can finally be considered a build artifact, and the real source is the Guix manifest and commit ID to use to build it.

I hope container image creation tools will converge towards such a declarative and traceable approach. That would be a significant improvement over the current state of affairs.

An update on compliance for containers

cyphar — Thu, 18 Apr 2019 11:34:55 +0000

> I suspect what you mean is they can't store a BOM _in any kind of standard way_.

Well, I said there wasn't an "easy way". There also doesn't happen to be a "standard way" but that's less of an issue if you can't even store it properly. Your suggestion appears to be to put the BOM in the layer data itself but I don't feel that's the best idea in the world -- BOM should be metadata (otherwise you're now talking about parsing the layer tar archives of an image in order to get BOM metadata). Not to mention you'd add even more magic files (.wh.* was enough of a pain, personally).

For an OCI image if you wanted to store BOM as metadata you'd need to store it in an annotation, but annotations aren't descriptors (typed pointers) in OCI which means that almost no client would actually be able to get the relevant data (and even if they could, you wouldn't get the same safety benefits of descriptors). In addition, the layering element is a problem (yet again[1]) because it means that you will have a BOM for each layer when really you'd want a BOM for the runtime rootfs you are actually going to be executing.

I believe all of this can be fixed with the right improvements to the spec, and I hope I can work on improving it in OCIv2.

[1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

An update on compliance for containers: but they are meant to be abandonware

walex — Thu, 18 Apr 2019 11:24:56 +0000

The concern about the legal and security implications of the content of container images, but it is also irrelevant: pretty much the entire business case for containers and container images is for them to be "abandonware" ("fire and forget" is an euphemism) black-boxes, to save a lot of sysadmin costs by not managing or even just maintaining them.

An update on compliance for containers

LtWorf — Thu, 18 Apr 2019 09:30:15 +0000

My company makes an "appliance" software which is basically an ubuntu-based distribution with some modifications and some proprietary software on top of it.

To sell this directly from google cloud market, google has some compliance procedures in place, so for every FOSS component that is not coming as a package in one of their supported distributions (ubuntu is supported) they require some licensing information.

For software covered by GPL or similar licenses, they require the source code to be included directly onto the image itself, so they avoid being redistributing GPL and later on not be able to provide the sources.

They do not allow any AGPL software to run on their machines.

Because we base on ubuntu and we package the internal proprietary software as .deb files, I started adding some license information as machine readable debian/copyright files, and then created a tool that scans all the packages that are not coming from the ubuntu repositories and makes a report that can be sent to google.

Of course this solution is very hand-made and requires the coypright files to be accurate in the 1st place, but it seems incorrect to say that cloud providers don't have interest in this. Being redistributors, they are the ones that are responsible for providing the sources when requested, so they do have an interest in compliance.

An update on compliance for containers

nybble41 — Wed, 17 Apr 2019 18:53:22 +0000

> ... puts the problem on the downloader, telling them not to download something if it's not legal from the source country to support downloads into the destination country ...

This is not my area of expertise, but my understanding is that merely making the information available counts as an "export" whether or not it's actually downloaded. I can't imagine that the responsibility for export control could be delegated to the downloader; if you have information subject to export controls then you must confirm that a prospective downloader can legally access that information *before* making it available to them.

An update on compliance for containers

pj — Wed, 17 Apr 2019 18:18:55 +0000

>OCI images don't provide an easy way to store a BOM directly in the image.

Container images store arbitrary files... but they can't store a BOM? I suspect what you mean is they can't store a BOM _in any kind of standard way_.

Maybe there should be the moral equivalent of a `.well-known` or `META-INF` directory that can store this kind of info? As gzipped text it should be small enough for no one to care very much. All that's needed now is to agree on a standard set of info and a place to put extra/vendor-specific info.... but there are worse ways to get to a good standard than by someone influential taking a stab at it and then listening to everyone complain about what cases their attempt doesn't work for.

An update on compliance for containers

nishak — Wed, 17 Apr 2019 18:13:19 +0000

For what it's worth, the output of Tern can provide some baseline on what type of metadata could/should be considered.

Another thing that would be useful in OCIv2 is some method of tracking the provenance of base images. Currently, that information is lost when an image is distributed.

An update on compliance for containers

jhhaller — Wed, 17 Apr 2019 17:31:50 +0000

One of the other concerns is export regulations. Docker Hub deals with these somewhat casually, which puts the problem on the downloader, telling them not to download something if it's not legal from the source country to support downloads into the destination country and/or the nationality of the people controlling the download. I doubt the is enough to keep Docker out of trouble. Export of source code is typically under fewer restrictions than binaries, at least from the US, but containers are generally binaries. It gets quite confusing when distribution servers get involved, such as Akamai. What's the source country? The place Akamai copied it from, or the place(s) where there is a cached copy, and was the process of making a cached copy an export?

An update on compliance for containers

cyphar — Wed, 17 Apr 2019 03:20:31 +0000

Having a BOM is not just useful for licenses, but also for simply knowing what packages are inside a container. Within openSUSE, OBS provides this information (we build our container images using KIWI) which provides some neat features like having automated *image* rebuilds when there is a *package* change in the dependency tree. Unfortunately this information is all done out-of-band at the moment because OCI images don't provide an easy way to store a BOM directly in the image.

However, I'm hoping that my plans for OCIv2 images would allow for an embedded BOM -- which would help solve not only the licensing problem but also the more fundamental transparency problem (what is actually in this container).