An introduction to Clear Containers
An introduction to Clear Containers
Posted May 27, 2015 1:57 UTC (Wed) by ras (subscriber, #33059)In reply to: An introduction to Clear Containers by Cyberax
Parent article: An introduction to Clear Containers
Sounds like the promised land. But it doesn't quite jar with reality. As a point of comparison, the "old way" of doing this was to install something like unattended-upgrades and let the system handle it itself. It's completely automated, with stuff all down time.
To do the same job in a container you say you rebuild it. But rebuilding is a far heaver operation - so much so that they provide tools to avoid it by persisting it as a .tar.gz. It can be done offline, but then how do you know when to do it? If you don't know you are up for rebuilding and restarting every container at least once day.
These kernel visualisation containers were born in Google. In Google I suspect none of this mattered because the software in the container was produced by them, and distributed in a container format. The rest of us run mostly software maintained by upstream distro's, distributed as packages that have to be individually installed and configured. Yes, Docker provides a bridge between the two worlds - producing container images from a distro's packages. But it damned primitive bridge. Doing of deboostrap and by a zillion apt-get install's every time you apply a security update just doesn't cut it.
We need the next step - something that marries the roles of distro and container. I suspect the next big move will have to be from the distro's. It will would allow (say) a Debian host to build a Debian container from Debian packages in a second or so, or alternatively allow the Deban host to maintain (eg, apply security patches) to all Debian containers under it's control it transparently.
Posted May 27, 2015 2:07 UTC (Wed)
by dlang (guest, #313)
[Link]
build from scratch instead of upgrading to create the new gold copy is a good idea because it means that you can't have 'quick hacks' work their way into your system that you don't find for years (until the next major upgrade when you do have to recreate them from scratch), but it is significantly more expensive to recreate the entire image from scratch than to just upgrade one piece of software.
I take the middle ground, I create a base image that has most of the system in it and just add the application specific parts when creating the per-function images.
if you only have one or two instances of each type of thing, and you are creating them completely from scratch, they it really does waste quite a bit of time (both CPU time and wall clock time)
Posted May 31, 2015 13:54 UTC (Sun)
by kleptog (subscriber, #1183)
[Link] (12 responses)
You're going to rebuild the entire image every time you make a release that needs to be deployed. You deploy all the latest OS updates at the same time so in practice it's no extra work.
Besides, we have buildbots that do nothing else than build Docker images on every commit. It takes a few minutes per image sure, but the result can be thrown into the test environment to see if it works and if it does you can use the same image to deploy to production.
I would love it if it were possible to create VMs as easily. I'm hoping someone will make a Docker-to-VM converter. Livebuild is good, but relatively slow.
Posted May 31, 2015 15:55 UTC (Sun)
by raven667 (subscriber, #5198)
[Link]
Posted May 31, 2015 22:57 UTC (Sun)
by ras (subscriber, #33059)
[Link] (10 responses)
Yes when you are deploying software you developed that makes perfect sense, and I'm guessing how it worked in the company that pioneered this technology - Google.
To me, a lone wolf, who must deploy a variety of stuff I didn't develop, it makes far less sense. I inherited a WordPress instance for example, and it's not the only one - I run many of these packages. If I tried to keep track of all the security vulnerabilities in them and all their dependencies and updated them manually, I'd have no time for anything else. The only thing that makes sense time wise for me is to reply on my distro so keep it patched. Which it does, and I'm guessing it does it more reliability than your updating your package at irregular intervals.
I suspect it's the little guys like me who are continually popping and asking "what good does this newfangled containerisation thing do for me". The answer is not much. In the short term the only real positive it brings is security. The mental model you need to reason about the isolation imposed by containers is far simpler than the alternatives.
The other observation I have is the way containerisation is done now is at odds with how the distro's work. Distro's like Debian are large collection's of little guys, each working on their own packages mostly in isolation. This is necessarily the case because we (I'm a DD) only have so many hours in a day. Thus if it were not possible to divide the large workload into thousands of tiny bite sized chunks, Debian wouldn't exist. Deploying the Debian "container" - ie a new release, is a huge amount of work which is why you see so few of them. Releasing a new one every time a new version of a package comes along (which is effectively what you are doing) is completely out scope.
So containers and distro's are like oil and water. They don't mix very well in most situations - yours being a notable exception. If they are going to mix something has to change. I can't see it being the containers - at the packaging level these isn't much too them. So it has to be the distro's. The first approach that strings to mind is the distro that is hosting the containers automagically keeps them patched. That requires both the host and container to be running the same distro - but I suspect that usually is the case. If that happened it would remove the major impediment to containerising everything for small guys like me.
Posted Jun 1, 2015 0:18 UTC (Mon)
by dlang (guest, #313)
[Link]
At Google they don't build "containers" and deploy them. They think in terms of "Jobs" that need X instances with Y cores and Z RAM. The fact that the implementation of this is in containers is not something that the normal Google developer or Site Reliability Engineer (the closes they have to sysadmins) ever think about. It's really far closer to the Mainframe job submission mentality than it is the 'traditional' server (even VM) model.
Posted Jun 1, 2015 1:02 UTC (Mon)
by dlang (guest, #313)
[Link] (8 responses)
Actually I see exactly the opposite. I think that the current mentality of people building containers where they install large chunks of a distro and run what's close to a full machines worth of software in each container is what's going to change.
Containers need to contain only the software actually required to run the software, and that is FAR less than just about anyone is putting in a container.
A decade+ ago I was working to simplify management of software using chroot sandboxes, setting them up so that they only contained files that were actually used by the software in question. (not just the packages listed as dependencies). The result is a much smaller footprint than any of the container definitions I've seen so far. Minimizing the container contents like this does wonders for your security requirements (you don't need to patch things that aren't there)
But containers need to evolve away from "install full packages and much of the OS)" and to something that is much more trailered for the job in question. Figuring out how to build such systems cleanly will help figure out how to build updated versions, but there is still going to be the question of how you update anything that contains enough state that you don't just replace it.
The idea of doing a CoW image as the base of many containers is trying to sidestep this bloat by spreading it's cost across many running containers (even if different ones use different subsets of the image), but it doesn't at all address the upgrade question. Saying that you layer filesystems so that you can replace lower levels in the stack only works until you need to change something higher up to work with a newer version of a lower layer.
Posted Jun 1, 2015 1:40 UTC (Mon)
by ras (subscriber, #33059)
[Link] (7 responses)
True. But it creates a different problem. Whereas before you had one installation to manage, now you have many. So while it is true each individual container contains less packages, for Debian every container will contain all the Debian essential packages. Or to put it another way, containerisation doesn't cause the total number of packages to drop. If you needed apache2, varnish, ntp and whatever else in the old setup, you will still need them in the containerised setup - albeit not installed in every container.
The net result result is while the total number of packages used doesn't change, but the number of deployments of them you have to manage (read: configure and ensure they are security patched) increases - in fact is multiplied by the number of containers you use in the worst case. On the up side I imagine the configuration of each container much is simpler, but on the down side you now have extra configuration to do - setting up virtual nics, allocating them IP's, mounting file systems inside the container, broadcasting the IP's so they can talk to each other. My guess is on the balance work involved in configuration isn't much different either way.
But this explosion in deployments is big deal if the sysadmin has to update and patch all of the containers, which is the case now. If the distro looked after it the work load reduces to what it was and it doesn't matter so much. And you get the security benefits for free.
In the long term this will be solved, and what I suspect is the real benefit containers have will make itself felt. Containers bring the principle of "private by default" modularisation to system building. The number of "running on the same system" assumptions will drop as a consequence, interdependencies will drop (despite the dbus's mobs valiant efforts to make everything talk to everything else), and things like huge apache2 config files managing 100's of sites will be a thing of the past. But that's a long way away.
Posted Jun 1, 2015 2:29 UTC (Mon)
by dlang (guest, #313)
[Link] (6 responses)
you are correct for how containers are being built right now.
I am saying that this needs to change
The fast majority of files in those "Debian essential" packages (and actually quite a few of the full packaged) are actually not going to be needed inside the container.
If you create a container, run it for a while (ideally exercising every feature in the software you installed), and then look at what files have an atime newer then when you started up the container, you would find that the vast majority of the files on the system were never accessed.
There is a lot of software that's needed for a 'minimal' system that's running completely self contained than is needed to run a piece of software inside a container that doesn't need to do lots of other things that you need to do on a full system (boot software, daemons, etc). If the software you are running is statically linked, you may not need anything beyond the one binary (in a 'best case' simplified example). Even a lot of the stuff that's typically done inside the container today could actually be done externally (job controls, monitoring, logging are pretty obvious wins), the question is at what point the value of splitting things out of the container is outweighed by the value of having everything bundled together inside the container.
Most of the container contents being created today are full distro installs (or pretty close to that), almost the exact same things that would be in a VM image or an image running on bare metal.
Posted Jun 1, 2015 7:39 UTC (Mon)
by kleptog (subscriber, #1183)
[Link]
There is the point made further up about how containers are less useful for deploying individual applications that you don't manage yourself like a single wordpress install. In our case we build two or three images but then deploy them a few hundred times with slightly different configurations. This changes the balance significantly and is vastly easier to manage than a few hundred VMs.
Posted Jun 1, 2015 10:08 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
About the only significant redundant piece is python-minimal that is needed for apt. Well and apt itself, of course.
In the end, we simply decided to use the official baseimages since several megabytes worth of dead disk space per container (no RAM overhead unless apt/python are actually used) are not worth maintaining our own images.
Posted Jun 1, 2015 17:16 UTC (Mon)
by dlang (guest, #313)
[Link]
Posted Jun 3, 2015 0:09 UTC (Wed)
by ras (subscriber, #33059)
[Link] (2 responses)
The 140Mb [0] that debootstrap installs maintains the debian distribution that lives inside of the container. The way things are done now it's a necessary part of the container. Docker files generally start with "take one minimal debian installation; apt-get install these packages ...". That can't happen without that 140Mb. If you get your containers to install their own security patches, that 140Mb is going to be needed for the life of the container. Even if you don't Debian's policy of not having explicit dependencies on "required" packages means it's very difficult to figure out what you can remove without writing your own software to follow the reverse dependencies (which I have done).
Part of the reason I say the distro's have to change is I agree this stuff shouldn't be in the container. If the distro's become container aware, the host can use it's copy of dpkg and so on to build and later maintain containers. If that happens you get the added benefit of security patches being applied automagically by the host as happens now in the non-container world, rather than having to do this manual rebuilding rubbish.
This is where my statement above, that the next step in move to containers is the distro's change, comes from. At the moment what we have is 1/2 baked.
[0] It's only recently realised that Debian minimal install is 140Mb. That's huge - and it's after I've pruned the cache's debootstrap creates. Damn Small Linux for example crams an entire distribution (kernel, GUI environment, 2(!) x Browser, a plethora of editors) into 120Mb.
Posted Jun 3, 2015 0:34 UTC (Wed)
by dlang (guest, #313)
[Link] (1 responses)
I think we differ mostly in that as far as you are concerned, fixing this is "the distros becoming container aware" while for me fixing it is "the container system becoming update/distro aware". The difference being which side is responsible for making the changes.
Posted Jun 3, 2015 2:56 UTC (Wed)
by ras (subscriber, #33059)
[Link]
That gets hard, because your container now has know a lot about the packaging system the distro uses. In Debian this means it would have run dpkg itself, which is possible because dpkg does take a --root parameter. But that means the container would have to handle dependency resolution. All of which is possible of course, and if we were only talking about Debian probably even easy for some definition of easy. [0] But we are talking about tracking every packaging system out there - including things like pypi.
They are not going to do that. Their success so far has been built on them avoiding doing it. Instead the user writes a script, the script uses some magic to build an image. The container's role starts in earnest after the image is built - they can deploy them across server farms, start them, stop them and even provide tools like etcd so they can configure themselves. It all works because the icky details of how to build and extend an image are all held inside the image itself. In that 140MB. That's why it's never going away without something changing.
If you are going to get rid of that 140MB there is one place I am pretty sure it isn't going to migrate to - and that is into the container software - eg docker. Debian providing tools that manipulate packages inside of a container, and the user running those tools from the existing docker script sounds like a much saner route to me. Of course this means the docker script would only work when run in a Debian host. Which is how we get to containers being tied to a particular distribution - while the container software (eg Docker) remains distribution agnostic. In principle the built containers could be distribution agnostic, but since Debian built it, it's not difficult for the Debian host to figure out what containers are effected by a security patch and notify the container software to do something about it. And thus you get to the containers being distribution specific too.
So we get back to my original point. All the changes that must happen to make this work are in Debian, or whatever distro is being used. The container software just continues to do what it does now. Thus my conclusion that the next step in the evolution in containerisation must come from the distro's - not the container software.
[0] The recent discussion on the debian development lists over how poorly aptitude does dependency resolution compared to apt provides a hint. "Easy" here means it could be done by someone - but even software written by Debian itself has trouble getting it right.
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
We tried that here as an experiment. Turns out that unless your application is almost statically linked pure-C app, you can't really remove that much. You still likely need glibc and all of its crap, libstdc++, OpenSSL, libz and so on.
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers
An introduction to Clear Containers