KubeCon EU 2016, part 1: Kubernetes 1.2

KubeCon EU, held in London March 10th, was the second conference dedicated to the Kubernetes container orchestration system. The sold-out attendance of 500 showed how popular the project has become since the release of version 1.0 by Google in July 2015. One week after the conference, version 1.2 was released, which included many long-awaited features.

Kubernetes is one of several "orchestrators" for Linux container infrastructure. While Docker handles packaging, installing, and running application containers, other software is required to manage container deployments across a cluster of computers, network them together, and monitor their status. The three leading software packages fulfilling this need are Kubernetes, Mesos Marathon, and Docker Swarm. As more users deploy containers in production, orchestration software has become the most intense area of open-source competition.

The conference organizers put together strong technical program with presenters from a dozen different companies, including Google, Red Hat, Kismatic, and CoreOS. It included talks covering Kubernetes's various storage-configuration options, scaling infrastructure, software-defined networking, and several different user-story presentations. This week, we will look at some new features for Kubernetes 1.2 that were presented at the conference, while next week we will finish up with some other interesting talks.

Kubernetes 1.2

Kelsey Hightower, developer advocate for Google, started off the keynotes by showing several of the new 1.2 features, including Deployments, ConfigMaps, and Ingress. First, he had to explain the updated timeline for Kubernetes 1.2, though. 1.2 was originally planned to be released in early February, but the release did not take place until after the conference. The delay seems to have been primarily due to a higher than usual number of bugs to close during the beta.

The biggest change in 1.2, or at least the one that will cause the most rewriting of configuration files, is the addition of a "Deployment" as a new top-level object. Kubernetes already had Services, which define particular endpoints that users or other applications might want to reach, such as an "HTTP-server". Users could then deploy multiple containers running this Service through a configuration object type called a Replication Controller, which have been renamed "ReplicationSets" in 1.2 to be consistent with the naming of other new objects. The other main object in the Kubernetes universe is the Pod—a small group of containers with close dependencies, requiring it to be deployed as a unit.

However, there was no object defining the event of deployment itself, which created a number of problems. For example, distinguishing between updating an existing set of Pods and deploying an entirely new Service could be confusing for users. More importantly, replacing production containers with upgraded ones required external, user-created scripting.

As of 1.2, users will create a Deployment in order to deploy a particular Service or application (defined as a collection of Services). Each Deployment is numbered so that it can be distinguished from prior Deployments of the same services. Hightower gave an example of a Deployment for the Node.js-based Ghost blogging tool. This deployment includes one Pod, with an nginx webserver and a Ghost blog server. A simplified version of the Deployment would look like this:

    apiVersion: v1
    kind: Deployment
    metadata:
      name: ghost
    spec:
      replicas: 1
      template:
	metadata:
	  labels:
	    app: "ghost"
	    track: "stable"
	spec:
	  containers:
	    - name: "nginx"
	      image: "nginx:1.9.12"
	      ports:
		- containerPort: 80
	    - name: "ghost"
	      image: "kelseyhightower/ghost:0.7.7"

To explain this in detail, we have to start at the innermost level of the tree. The containers section contains a list of the actual container images required for the Pod, which will be deployed as a unit. In this case, that's a Ghost application server, and an nginx web server to act as an HTTP front-end for it. It also contains some additional information, like the specific container image versions to use and any ports to be exposed to other services in the cluster. The labels section contains a set of user-selected key-value pairs that can be used within Kubernetes to select that Deployment, such as for network mapping.

This new definition allows users to increase the number of Pods by updating the number of replicas and "re-applying" the Deployment file, which Kubernetes treats as a "state definition": it will shut down or start Pods until the desired number matches the number given. Users could do this previously using Replication Controllers, although the configuration was somewhat more complex. To do that, users would change part of the configuration above to:

    spec:
      replicas: 4

Then, a user would execute kubectl run to update the deployment definition, and Kubernetes would start bringing up more Ghost Pods to make the required number. Note that all such settings are also accessible through the Kubernetes API, using the same heirarchical structure.

With Deployments, users gain some useful new abilities. For example, Hightower demonstrated creating a rolling update of a service using Deployment Strategies, by specifying that only one container at a time can be down. This is the default behavior when you update a Deployment, and adds functionality into Kubernetes that had previously been available in Red Hat's OpenShift platform. To update Ghost, Hightower changed the following lines:

    - name: "ghost"
      image: "kelseyhightower/ghost:0.7.8"

Then, he re-ran the deployment and gradually the four Ghost Pods were replaced by ones running a newer version of Ghost—one at a time. This functionality, which allows upgrading services without downtime, was previously available using the kubectl rolling-update command, but that older command was less flexible and had several problems. More importantly, the Deployment Strategies API will allow users to implement other patterns for deploying new versions, such as Blue/Green and Canary.

Next, Hightower showed the audience a typical "docker-entrypoint.sh" shell script used to start a container. He called these "init scripts for hipsters," and said he'd seen ones that were longer than the application they supported. One of the motivations behind using such scripts is the common Docker practice of passing configuration settings via environment variables, since many applications require configuration at container launch time.

To provide a better way, 1.2 now supports a new way to configure containerized services: a ConfigMap. This API allows users to pass configuration files into containers from the Kubernetes controller node as files. For example, this YAML snippet tells the nginx server to run using a "ghost.conf" config file that connects it with Ghost:

    - name: "nginx-conf"
      configMap:
	name: "nginx-ghost"
	items:
	  - key: "ghost.conf"
	    path: "ghost.conf"

The ConfigMaps are mounted on a memory-based tmpfs on each node, ensuring that they don't stay around as a potential security hole when the container is shut down.

The third feature Hightower demonstrated was "Ingress", which gives users explicit control over what ports are accessible from outside the Kubernetes cluster. Previously, such access has been a side-effect of Pod and Service declarations and, as such, was difficult to understand. Now, access from outside the cluster for a service can be added like this:

    apiVersion: v1
    kind: Ingress
    metadata:
      name: ghost
    spec:
      backend:
	serviceName: ghost
	servicePort: 80
      tls:
	- secretName: "ghost-tls"
	  hosts:
	    - "*.example.com"

The big technical advantage of having explicit Ingress controllers is that Kubernetes can support TLS connections for load-balanced services, as well as HTTP-based routing. In the above example, TLS-encrypted connections to the Ghost service are supported for the "example.com" domain. This requires linking in a Secret, which is defined elsewhere in the configuration files, that contains the TLS certificate and key.

Kubernetes 1.2 and stateful containers

One of the major issues with the containerization of services on Linux has been managing those requiring persistent state, such as databases and Content Delivery Network (CDN) servers. Members of the Kubernetes project aim to solve this problem with a series of feature advances over the next few releases. In "The State of State," Matthew Bates, co-founder of JetStack, brought the audience up to date on where Kubernetes is with stateful services.

He started by citing the TechCrunch article "I want to run stateful containers, too," which lays out all of the reasons users want their databases and other state to be in containers. Having a single infrastructure that controls all services helps with efficiency, scheduling, scaling services, and allows automating everything in a consistent way. However, there are some additional features needed.

Bates went over how state in Kubernetes works now: you have Volumes that use a Volume Plugin to store data. The simplest version stores data on the host's filesystem, but others support using network storage, such as NFS or Ceph. This allows the data to outlive restarting containers. The next step is PersistentVolumes on network storage, which are declared as an object in Kubernetes and can be mounted by one or several containers, depending on how they are set up.

One problem with Volumes, though, is that there was no way to ensure that only one container at a time could use a particular storage volume, especially if it was locally mounted storage. One new way to do that is the 1.2's DaemonSet: a Kubernetes object that defines a service to be run only once per machine. While DaemonSets were designed more for handling other per-node services, like log management, they can also work for databases that need to use most of the disk on a node. A better way to control this is the "node affinity" tags, which let users define policies for what services should go together (or not) on the same nodes. This is another new addition for 1.2.

Volumes only deal with one aspect of persistence: storage. Many stateful services, especially databases, expect stable addresses, both for being reached by applications and by other nodes of a clustered database. Kubernetes is starting to address this in upcoming services, such as a draft API nicknamed "PetSet."

Since "PetSet" is currently just a pull request (PR), its API is not yet fully described. However, the idea is to be able to have a Pod that can maintain identity and network location across restarts and upgrades. This would use some kind of parameterized templates. He appealed to potential users to comment on the PetSet PR.

Those two talks gave a good look at the new features in Kubernetes 1.2. Next week, we will cover several more talks from KubeCon EU: one that looked at the Kubernetes community and the development plans for 1.3, another discussing the project's decision to join the Cloud Native Computing Foundation (CNCF), and, finally, a look at how Trusted Platform Module support, recently added to CoreOS, can be used to audit container security. Stay tuned.

[ Josh Berkus works for Red Hat. ]