Development

Deploying Docker

The Docker project is not (like its name might suggest) yet another desktop application launcher. Rather, it is an application deployment system designed to make it easy for programmers to develop an application on one type of machine (such as a work laptop) and package it for predictable usage on a variety of other machines (such as a cloud instance or server farm). A Docker package is a container that can be deployed and run in nearly any Linux environment, but it is not a virtual machine image—instead, it is a self-contained copy-on-write duplicate of a master application image. It comes with some awkward technical baggage, but the ideas behind Docker proved compelling enough that one Red Hat developer recently took it upon himself to figure out how to adapt the system for standard Linux distributions.

The Docker site compares its deployment model with the idea of the standard shipping container: just as standard containers allow freight companies to move cargo on any vessel without modifying either the vessel or the payload, a Docker package is designed to be self-sufficient. It can be packaged up anywhere, then unpacked and installed anywhere. Each Docker application runs as its own LXC process container, with full network and resource isolation from the others. Cgroups and namespaces must also be enabled, although Docker will run with some cgroups (such as the memory controller) disabled.

The same claims would be made about VM deployments, of course. Where Docker differs is that it only containerizes the application and its dependencies; the process container runs directly on the host OS. This makes for smaller packages, less overhead, and faster application restarts. Multiple copies of each Docker container can be deployed together on the same server, sharing libraries. In this scenario, Docker again saves space by storing the containers on a union file system (specifically, AnotherUnionFS or AUFS). Only the differences between the various copies of the application need to be saved, so read-only portions of each application take up zero additional space.

Because each Docker container is a copy of the same original, starting up an additional instance of the application is also far faster than spinning up another VM. The AUFS filesystem can also be exploited when rolling out application updates, overwriting only the changed portions of the application.

Docker development is sponsored by dotCloud, a platform-as-a-service vendor. The system undoubtedly makes for a good base on which to build a cloud service, since the marginal savings over a VM approach continue to scale up as more and more applications are deployed. Nevertheless, it has attracted attention from cloudless developers as well, who would like to see it available on general-purpose machines running Red Hat Enterprise Linux (RHEL), Ubuntu, and other large distributions. On those systems, the cloud-scalability advantages might not be a significant factor, but the other ease-of-deployment features could still prove helpful.

The problem is that Docker relies on AUFS for so much of its functionality. AUFS is not part of the mainline Linux kernel; it is a patch set. Furthermore, the only major distribution to build AUFS support in the kernels it ships is Ubuntu—and Ubuntu has announced plans to deprecate it in favor of an upstream replacement.

That problem evidently got Red Hat's Alex Larsson thinking. Larsson wrote a blog post on October 15 describing his recent effort to get Docker working on top of a different filesystem. The obvious solution might be to use overlayfs, which seems to be on track for landing in the upstream kernel, but Larsson felt that it was not yet ready for use. Similarly, Btrfs has copy-on-write functionality, but Btrfs is not yet stable enough for many users.

Eventually, Larsson decided to try porting Docker to use Logical Volume Manager (LVM) thin provisioning with a little trickery from the kernel device-mapper: he creates a copy-on-write block device pool with LVM thin provisioning, then creates an ext4 "base" device on top of it. Each Docker container is then created as an LVM snapshot of that base device. The LVM thin provisioning means that the copy-on-write device only takes up space for the actual files it uses; since the LVM snapshots are layered on top of the base image, they automatically reuse the files below.

Larsson reports that there are still some issues to work out (such as how to overcome the various maximum sizes on the filesystems used), but he hopes to have the work land in time for the upcoming Docker 0.7 release. That may not make Docker ready for deployment on vanilla RHEL machines any time soon, but it is a step in the right direction. Larsson said he thinks the long-term solution will probably be to port Docker over to overlayfs, but in the meantime, the LVM-based approach looks like a good—and clever—workaround.

Comments (11 posted)

Quote of the week

— Alex Gaynor

Comments (6 posted)

Wayland and Weston 1.3 released

Version 1.3 of the Wayland protocol and Weston reference compositor have been released. In the release announcement, Kristian Høgsberg says that there isn't much that's new in the Wayland release, which is a sign of its maturation. New pixel formats support, additional documentation, language binding support, a few bug fixes, and more. This cycle for Weston was more active with the addition of hardware-accelerated screen capture, libhybris support, support for multiple input devices of the same type, better touch support, new launching options, and more. "We're going to try something new for 1.4 - we'll do an alpha release a month before the scheduled release. I'm looking at Jan 15, 2014 as the release date for 1.4.0, and we'll do an alpha release on Dec 16. The motivation here is to get a snapshot out a bit earlier so we can start testing earlier and hopefully uncover bugs earlier."

Comments (19 posted)

Samba 4.1.0 released

The Samba 4.1.0 release is out. This version adds improved SMB2 and SMB3 protocol support, encrypted transport over SMB3, server-side copy operations, Btrfs filesystem integration, and more. This release also removes the long-unloved SWAT web-based administration tool.

Full Story (comments: 1)

Cinnamon 2.0 released

Version 2.0 of the Cinnamon desktop has been released. There are some improvements in features like edge tiling, edge snapping, user account management, and more, but much of the work in this release appears to have been done to the supporting infrastructure. "Prior to version 2.0, and similar to Shell or Unity, Cinnamon was a frontend on top of the GNOME desktop. In version 2.0, and similar to MATE or Xfce, Cinnamon is an entire desktop environment built on GNOME technologies. It still uses toolkits and libraries such as GTK or Clutter and it is still compatible with all GNOME applications, but it no longer requires GNOME itself to be installed. It now communicates directly with its own backend services, libraries and daemons: cinnamon-desktop, cinnamon-session and cinnamon-settings-daemon."

Comments (55 posted)

PyPy's new garbage collector

The PyPy status blog has a detailed description of the new incremental garbage collector adopted by this performance-oriented Python interpreter project. "The main issue is that splitting the major collections means that the main program is actually running between the pieces, and so it can change the pointers in the objects to point to other objects. This is not a problem for sweeping: dead objects will remain dead whatever the main program does. However, it is a problem for marking. Let us see why."

Comments (none posted)

Development newsletters from the past week

• Caml Weekly News (October 15)
• What's cooking in git.git (October 14)
• What's cooking in git.git (October 16)
• Haskell Weekly News (October 10)
• Perl Weekly (October 14)
• PostgreSQL Weekly News (October 13)
• Ruby Weekly (October 10)
• Tor Weekly News (October 16)

Comments (none posted)

XBMC DevCon 2013 LiveBlog

Nathan Betzen of XBMC has posted a round-up of the events at the project's recent XBMC DevCon. The article was started as a live-blogging piece, but the conference has now concluded, leaving Betzen's account as a good record for those unable to attend. Issues discussed at the event include revising the settings system, 3D support, and adding the ability for multiple XBMC instances on a network to share content with each other.

Comments (none posted)