LWN.net Weekly Edition for February 18, 2016

Red Hat, Fedora, and containers

DevConf.cz is an annual conference in Brno, the second-largest city of the Czech Republic. While the conference is open to the public, both as attendees and as speakers, the majority of the 1600 people at the conference were Red Hat staff. As a first-time attendee and newly minted Red Hat employee, it felt like DevConf was a Red Hat internal technical conference that the public was invited to.

Fortunately, the company supports and contributes to enough different open-source projects to make for an interesting and varied program. Topics covered included Fedora, Project Atomic, OpenShift, CentOS, OpenStack, JBoss, and many smaller projects. And, thanks to the Red Hat engineering center in Brno, presentations tended to be technically detailed.

If there was a theme to this year's conference, though, it was Linux containers. Session after session either discussed technology and techniques for containers, or demonstrated other tools in conjunction with container infrastructures. This included sessions on immutable infrastructure, containerizing distributions, and the future of Fedora.

Red Hat containers with Project Atomic

Pretty much every company in the world of software today has a container story and even container products; Red Hat is no exception. Most of Red Hat's container efforts are funneled through Project Atomic, which is an umbrella project for multiple open-source tools and platforms. Among these are Atomic Host, the Atomic Developer Bundle, RPM-OSTree, as well as Atomic.app and the Nulecule specification. Red Hat staff also contribute to Docker, Kubernetes, and other third-party container tools, and the company is a member of the Open Container Initiative.

Atomic Host is probably the most visible of these efforts. Not really a project in itself, Atomic Host is a re-distribution, or "spin", of each of the three Red Hat platforms: Fedora, CentOS, and Red Hat Enterprise Linux (RHEL). Atomic Host is meant as a "thin OS" that only has the tools required to be a container-management platform, with the idea that all applications are installed as containers instead of packages. Its own software is managed through RPM-OSTree (see below), and it includes distributions of popular container tools such as Kubernetes, Docker, Flannel, and etcd. Atomic Host can be considered an alternative to CoreOS's Tectonic.

The Atomic Developer Bundle (ADB) is a virtual machine full of tools for building application container images. Packaged as a Vagrant virtual machine, it's intended to make it easy for developers on Windows and Mac laptops to get into container-based development. The ADB is less important on the Linux desktop, where all of these tools are available natively. The project serves the same purpose as the Docker Toolbox.

Another problem Red Hat engineers wanted to solve was to provide the ability to bundle important metadata and even multi-container applications without attaching them to a specific orchestration system. The Nulecule specification, and its sole implementation, Atomic.app, is meant as an answer to this. Packaging containers with structured metadata allows the Atomic.app software to generate code for any of several orchestration systems, such as Kubernetes or Mesos.

While many of the products based on these projects are still under heavy development, Red Hat has released an all-new version of their Platform-as-a-Service cloud, OpenShift, which is built on top of Project Atomic components. The new containerized infrastructure was praised at the conference as being more scalable, manageable, and faster than the previous design. OpenShift is available as an open-source stack in the OpenShift Origin project.

While it was already apparent that Red Hat was involved in Linux containers, DevConf.cz made it clear how central containers are to its plans for the OS in the future. Various presentations explained how the staff and executives plan to make use of container technologies and ideas to change how Linux is distributed and deployed. One of these ideas is "immutable infrastructure."

Immutable infrastructure and OSTree

Two presenters, Adam Miller of Fedora Engineering and Colin Walters of Platform Engineering, each explained in their presentations how the move to containerization, coupled with other new ideas, will change things.

In "The Magical Future", Adam Miller explained the idea of "immutable infrastructure". This has two principles, he said: full automation and the immutability of each component, which doesn't change from development through testing to deployment. The idea is that everything you deploy should be a "build artifact" that doesn't get configured at runtime. Instead, all configuration management happens at build time, the only exception being differences between development and production environments. Even those should be handled by read-only configurations; one per environment.

"The idea is that you don't configure services in the environment, you configure and re-deploy," explained Miller.

This principle of immutability is familiar in the Linux container world. Miller introduced the idea of an "immutable OS", based on RPM-OSTree on top of Atomic Host. OSTree, introduced in 2011, manages a root filesystem in a way that is similar to Git commits, where all files in a commit change at once. RPM-OSTree, created in 2014, takes this concept and applies it to RPMs and sets of RPMs. This makes updates to the Linux server "atomic" in nature, meaning they update or roll back as a unit.

The Atomic Host versions of Fedora and CentOS work on this principle of immutability by using RPM-OSTree. Instead of updating individual packages on each host, each server is copied from a "parent" image in the same way that Docker containers are copied from a filesystem image. No local changes to installed software are possible, only updates to synchronize with changes in the parent image.

Walters expanded on this in his session about "Containerizing the Distribution." Walters was recently involved in building OpenShift version 3, which required a total overhaul of the platform to use containerization and immutability. According to him, while containers have given us "new boxes", they haven't really changed how we build software.

Walters then demonstrated building a container using RPM-OSTree instead of "docker pull". Each "branch" of the OSTree represents a package, and can be treated like a "layer" of a container image. He said that this approach is superior because it makes it much easier to maintain common library packages across a container infrastructure. Right now, this approach is working only using a hacked version of the Mock build-and-chroot tool.

Walters also did a demonstration of container image creation starting from RPMs. Again using OSTree tools, he showed creating a Docker image directly from RPMs and then pushing it to a shared registry. Ideally, he explained, the registry would be on shared storage so container images could simply be mounted by the individual servers.

Containerization and the future of Fedora

While the presentation by Denise Dumas, VP of Platform Engineering at Red Hat, and Mathew Miller, Fedora Project Leader, wasn't marked as a keynote on the program, it might as well have been. Dumas's half of the talk was full of grand ideas and ended with a call to action. In it, she presented her vision of a new direction for the Fedora project, which unsurprisingly involves containers.

Part of the Fedora project values are the "four Fs": Freedom, Friends, Features, and First. To that, Dumas would like to add one more: "Faster". Increasingly, she explained, users are looking to rapid delivery of applications from their platform. She wants Fedora to help Red Hat find ways to deliver an OS that is faster, smaller, and more modular, while still being more secure.

"Two releases a year used to be fast," she said. "We want to be moving continuously, we want to be DevOps." From her perspective, Red Hat does a lot to support Fedora. In return, she wants Fedora to be the place where it can experiment with what it means to be an OS in 2016, or in 2020. Dumas claimed that "disruption is coming," and that Fedora should be part of it instead of being disrupted by it.

A lot of her vision for Fedora centers around containerization and immutability. This includes both Fedora Atomic Host and the new Fedora Atomic Workstation. The latter uses OSTree and xdg-app, a tool for sandboxing GUI applications, to deploy images and updates to developer workstations. Specific applications could be installed as containers, as they are on Atomic Host. Right now Atomic Workstation is still in early alpha stages, with plans for a prototype of Fedora 24.

Miller agreed with Dumas, saying "Fedora Atomic is the future of the operating system [...] Look at RancherOS. We need to be in the forefront of that." Most of his portion of the presentation reviewed the last year of Fedora's progress. Fedora had contributions from around 2000 people last year, of whom about 35% are Red Hat staff and the rest are from outside. Fedora Workstation, a variant of the distribution aimed at software developers, has been popular. So has the network install option for new machines.

He also showed the statistics from the various "spins" of Fedora, such as KDE, Security Lab, and Robotics. Fedora KDE has been popular since it was the first version of the OS to include Plasma 5. Other spins aren't used by as many people, but are the primary OS for their audiences. For example, Fedora Robotics has been the OS of choice for the DARPA-sponsored robot soccer matches.

The distribution is also replacing the venerable X server with the new Wayland graphics server. That will happen in either Fedora 24 or 25, depending on readiness.

Miller also mentioned the Fedora Layered Docker Image Build Service, which was explained in more detail in a later presentation by Fedora Release Engineer Dennis Gilmore. The new service will provide standard builds for images to be distributed as Fedora applications. In Fedora 24, the service will be simply available, but by Fedora 25 it is expected that some applications will be distributed as containers instead of as traditional packages. The project will also be creating its own container-image registry to support this.

More to come

Of course, not everything on the road to Red Hat containers has been a smooth ride. Dan Walsh detailed some of the technical conflicts with Docker, in his presentation "Systemd vs. Docker", which will be covered in a later article.

DevConf.cz was full of all sorts of other interesting projects thanks to presentations by what seemed like half of Red Hat's engineering staff. This also included the Cockpit server management GUI, the FreeIPA identity management server, a new configuration management tool, the CentOS build service, and more. Look for further coverage in the coming weeks.

Comments (3 posted)

Learning about community

Communities that form around open source and open hardware tend to have a strong DIY — Do It Yourself — focus. At its best, this can result in innovation and value creation on multiple fronts. At its worst, this risks getting caught up in NIH syndrome, where anything Not Invented Here seems unworthy of our time. Chris Cormack brought a message [WebM] to linux.conf.au to warn against letting NIH affect our approach to community building and to encourage us to look beyond ourselves to learn about community. He himself identified as a member of two very different communities and found that the wisdom of the more ancient culture was quite beneficial in building the nascent one.

In the first place he is a Māori — the people who have inhabited New Zealand for the last 700 years. He does not look like a typical Māori and admitted that he is probably one of the palest Māoris you will see. He described himself as a "stealth" Māori, though he can trace his lineage back 47 generations. Today the south island of New Zealand is known for its spectacular beauty but, when Cormack's forebears arrived centuries ago, they found a land that was "freezing cold, where nothing would grow, and that had no land mammals". Without the food, transport, and labor that such mammals might provide, building a strong, healthy community was important for survival.

The second community Cormack identified with was the development community for the Koha open-source library-management system. Koha was built by Cormack and some colleagues when a local library discovered, with only a few months to spare, that its current proprietary system had a Y2K problem that could not be fixed. Motivated by those immortal words "how hard can it be?", they started learning about libraries and building software — managing to go live on January 3, 2000.

The Koha community has grown over the years (though not so much in New Zealand or Australia as might have been hoped) with installations around the world, translations to "30ish" languages and a great deal of diversity. Cormack particularly noted that the combination of librarians (traditionally female) and developers (traditionally male) has resulted in an unusually good gender balance. He was also rather proud to have received security fixes from a Benedictine monk, accessibility fixes from a blind Buddhist monk, and was still hoping to hear from a Trappist monk as they apparently make good beer.

Cormack noted a few Māori proverbs that focused on community, several of which had close English equivalents such as "many hands make light work" or "we stand on the shoulders of giants", but structured his presentation around five Māori terms that are part of common discourse in that culture. They serve to sum up that community's wisdom and to remind us of some important values. While these terms might superficially seem familiar, some carry an emphasis that can challenge our thinking.

Mana tangata — from where comes your reputation?

"Mana", we were told, is a term you will often hear in New Zealand. It can be translated as "respect" or "prestige", but it is more; it is much richer than these terms. A possible cognate that came up in other talks during the week is "reputation" — how we are known to and are perceived by others.

In her keynote [WebM] on Friday, where she discussed some of her research as an anthropologist looking at people interacting with technology, Genevieve Bell observed that people have a wide range of responses to issues of privacy: some are very protective, some give their data away without a second thought. However, when the conversation is turned around to be about reputation, responses are much more focused. People care about their reputation, realize that it can be based on even the tiny things they do, and are concerned about ubiquitous surveillance affecting their reputation even if they don't care about privacy.

"Mana tangata" presents a sharp contrast to this perspective of reputation based on what we have done. Rather, it refers to respect/prestige/strength/value/reputation that comes not from what you have done, but from the community you are a part of. Cormack offered a quote that he particularly liked from author Michael P. Shirres, who explained the concept this way: "To be a person is not to stand alone, but to be one with one's people and the deeper the oneness the more we are truly persons and have that mana tangata."

This view of reputation or prestige coming from community does not conflict with the view of it coming from actions, but is really just an alternate perspective that can lead to a different way of thinking. Cormack demonstrated this by linking mana tangata with the importance of welcoming people into a community. By doing this, by connecting people with ourselves, we build up their mana as well as strengthen our community.

This idea of the importance of giving, not just of gaining, reputation is central to the thoughts Katie McLaughlin had to share in her five minutes of "lightning talk" in the final session of the conference. In part, McLaughlin was championing the "#LABHR" initiative: Let's All Build a Hat Rack. To "hang your hat on" something is an idiom for basing your reputation on that something. Building on that analogy, a hat rack could be a collection of skills, experiences, contributions, etc. that together form a person's reputation. We can all participate in building hat racks by publicizing and celebrating the contributions of others, particularly the otherwise unseen contributions.

This is a slightly different emphasis than the one presented by Cormack. In mana tangata, the community doesn't provide the reputation, it is the reputation. However, it is a practical way of seeing reputation as something that a community gives, rather than something an individual must gain.

Whakanuia — effective teaching

"Whakanuia" is pronounced with an "F" sound for the "Wh". Cormack summarized it as an approach to teaching focused on "reward the good, ignore the bad". He gave examples of a couple of gifts he had received from grateful members of the Koha community. These were of minimal monetary value (e.g. a jar of peanut butter), but of significant community-building and motivational value. Little rewards can have a big effect.

In the other direction, Cormack described his "unsung heroes" announcements. Many projects announce lists of new features and fixed bugs. For Koha there are regular lists of contributors and contributions. This is more than just a list of authors from a Git log; it identifies a mix of contributions both tangible and intangible, both named and anonymous, as every contributor deserves to be celebrated.

Cormack did not expand on the importance of ignoring the bad, and maybe that is for the best. Maybe it is beneficial to just save our energies for celebrating the good that we see.

Kaiakopono — mentorship

This is probably the least challenging of the ideas that were presented. "Kaiakopono" means much the same as "mentoring", though perhaps with an extra spoonful of "welcoming" thrown in. Cormack borrowed an interpretation from author Esther Schindler, who wrote: "Not everyone needs to be the welcome wagon — but someone ought to be."

The importance of mentoring seems to be well understood in open-source communities; there are a range of programs that encourage and support it. There is little to take away from this Māori term except maybe to be reminded of what we already know is important. It is also rewarding: Cormack rejoiced in the "massive buzz" he gets when someone he'd been helping doesn't need help any more.

Korerorero and whakaaro whānui — conflict and resolution

Conflict and disagreement are inevitable in any community. How a community handles those inevitabilities can shape how the community is perceived — it certainly appears that many see the Linux kernel community in terms of the conflicts on its mailing list.

Korerorero, according to Cormack, is somewhere between "argue" and "discuss". It is about coming together to address issues and to present opinions and to gain understanding. It is also about eating and drinking together. It seems that something different happens when discussions, particularly passionate discussions, are held around a meal table. Maybe it is the reminder of community, or perhaps the food distracts those with less firm opinions. Cormack highlighted the success of the "KohaCon" conferences that meet in a different country each year; he noted that all of his photos from the conferences seemed to have wine bottles on the tables.

Face-to-face meetings are clearly not always possible in a global community, so when they do take place, it is important to understand how to make the most of them. Learning from a different community that handles meetings effectively certainly seems like good advice. This makes one wonder how the Kernel Summit would go if the meetings were over a meal instead of between meals.

The outcome of korerorero will hopefully be "whakaaro whānui". This is people in agreement. It may not be complete unanimity, but Cormack characterized the outcome as one in which no one may be blamed if the result doesn't work out, but where everyone shares the credit when it does. This seems very close to the "rough consensus" that has been a big part of enabling the IETF (Internet Engineering Task Force) community.

This was highlighted by George Fong in his keynote address [WebM] on Tuesday, when he reminded us of some of the details of how the Internet was built; he noted the IETF, which has been quite successful over many decades and "simply works" despite the fact that: "There isn't a single law in place that governs them, there isn't a single overarching governance process which is imposed on them."

To explain the reason for that success, he quoted David Clark who is recorded in "The Tao of IETF" as saying: "We reject kings, presidents and voting. We believe in rough consensus and running code."

Fong's thoughts on the IETF can be used to assess other communities. Running code is something that is highly valued throughout the open-source world. Whether rough consensus is so highly valued is less clear. It is certainly not hard to identify projects, both past and present, that demonstrated a degree of success with real working code, but for which there was no community consensus — rough or otherwise. Some of these have brought about both real technological value and significant community division. Whether the one was worth the other is an interesting question. It probably depends on which one values more: the technology or the community.

He aha te mea nui?

That question leads quite nicely to the title of Cormack's talk, "He aha te mea nui?", which roughly translates as "What is the most important thing?". The traditional answer is "he tangata, he tangata, he tangata": the people, the people, the people. It is clear that for Cormack and the Koha community, it is the people and the community that are most important. The technology certainly has importance, but its true importance derives from how it provides access to knowledge, allows people to contribute, and encourages everyone to be involved.

Whether the community exists to serve the technology or the technology exists to serve the community, it is clear that a healthy community is important. But it is also often clear that technologists aren't experts in that area. Cormack's final encouragement was to learn from communities by joining them. There are plenty of communities around us that have been learning from their mistakes for much longer than the Internet or free software has been around. We might learn something by observing, by volunteering, or by joining. Many of the challenges we face are not new at all.

Comments (1 posted)

Winning the copyleft fight

Bradley Kuhn started off his linux.conf.au 2016 talk by stating a goal that, he hoped, he shared with the audience: a world where more (or most) software is free software. The community has one key strategy toward that goal: copyleft licensing. He was there to talk about whether that strategy is working, and what can be done to make it more effective; the picture he painted was not entirely rosy, but there is hope if software developers are willing to make some changes.

Copyleft licensing is still an effective strategy, he said; that can be seen because we've had the chance to run a real-world parallel experiment — an opportunity that doesn't come often. A lot of non-copyleft software has been written over the years; if proprietary forks of that software don't exist, then it seems clear that there is no need for copyleft; we just have to look to see whether proprietary versions of non-copyleft software exist. But, he said, he has yet to find a non-trivial non-copyleft program that lacks proprietary forks; without copyleft, companies will indeed take free software and make it proprietary.

As an example, he noted that the version of OpenStack running behind his Rackspace account clearly has features that are not in the free version. He does not know of a single OpenStack product that does not contain at least a few features that have not been pushed back upstream.

There is more and more non-copyleft software out there, he said, but copyleft itself remains the most viable strategy to bring about software freedom. Other strategies can be employed, but we need to continue with copyleft as the centerpiece. Unfortunately, it is not working as well as we would like for one simple reason: there is less copyleft software being produced.

Corporate opposition to copyleft

There is not much support for copyleft in the corporate world; Martin Fink's recent LinuxCon Europe talk was the first positive words he's heard from a company about copyleft in quite a while. In general, though, companies are embracing free software; even Apple is, to an extent. They have discovered that they cannot beat the free-software community, so they have now switched to an approach of coopting that community. The result is the emergence of groups claiming to speak for the community, saying that copyleft is no longer wanted. He has heard developers say that users prefer permissive licenses and that it's impossible to get funding for work on copyleft-licensed software. Why are they saying that? In the end, Bradley said, if you hear things enough times you may well start to believe them.

As one might imagine, he sees the lack of enforcement for the code that is under copyleft licensing as a big problem. The only people who can change that, he said, are developers who hold their own copyrights in the code. Companies, which hold most copyrights at this point, will never do enforcement. Or, on the rare occasion when they do, it is to support their own goals rather than to defend the freedom of the software. One example is MySQL, which was happy to use the GPL as a way to force companies to buy proprietary licenses. Red Hat alleged GPL infringement against Twin Peaks Software as a way of defending itself against a patent troll; in the end, Red Hat got its patent license, and Twin Peaks Software's code remains proprietary.

The Software Freedom Conservancy recently published a set of principles that, they say, should govern GPL enforcement; the first of those is that the primary objective is to bring about license compliance. Since then, no trade association or company has endorsed those principles.

That is the case, he said, because lawyers in the corporate world have gained control over policy with regard to the GPL, and they are actively coordinating to try to push the GPL aside. They hold "secret meetings" in which to work through scenarios of possible enforcement actions and how to prevail against them. There have been "two meetings" with the purpose of finding ways to increase corporate ownership of GPL-licensed software as a way of curtailing enforcement by individuals.

The result, he said, is a crisis for the concept of copyleft. If copyleft is the right strategy, then some way has to be found to deal with these challenges. We need a plan.

Defending copyleft

The copyleft strategy works best when copyrights are held by people and institutions that hold software-freedom principles. That means individuals and charities, he said; companies and trade associations do not subscribe to those principles. Thus, he said, developers who support software freedom should go to their employers and insist on being allowed to hold their own copyrights. Perhaps developers should unionize to demand this right. Then they should join one of the Conservancy's coalitions for enforcement.

He also called for developers to go back to volunteer coding — off hours, it is not necessary to quit one's job to do this (though he did note that some companies will claim ownership of even an employee's off-hours work). Early free software was written this way. Bradley said that he works nights and weekends for the cause; he would like for the development community to do the same. Developers should also be willing to fork permissively licensed software under copyleft when necessary.

But copyleft won't work if we don't enforce the terms of the license, he said. Enforcement is getting harder: companies are actively working to replace copyleft-licensed software. Toybox and LLVM are two examples there. Given that, which essential copyleft-licensed software remains?

The one thing that companies haven't tried to rewrite, he said, is the Linux kernel. Thus Linux is the battleground over which the future of copyleft licensing will be determined. An area of immediate interest is proprietary modules, and whether they are derived works of the kernel; he has not yet seen one that didn't look that way to him. There are corporate lawyers who want to fight over this issue; it is a fight that the community will have to take on. If we shrink from this particular confrontation, we will no longer have effective copyleft licensing.

Once upon a time, he assumed that most license violations were innocent mistakes. Those days are coming to an end. Most violations, he alleged, are done deliberately by companies that hire lawyers in advance to plan strategies should it come to a fight. The community is helping these companies with its openness. This is, he said, a zero-sum game; the two sides have diametrically opposed goals.

GPL enforcement requires more than copyright holders, though; it also requires funding. That is a problem; enforcement as a profit-making enterprise is a path to corruption. Violators will always ask how much it will cost to buy permission, but if owners take money without demanding full compliance, they are just another player in the proprietary relicensing business. Companies will not support this work, so the only way is for the community to support it. He reminded the audience of the Conservancy's funding drive, which continues through February.

He concluded by saying that his plan can easily be described as simple-minded. It depends on a community of individuals coming together to stand up for software freedom and the GPL. But this should be possible: uniting and working toward a common goal is just what the community is good at doing.

The video for this talk is available on the LCA site.

[Your editor thanks LCA for assisting with his travel expenses.]

Comments (247 posted)