Embedded Linux developers meet for a Yocto project summit

On December 1st and 2nd, a group of about 40 leaders and engineers in the embedded Linux community met in San Francisco for a for two-day summit. On the table was the Linux Foundation's Yocto project, which was announced in October at the Embedded Linux Conference Europe (ELCE), and aims to make the development of custom embedded Linux systems simpler and easier. The Linux Foundation wants to see broad industry adoption of and involvement in Yocto, and this meeting was called to provide an overview of the project and to set up a community governance process. As such, the engineers and management from a wide cross-section of the industry were invited including representatives from Intel, Wind River, MontaVista, Mentor Graphics, Linaro, Dell, Texas Instruments, Qualcomm, Red Hat, and HP.

Jim Zemlin, executive director of the Linux Foundation opened the summit and welcomed all the attendees with the results of a survey that attendees were asked to fill out before the meeting. In typical Zemlin style, he pronounced that in the embedded space, "Linux is kicking ass". The bad news is that there is a shortage of well-trained embedded Linux developer and, in particular, system engineers. There is a desperate need to get more people "in the tent". The merger of the Consumer Electronics Linux Forum (CELF) with the Linux Foundation, announced in October, along with the Yocto project creation announced at the same time, means that the Linux Foundation is now directing significant resources toward improving the state of embedded Linux development.

As a first step, the Zemlin announced that Richard Purdie, the leader of the Poky project, has become a Linux Foundation fellow in the role of Yocto project architect.

Before the summit, attendees were surveyed about their view of the Yocto project and what they would need to make it a useful technology. Rather than attempting to analyze the results, Zemlin provided the raw answers to the room (with names removed) so that attendees could make of it what they would. Not surprisingly, it appears that few people are familiar with the Yocto project, and many were hoping to come out of the meeting with a better understanding of what it was and whether or not it would be useful for their work.

An overview of the project

The rest of the morning was devoted to an overview of the Yocto project by Purdie and Dirk Hohndel, Intel's Chief Linux and Open Source Technologist. According to Hohndel and Purdie, Yocto is an umbrella project covering several initiatives with the goal of reducing the duplication of effort required to build custom embedded Linux systems. While there are numerous tools for building embedded systems, such as OpenEmbedded and BuildRoot, all of them still require a lot of pain and effort to stabilize into something usable. Hohndel wants to "raise the bar and produce a great shared base technology which is easy to build products on top of". Yocto aims to achieve this goal by directing effort to improving the build tools, providing tested releases, and improving documentation.

The most prominent component under the umbrella is the Poky build system which is a derivative of OpenEmbedded. Additionally, Yocto sponsors related projects including Pseudo: a fakeroot replacement, Swabber: a tool that detects cross-build contamination from the host filesystems, Eclipse and Anjuta plugins, and an SDK generator. Documentation, testing, and a build/test infrastructure is also covered. An complete list of projects can be found on the Yocto website.

Many questions were asked about the relationship between Poky, Yocto, and OpenEmbedded. Rather than reiterate the discussion here, an openembedded-devel mailing list posting from Purdie provides an excellent overview, and covers all of the points touched on in his presentation.

Poky is closely related to OpenEmbedded, and both projects share the BitBake build tool and much of the build package metadata. Poky is sometimes characterized as a fork of OpenEmbedded, yet it does not appear to be a competing fork since a great deal of care has been taken to maintain compatibility between the two projects, and there is a lot of cross-pollination of metadata and features. Poky differs from OpenEmbedded in its focus and policies. According to Purdie, while OpenEmbedded is a wonderful tool, it is actually quite difficult to base a product on top of it. Most companies using it need to make a large number of changes. Poky, on the other hand, aims to be a well-tested, stabilized set of build metadata in a form that requires far fewer changes for use in a commercial product.

Questions were also asked about the relationship between Yocto and MeeGo, and if the two projects were duplicating effort. The answer is simply that the two projects address completely different segments of the embedded Linux ecosystem. Where MeeGo is about a common application framework and environment with a stable ABI and a consistent interface, Yocto is instead focused on highly customized device-specific embedded systems where all the choices about what is included is up to the system builder.

Much concern was expressed about whether or not Yocto will be effectively perceived as a single vendor project since it is being championed by Intel and its subsidiary, Wind River. Ironically, Intel and Wind River employees were within the group expressing this concern. It would appear that the developers of the project deeply believe in the Yocto approach, and don't want to see it written off as an Intel-only tool. Both Hohndel and Zemlin expressed several times that they really want to see Yocto become a vendor-neutral project that improves the state of embedded Linux for everyone. For this reason, ownership of the Yocto and Poky trademarks have been transferred to the Linux Foundation, and the steering group is open to all eligible member companies.

Project governance

After lunch the meeting split into a technical track and a governance track. The governance track was focused on choosing the structure for the Yocto steering group. Prior to the summit, a governance structure was drafted to be discussed at the meeting. As much as possible, the governance structure is to take a hands-off approach to the day-to-day and technical decisions of the project, while still providing the advantages of a legal entity and to have a mechanism for managing engineering and financial resources.

A steering group will be formed with one seat held by each of the initial participating companies. In the spirit of hands-off governance, the membership agreement is low-key with primary requirements being to maintain at least one full time Yocto engineer and, in Zemlin's words, "don't be a jerk". He wants to protect against the possibility of a company joining the steering group in order to sabotage the project, and so the membership agreement provides a mechanism for removing unfriendly members. It would appear that the history lessons of SCO and UnitedLinux have been learned, and Zemlin wants to protect against the possibility of the entire project getting sabotaged by a single hostile member.

From the companies represented in the governance track, the reception was generally positive. A few people voiced concern that Yocto would be yet another embedded Linux solution to support over and above Android and OpenEmbedded. However, the close relationship between Poky and OpenEmbedded somewhat mitigates those fears and some hope was expressed that the two projects would continue to work together, and possibly even merge.

Concern was also expressed that the Linux Foundation membership requirement effectively excludes independent and community developers from holding a seat on the steering group. According to Zemlin, the intent of the membership requirements are to raise the bar so that only committed companies become involved. The steering group is completely free to waive requirements as needed to make sure that key people are not excluded from participating.

Zemlin proposed polishing up the membership agreement and sending it out to prospective members in the next week or so with the goal of forming the initial steering group by mid-December. Once the initial group is formed, the group itself can vote to change and adapt the steering requirements as needed by the project. In fact, one of the first tasks that will be put to the steering group after it is formed is to review the membership policies and make decisions about how the group should be composed. Questions like how large it should be, how many seats will be held by community developers, and how members are selected, will be addressed.

Preparing for 1.0

Meanwhile the technical track set out to spend the afternoon discussing the current state of the project and what needs to be fixed or completed before cutting a 1.0 release. A lot of time was devoted to discussing the differences between OpenEmbedded and Poky, and by the end of the day the technical track probably spent about as much time on governance issues as the governance track did.

The development model is one of the major differences between OpenEmbedded and Poky, and has been characterized as a push vs. pull model. "Push" refers to the fact that many OpenEmbedded developers have direct commit access to the source repositories. Poky has instead adopted a "pull" model where one developer maintains the canonical source tree and other developers send pull request to the maintainer of said tree (similar to how Linus Torvalds maintains his Linux tree).

OpenEmbedded and Poky also differ in policy about what recipes are accepted into the core repository. OpenEmbedded has historically taken a laissez-faire approach where anything is acceptable, as long as it doesn't break other things, and so it includes a huge number of recipes. Unfortunately it's openness is double-edged sword since many of those recipes are broken, unmaintained, or for ancient versions of software. It is easy to get new recipes into OpenEmbedded, but it is also very difficult to know which recipes actually work.

Poky on the other hand has strict policies about what is acceptable in the core and only maintains recipes that are actually used and tested. Typically Poky will not contain more than two recipes for a given package; one known working version, and an unstable testing version. It also keeps the scope limited to the core recipes required to get a working build. It is not surprising that the 700 recipes currently in Poky are an order of magnitude fewer than the 8000 in OpenEmbedded.

Instead of adding recipes to the core, Poky is extended by using "layers". Layers are a mechanism for adding additional recipes for things like board support, special toolchains, new architectures, and applications on top of the Poky core. Since anything in Poky can be overridden or extended with a layer, Poky users have lots of flexibility to adapt it to their needs. Mark Hatle commented that a typical Wind River build will consist of about 7 layers; the core, a toolchain layer, the kernel layer, a board support package layer, and one or more user-space layers.

Poky and OpenEmbedded

The push vs. pull debate came to the forefront when the idea of merging Poky with OpenEmbedded was raised. Some developers in the Yocto project would like to see Poky work more closely with OpenEmbedded, and effort has already been expended in that direction. Koen Kooi, the founder of the OpenEmbedded-based Ångström project and a Texas Instruments engineer, stated that he's prototyped using the OpenEmbedded recipes required by Ångström as a layer on top of Poky. It isn't a full OpenEmbedded port, but it does demonstrate feasibility.

Specifically, the proposed idea is to use Poky as a "common core" layer for both the Yocto and OpenEmbedded projects, and OpenEmbedded would be maintained as layers on top of Poky. The advantage would be a larger block of common functionality being used and tested by both projects, which presumably would result in a better quality core. Several OpenEmbedded developers were in the room, and while all seemed favorable to the idea, concern was expressed that the OpenEmbedded community at large would not take kindly to the change for a few reasons.

Requiring the developers to switch to a pull model for the core code is the first concern, but it ended up being easy to address. Since each layer gets its own source repository, layer maintainers can make their own decisions about development process and commit access. Plus, since anything can be overridden by a layer, an OpenEmbedded layer would be able to change anything in the core code that doesn't fit OpenEmbedded's needs.

Second was the concern that it would be viewed as a "hostile takeover" of OpenEmbedded by Poky. Much hand wringing and worry accompanied this possibility and a fair bit of time was consumed by this discussion. Politically, any such move would require the agreement of the OpenEmbedded board of directors, and have general assent from the development community at large.

There was also some discussion about the name, and whether or not using the "Poky" name would be palatable to the OpenEmbedded community, or if it would be viewed as diluting the "OpenEmbedded Brand". However, this seems to be the least of these concerns as both Poky and OpenEmbedded developers expressed flexibility on naming issues if it actually becomes an issue.

While many of the concerns raised during this discussion were important and needed to be considered, there isn't much evidence that there is actually a problem yet. The OpenEmbedded developers took an action item away from the meeting to continue discussing using Poky as the base layer for OpenEmbedded with the rest of the developer community, with the intent of having some resolution by the end of December. Since a proposal to include OpenEmbedded representation in the Yocto steering group is also on the table, the initial formation of the steering group will probably be delayed until early January to give the OpenEmbedded community time to resolve their concerns and to nominate a representative.

After that, the summit seemed to wind down and focus on technical details or the finer points of what Yocto would provide. Tim Bird's statement that he's always like the design of OpenEmbedded, but has never been able to get it to work became somewhat a mantra for the remainder of the event, and "solve Tim's problem" was often heard in relation to making Yocto a reliable and easy to use set of tools. The Yocto 1.0 roadmap was touched upon, as well as work needed on the BitBake user interface and the IDE integration features. More questions were asked to clarify details about policy and governance issues, but for the most part no more big issues came up.

A number of action items were identified as the summit wound up. The business folks from represented companies are going to get together and come back with some form of joint declaration of support for the project in the next month. The OpenEmbedded representatives will discuss with the rest of their community about closer cooperation with Poky and Yocto. Jon Masters asked if there was a regular technical conference call for the project. Hohndel replied that there is currently an Intel/Wind River call, but that the project needs to decide if continuing with a conference call is the best way to communicate.

Clearly the Yocto project is still in its infancy and there is a lot of work to be done before it can truly be considered a vendor-neutral project. However, judging from this meeting and from an informal poll of attendees afterward, the project seems to be off to a good start and it is likely to be adopted by the major embedded Linux and silicon vendors. Overall the summit ended with a positive and optimistic tone any by all indications Yocto is a project that embedded Linux engineers should be keeping their eyes on.