Kernel development

Kernel release status

Stable updates: 3.14.4, 3.10.40, and 3.4.90 were released on May 13.

Comments (1 posted)

Quotes of the week

Comments (2 posted)

Braking CPU hotplug

The subsystem in question is the one that handles CPU hotplugging. There are a number of reasons why one might want to add a CPU to (or remove a CPU from) a running system: the hardware may actually support physical addition or removal of CPUs, or one might want to offline a misbehaving processor. In the virtualization world, CPU hotplugging is an obvious way to adjust the processing power available to specific guests as they run. The feature clearly has value, and nobody would seriously suggest removing it. But nobody is happy with how CPU hotplugging has been implemented.

Changing the set of CPUs in a running system is a complex task; there is a vast amount of per-CPU state at numerous levels that must be managed. In such a situation, it makes sense to have an overall mechanism that manages the complexity, breaks it down into simple steps, and ensures that those steps are run in the correct order. Unfortunately, the Linux kernel does not have that sort of mechanism; instead, it has a confusing array of notifiers and callbacks that is hard to reason about or make changes to. And, unsurprisingly, it has bugs; developers who have gone looking for bugs in this area have had little trouble finding them.

In fact, bugs in this area are so plentiful that Borislav Petkov wants to make them harder to find. His patch introduction reads:

His solution was simple: insert a one-second delay into each CPU hotplug operation. Slowing things in that way minimizes the number of operations that can be tested and the amount of concurrency between operations. It should reduce the flow of bug reports nicely.

There is one tiny little problem, of course: this patch does not actually fix any bugs; it just hides them from view. Andrew Morton was quick to point out that this patch would almost certainly result in fewer CPU hotplug bugs being fixed. But Thomas Gleixner thinks that may be a good thing: "if people would have spent the same amount of time to rewrite the hotplug mess, we would have a way bigger benefit. But no, we prefer to add more layers of duct tape and bandaid hackery to it."

Thomas, of course, has in the past tried to do just that sort of rewrite; that work was covered here in February 2013. He took the time to break the hotplug and hotunplug operations down into a long list of discrete steps; he then built a system to run those steps in a well-defined order. It was a far cry from a full solution to the problem; most of the existing hotplug code remained in place and was just called differently. But it provided a framework on which a more complete rewrite could be done over time.

The only problem is: nobody did that rewrite. Thomas ran out of time and moved back to other tasks, and nobody else picked up the work, so the patches have languished since their initial posting. The work that has been done in that area, instead, is the application of increasingly complex bug fixes (recent example) as problems turn up and developers try to make the existing implementation work. These fixes may address specific bugs, but they do not address the complexity and unmaintainability of the system as a whole; indeed, they tend to make those problems worse.

It is frustration with the addition of more "duct tape and brown paper bags" that led to Borislav's patch to slow the hotplug system down. In the end, the developers who have to work with this part of the code don't want more bug fixes; they want the code to be made simpler and easier to understand so that, in the end, there will not be a need for an endless stream of fixes that just add more complexity to the code. Making it harder to find bugs in this subsystem is a heavy-handed way of trying to direct developers' attention elsewhere.

Naturally, the patch is more of a statement than a serious attempt to change the kernel; it would be surprising if this patch were merged. In a world where kernel subsystem maintainers cannot force developers to work on a specific area, and where no company managers have seen fit to direct their employees to solve the CPU hotplug problem, one has to be creative sometimes to get things done. One might hope that this patch posting would be a strong enough hint to get somebody to work on the problem. Unfortunately, Thomas may have inadvertently sabotaged that effort by saying that, if nobody else gets around to rewriting the hotplug CPU subsystem, he will jump back in and do it himself.

Comments (1 posted)

Supporting Allwinner SoCs

Support for Allwinner system-on-chips (SoCs) made news last June due, at least in part, to some confusion over whether there would ever be support for those chips in the mainline kernel. As it turned out, some support had already been merged at that point, with more to come. Maxime Ripard has been spearheading that work; he reported on the progress and plans in a talk at the 2014 Embedded Linux Conference.

Ripard works as an embedded Linux engineer and trainer for Free Electrons, and lives in Toulouse, France. He has worked on kernel support for the Allwinner sunXi SoCs, as well as on the Buildroot build system and the Barebox bootloader.

Allwinner

After introducing himself, Ripard introduced Allwinner and its SoCs. Allwinner is a fabless SoC vendor that was founded in Zhuhai, China in 2007. All of its products have been ARM-based, starting with ARM9 SoCs that ran Allwinner's own operating system (MeliOS), not Linux. The company specializes in multimedia SoCs, he said.

No one really noticed Allwinner until its SoCs started showing up in tablets. He put up a pie chart (slide #5 in his slides [PDF]) that showed Allwinner as second only to Apple in tablet market share, with 15%. In fact, Allwinner has as much share as TI and NVIDIA combined, and it is getting stronger, Ripard said. Its SoCs are "in every cheap tablet you can buy".

Allwinner has a number of different SoCs in its lineup, running the gamut from single-core Cortex-A8-based SoCs (sun4i, sun5i) all the way up to the upcoming A80 (which may be called the sun9i) that is a big.LITTLE system with four Cortex-A15s and four Cortex-A7s. The A80 is scheduled to ship anytime now, he said, and "should be quite interesting".

There are two hardware vendors making low-priced development boards for Allwinner SoCs. Olimex has the OLinuXino, which is open-source hardware. The company has been quite supportive of the open-source community by providing free samples and prototypes. The other is Cubieboard, which has cheaper, non-open-source hardware. There are other Allwinner-based boards out there as well, Ripard said, including pcDuino, MarsBoard, and BananaPi.

Allwinner's code

So there is good hardware support for Allwinner SoCs, but "on the software side, things were a bit less shiny", he said. The available kernels were Allwinner's 3.0-based kernel for single-core SoCs or a 3.3-based version for multi-core Cortex-A7-based SoCs. Each SoC has its own kernel tree, with some drivers that are completely different between the trees even though the underlying hardware is the same. Given the kernel versions in question, though, the trees are in reasonable shape; it is "messy code, but still getting somewhere".

But if you look at those kernels from the point of view of the 3.15 kernel, there is a lot of outdated code. For example, there is no use of or support for the common clock framework, pin control (pinctrl), or dmaengine. Allwinner has its own code to handle all of those pieces. There is no consolidation between the different SoCs and customers are stuck with those kernel versions and features. And then there is FEX ...

FEX is a binary file that is compiled from a human-readable text file. It provides much of the same information that comes from device tree (DT), including what devices are enabled, memory timings, clock frequencies, pin multiplexing, and so on. It is used in the boot chain to support a single boot image for multiple boards.

DT and FEX are an example of parallel evolution, Ripard said. In fact, FEX is evidence that the idea behind DT is sound, since FEX serves much the same role with a different implementation. But FEX has some deficiencies. There is no way to say which driver is needed in FEX, so every driver needs to be enabled in the kernel. Drivers have to consult the FEX file to see whether they should be probed or not. The logic is backward compared to DT, so FEX doesn't really scale to more than a single SoC. It is useful for Allwinner, he said, but not useful for Linux in general.

To emphasize that, Ripard listed the pros and cons of FEX. It has most of the pros of DT: it allows for a generic kernel image and it separates the hardware description from the code. It also allows even non-kernel-developers to quickly get to a kernel that boots on the hardware. On the con side, though, FEX is completely non-standard and only works for a single platform. Essentially, it is not generic enough.

But FEX had its "moment of glory" in June 2013, Ripard said. That is when there was something of a disagreement about Allwinner support in the kernel, with Luke Leighton claiming it would never happen without FEX support. But Ripard and others had already started working on DT-based Allwinner support nine months earlier. In fact, as part of the discussion, Leighton showed why FEX was unsuitable for more than just a single SoC—it was not possible to support a driver for new hardware in Allwinner's FEX-only kernels.

Community support

The community efforts to support Allwinner SoCs began in 2010 around the EOMA-68 devices from Rhombus-Tech. That community eventually split in 2011, with the most active branch moving to linux-sunxi.org. That community is "very active these days" and not just on the kernel. It is, however, made up of "free software extremists", Ripard said, who refuse to even look at SoCs with non-free GPUs, for example.

The linux-sunxi community has done a lot of useful work, though. It has created hardware and board support package (BSP) documentation as well as various tools. It has also reverse-engineered some of the hardware, including USB, NAND, and H.264 video support in some SoCs. It maintains a fork of Allwinner's BSP, but supports all SoCs in a single kernel. That kernel is 3.4-based, which is "not that useful any more".

The pros of the linux-sunxi output are that it maintains code that Allwinner does not, it also allows users to put Android, Fedora, or Buildroot onto a cheap tablet. The code is "slightly better" than Allwinner's as well. On the con side, though, the kernel is "stuck in the past" and the community is not taking a long-term view. It just keeps working on the same code base without considering getting changes upstream to reduce its burden over time.

Mainlining

So, in late 2012, he started working on mainlining support for Allwinner SoCs. The first code, for the Allwinner A13, was merged into 3.8, though it didn't do much other than boot to user space, he said. Other features and support for other SoCs followed quickly. Early on, it was mostly Ripard and Emilio López contributing, but now there are lots more contributors, some of whom are quite active. That is a good sign that the mainline Allwinner support "is gaining traction".

One of the first challenges was to rewrite most of the core drivers to use newer infrastructure like the common clock framework and dmaengine. Most of that was completed by 3.13 or 3.14, Ripard said. Another task was to move away from FEX and to start using DT. In addition, the code needed some cleanup to make it fit for submission. All in all, it was "similar to every mainlining process you might have heard of".

Losing FEX support, though, meant that a generic kernel could not be booted on every board available any more. The solution was a runtime FEX-to-DT translator called sunxi-babelfish. It behaves like a normal U-Boot kernel image, but instead reads the FEX file and constructs a DT file that gets passed to the DT-based kernel that is next in the boot progression. So mainline kernels can now boot on most available boards and tablets.

All of the core kernel code is working now, except for DMA on some older SoCs. The DMA support for more recent SoCs has not yet been merged, but has been submitted. Virtualization support for Xen and KVM is being worked on. Networking is working, as is SATA support. MMC and NAND support are in the review process right now. Most everything needed for a headless device is available; it is the "user-friendly pieces" (such as display, audio, and touchscreen drivers) that are missing.

Allwinner's latest kernel is still 3.3-based, but it is using some of the code from the mainline (for the clock framework, pinctrl, and dmaengine, for example). It is still using FEX and is "a long way" from using DT, he said. The company just joined Linaro, which is an acknowledgment that the mainline exists and that the company wants to be involved in it somehow.

The piece that remains is "all the hard work", Ripard said. That includes audio and video drivers, which have been started and are being worked on as part of a Google Summer of Code project. There is also work needed to support the newer SoCs (A23 and A80). The development boards are affordable (as are the tablets), which makes it easy to get started. There are "plenty of things to do".

Ripard learned a number of lessons along the way. He started the project as a way to learn about kernel development and it was a great way to do that. He recommended that others in a similar spot take their time and to not be in a big hurry. "Eventually the maintainer will review" your patches, he said. At times, he was "a bit too eager" to trash the existing code and start from scratch; it is sometimes better to clean up what is already there.

In all, Ripard's presentation showed a healthy community taking the proper approach to supporting an SoC family in Linux. While there is certainly nothing wrong with the approaches taken by Allwinner and linux-sunxi, they were never going to result in mainline support for the popular SoCs. Should Allwinner and/or linux-sunxi want to switch to the mainline kernel somewhere down the road, the work that Ripard and others have done will be there waiting for them.

Comments (3 posted)