The 2022 embedded Linux update

A regular feature of the Embedded Linux Conference (ELC) has been an update on the state of embedded Linux from conference organizer Tim Bird. It has been quite a few years since I had the opportunity to sit in on one, so I took one at the 2022 Open Source Summit North America (OSSNA) in Austin, Texas. OSSNA is an umbrella conference that contains ELC and a whole lot more these days. Bird gave a look at recent kernel features from an embedded perspective, talked a bit about some different technology areas and their impact on embedded Linux, and also tried to answer a question that Andrew Morton posed in a keynote at ELC in 2008.

Among his many hats, Bird was the program committee chair for ELC and he is a principal software engineer at Sony Electronics. He started by saying that he was trying to squeeze a talk that he gives fairly regularly at other events down from its usual hour and a half—to 40 minutes. So he warned attendees that he would be moving fast. His goals with the talk were to introduce new technologies that have been added to the Linux kernel, so that attendees could perhaps incorporate them into their products, but also to start a conversation in the ecosystem about areas that need attention.

Past

Bird began his journey with Linux in 1993 and switched over to embedded Linux in 1998 at Lineo. He moved to Sony in 2003, where he was hired to help found the Consumer Electronics Linux Forum (CELF), which started the "Embedded Linux Technical Conference" in 2005. That is the direct predecessor to ELC. CELF moved under the Linux Foundation umbrella in 2010 as the Core Embedded Linux Project.

In the late 1990s, a lot of time and effort went into convincing companies that Linux was a good choice for embedded projects. The competition in those days was the commercial realtime operating systems (RTOSes) and "whole lot of roll-your-own bare-metal stuff that companies were doing on their own". The main architectures at the time, Intel, Arm, and MIPS, were supported by Linux, but no major chip vendor was working on the architecture support for the kernel; that work was done by volunteers and the embedded Linux companies. Distributions were either roll-your-own or came from one of a few vendors (e.g. MontaVista, Lineo, TimeSys)

One of the first things that CELF did once it got started was to analyze Linux to identify areas in need of work in order to make it suitable for embedded devices. Five specific areas were identified by CELF in 2003: system size, boot time, power management, realtime, and security. The organization then funded several kernel features in those areas (and some others, like filesystems).

So, a lot of work has been done, and it has been almost 20 years, "are we done yet?" That question came from Morton's keynote in 2008, perhaps as a bit of a joke, but it still haunts Bird to this day. Incidentally, he noted that video of the talk still exists thanks to Bootlin, which was known as Free Electrons back when the company recorded the talk (and many others in those days). Morton's talk is still relevant; "the stuff he said back then still applies today, it's amazing".

Kernel features

Then Bird went on a bit of a whirlwind tour of the last year's kernel releases from the perspective of features and changes of interest to embedded developers. There are, of course, lots of features going into each release for other use cases, which he would not be talking about. In addition, he would not be mentioning much about the different drivers or new system-on-chip (SoC) support that have also come with the new kernel releases, he said.

He began with Linux 5.14 from August 2021, which saw the addition of the osnoise and timerlat tracers that are useful for investigating realtime kernels, new Qualcomm and MediaTek drivers for a variety of devices, and a simpledrm driver that provides a direct-rendering manager (DRM) interface for simple framebuffer devices like those often found in embedded systems. The headline embedded feature in 5.14 was the addition of the memfd_secret() system call. It creates a region of memory that is inaccessible by other processes; even the kernel will have difficulty accessing it, so it is a good place to store secrets, like cryptographic keys. Bird recommended an LWN article and the commit for more information.

Kernel 5.15 in October 2021 brought sleeping spinlocks, which is a core piece of the realtime (i.e. PREEMPT_RT) patch set; he would have more to say about realtime later in the talk. The scheduler gained support for asymmetric systems, where cores on the same chip are not able to run both 32-bit and 64-bit code, so the scheduler needs to keep that in mind as it moves processes to different cores. Those working on network-attached storage (NAS) devices may want to look at the in-kernel SMB server, ksmbd, that was merged for 5.15.

Another feature that came in that kernel was printk() indexing, which gives the developers of tools that process log files a way to know when printk() messages change; that allows the tools to adapt rather than to silently fail when regular expressions suddenly fail to match. Since many embedded developers use older kernels—often provided by an SoC or board vendor—they may be surprised someday when they move to a 5.15 or later kernel because the mainline now builds with the -Werror flag by default. That flag turns compiler warnings into errors, so code that builds with warnings will cause the build to fail when it is used on more recent kernels.

In addition, the "data access monitor" (DAMON) tool was added. It provides a way to record and visualize data access patterns in the kernel. It is a diagnostic tool, like other tracers in Linux, "but is it actually more?" And, in fact, we see in 5.16 that DAMON has other functions beyond just monitoring, Bird said.

The 5.16 kernel in January 2022 also saw the addition of features for the Enhanced Read-Only File System (EROFS). In particular, it got support for multiple devices. The io_uring facility for asynchronous I/O has come on rapidly; it may be worth a look for embedded products. In 5.16, io_uring got support for security policy enforcement for SELinux and Smack. DAMON operating schemes (DAMOS) was added to help with proactive reclaim; DAMOS can offer hints to the kernel about what memory regions should be targeted. That may be particularly useful for embedded products with limited memory, he said.

In March, the 5.17 kernel was released. It can now decompress kernel modules within the kernel itself, instead of having to rely on user space to do it. This is important for certain secure boot scenarios and for the LoadPin security module. The realtime Linux analysis tool (RTLA) was also added; it uses the osnoise and timerlat tracers that were added in 5.14. In addition, some flag fields in the FUSE_INIT command for the Filesystem in Userspace (FUSE) facility have changed. So, systems using FUSE filesystems should be checked to ensure compatibility when moving to 5.17 and beyond.

The 5.18 kernel was released in May with several things of interest to the embedded community. For example, support for older Arm MMU-less architectures (ARMv4 and ARMv5) was removed. "It is not always true that the kernel just keeps getting bigger and bigger, sometimes stuff is removed." There is still support for MMU-less systems in the ARMv7 series, however. Beyond that, lots of RISC-V support was added, the tracing subsystem added user_events for dynamic tracepoints in user-space applications, and the kernel is now built with C11, rather than C89.

The big news in the upcoming 5.19 kernel is the near completion of the Arm multi-platform kernel work. Over the last 12 years, there has been an effort to consolidate all of the Arm kernels so that a single kernel image can be booted on all Arm systems just by providing a devicetree file specific to the board. There is support for a new RISC architecture, LoongArch, that was added, as well as a new hardware timestamp engine that he would be talking about further later on.

Bird spent a bit of time going over some kernel development statistics that will be familiar to LWN readers. That is because most of the stats, and many of the links he used in his slides come from LWN, he said.

Technology

Apologizing again for moving rapidly, he moved on to different technology areas that would be of interest to embedded developers. He said that another set of slides from a longer version of the talk might help fill in some of the gaps. He started with audio, noting that PipeWire is an up-and-coming project that embedded companies may want to look into, but that PulseAudio is still a big player, with a release of version 16.0 in May.

He listed a few "core kernel" changes that he had already talked about, but did expand a bit on the switch to C11. That came about because of the problems with speculative execution vulnerabilities (e.g. Spectre) and their interaction with kernel macros. It turns out that being able to declare variables inside loops, which is not possible using C89, alleviates some types of problems of that nature.

The other "big news in the core of the kernel" is the Rust for Linux project. It is currently an out-of-tree patch set of around 35,000 lines of code, which is considered experimental. The patches were recently moved to a newer version of the Rust language (1.59); "there's some grousing" about how fast the language version changes, he said. There have been a number of debates on various aspects of the project, including one going on that week on the ksummit-discuss mailing list. Most kernel developers have a "wait and see" attitude toward the project, Bird said.

For filesystems and I/O, various things are maturing, including io_uring; there are some zero-copy networking patches for io_uring that are "flying around", he said. Meanwhile, EROFS and Flash-Friendly File System (F2FS) have better support for compression and for extended attributes (xattrs).

In graphics, there are "a couple of small things and a couple of big things". In the former category are the addition of the simpledrm driver in 5.14 and that the framebuffer device (fbdev) subsystem got a new maintainer. The latter is particularly important for the embedded space where there are older devices that still need to be supported.

In the big-things department, the Mali GPUs now have a fully compliant OpenGL ES 3.1 open-source driver, called Panfrost. Also, NVIDIA has announced that it is releasing some of its code as open source. "This represents a big shift for NVIDIA". He is not sure what effect that will have on the nouveau project that has been working on open-source drivers for NVIDIA hardware, however.

"There is a steady stream of weird networking stuff going on." There is not a lot of work on new protocols and the esoteric ones that do get added are not really relevant to the embedded world, he said. He did note that the new mechanism for providing reasons for dropping packets will help in diagnosing problems of that nature.

Realtime and more

The progress with the PREEMPT_RT patch set is "really, really big news", Bird said. More and more patches for realtime Linux have been going into the mainline over the last few releases, starting with the sleeping spinlocks added in 5.15. Sleeping spinlocks allow another task to be scheduled while a lock is being held, which is how the realtime kernel can provide the low-latency guarantees. They are "probably the core feature of the PREEMPT_RT patch set".

At this point, the realtime patches have been "extensively tested and verified" over the 17-year history of the project. A natural question is: "what's left?" The remainder is about 1300 lines of code in around 50 patches. While that may sound like a lot, it actually is not; "compared to where this was before, this is unbelievable". In fact, 1700 lines of code have been merged from the patch set just since February.

In the security realm, he noted some control-flow integrity (CFI) changes that went into 5.13 to help ensure that jumps in the kernel went to places where they are expected to go. The memcpy() bounds checking went in for 5.16 to help keep memory copies from overrunning their buffers. There is also a steady stream of Spectre mitigations being added, though he was interested to note that a Spectre mitigation was removed in 5.16, as well, since it was not actually buying much in the way of increased security and it had a substantial performance cost.

In the kernel testing world, there is lots of testing going on, with many reports being made to the upstream kernel developers, coming from the 0-day test robot, KernelCI, Syzbot, and others. A new automated test framework, Compass CI, is one to keep an eye on, Bird said. It is being developed by Wu Fengguang, now at Huawei, who developed the 0-day test robot. The test suites, such as LTP, kselftest, and Kunit, all have new releases and new tests as well.

For toolchains, GCC released version 12.1 in May. It comes with better handling for uninitialized variables, along with some static-analysis features. LLVM also had a release in May (version 14.0.4). At this point, some people are using LLVM for building a whole distribution, not just the kernel, Bird said.

The hardware timestamping engine support that came in 5.18 is a nice addition to tracing features in Linux. It provides a mechanism to access hardware that can automatically attach a timestamp to a hardware event, without requiring a software interrupt, which is useful for low-overhead tracing.

That concluded his whirlwind tour, which was necessarily brief, but the idea is to give the audience some ideas of search terms they can use for various features. That should allow them to seek out more information from different sources to help as they develop new products and move to newer kernels.

Mars

Bird turned to an "interesting embedded Linux use case" that had been in the news quite a bit of late: the Mars Ingenuity helicopter. Bird said that for years he has been asking: "have we got Linux on Mars yet?; it turns out we finally did." The helicopter has some commercial off-the-shelf (COTS) hardware that is running Linux.

Ingenuity landed in February 2021 and its first five flights in April and May of that year were a "technology demonstration", but it worked so well, more flights have been done—29 in all at this point. One of those flights was able to take photos of the debris field of its "backshell" that came off as part of its descent to the surface. That was the first time NASA has been able to get such detailed pictures of a crash site, Bird said. Not only does it make for a interesting photo, but it is detailed enough to be analyzed to learn more about Mars.

Bird seemed particularly excited about the helicopter's ability to survive in the harsh environment; for example, it is now the Martian winter, and dust has reduced the amount of charging the solar panel can provide, so the device has to completely shut down at night. That means the COTS CPU is running well out of spec because it gets to -80C overnight; amusingly, some of the hardware in the helicopter came from the SparkFun hobbyist-electronics company, he said to laughter.

The cold has led to a loss of communications at times, due to clock problems, and it has also led to a broken inclinometer. NASA plans to send a software patch that will allow the helicopter to use other sensors to replace the inclinometer data. "They are going to update in the field on Mars." His slides have lots of links for more information. "I think it's one of the most interesting uses of embedded Linux in the solar system," he said with a chuckle.

Scorecards

Bird had three "scorecards" to judge the progress of embedded Linux over the years, for technology, development, and markets. Back in 2003, CELF had identified five areas that needed attention and, based on contributions to the kernel over the last few years, those areas are apparently now "done". There are few patches for system size, boot time, and power management these days, realtime is "done" as he said earlier, and security is always a work in progress. "So congratulations everybody, we did it."

Things are not quite that way, of course. There are few system-size patches anymore "because we kind of gave up". Linux is never going to run on one-cent processors, he said, so the 10-trillion IoT sensors will be running something else. A similar situation is at play for boot time; devices these days are using a variety of tricks, such as suspend/resume or going into a low-power state, to avoid booting at all. "You think your TV is off, it's not off."

Power management is more of a gray area, he said. Most of the needed features are upstream, but it is up to board vendors to actually integrate those features. Realtime is basically done, though it, like most of the rest of these areas, will require maintenance to ensure that it continues to work well.

On the development scorecard, he said that the community is doing quite well, overall. There are plenty of build system and distribution options for embedded systems, lots of different companies providing training and consulting, the toolchain support is in good shape, and the debugging options are plentiful.

Two development areas are still in progress, however. There are gaps in the test coverage, particularly in the areas of automated testing and hardware testing. In addition, while there has been enormous progress in terms of hardware support, much of the code from SoC and other vendors is not actually getting upstream. That means that companies making embedded Linux devices have to carry a lot of out-of-tree patches, thus increasing their technical debt.

Looking at markets, he had identified a non-comprehensive list of different types of embedded products, which shows that Linux is doing quite well. For example, Walmart and Amazon are working on drone deliveries using drones that run Linux; lots of commercial drones run it as well. Robots running the Linux-based Robot Operating System (ROS) are projected to be 55% of the commercial robot sales in 2024. Software in cars is largely Linux-based these days, as well.

One area that has surprised him is the use of Linux in space. He expected it to take a lot longer, but Linux use is improving quickly in that realm as well. Starlink and SpaceX use Linux, as does the Ingenuity helicopter, and some cubesats. As more commodity hardware goes into space, Linux will go with it, he thinks. Routers, mobile phones, and consumer electronics are dominated by Linux. In fact, in some areas, like TVs and DVRs, Linux has nearly 100% market share. "In consumer electronics, we're just killing it."

Overall, the embedded Linux community is "doing really well". Linux is widely deployed, at "world domination" levels. It is not going to be in everything though, and we won't see Linux running on a cereal box, perhaps ever. The core kernel has the support needed for embedded use cases, though there will always be a need for new drivers and other code to support new hardware.

He concluded that the bottom line is: "We're not done yet, but that's OK." Linux is ready to be used for lots of different kinds of devices. He hoped that his whirlwind had helped attendees get ready to design and build their next product and to hopefully become active and productive members of the ecosystem and community.

[I would like to thank LWN subscribers for supporting my travel to Austin for ELC.]