Linux in mixed-criticality systems

For the moment, this work is focused on automotive systems, which have a bunch of non-critical tasks (user interaction and displaying multimedia, for example) and critical tasks (such as autonomous driving and engine control). These tasks can be (and often are) handled with independent computers running different operating systems, but there is a lot of interest in combining these computers into one. The result, should this effort be successful, would be a system that is both cheaper and more flexible.

One way of doing this would be to turn Linux into a fully realtime system. In the past, dual-kernel approaches, such as RTLinux, RTAI, and Xenomai have been developed toward that end. More recently, the PREEMPT-RT patches have seen the most attention, though Scordino described the result as "soft realtime". The problem with all of these systems is certification for use in safety-critical settings, which is hard (if not impossible) for a kernel as large as Linux. In addition, regulations can prevent its use anyway; European automotive regulations do not allow the use of a shared system for both non-critical tasks and engine control, for example.

So attention has shifted to an alternative approach: resource partitioning that can create a hard separation between tasks on a single platform. Modern processors support this under most common architectures. If one of these systems could be successfully partitioned, it would become possible to use Linux for non-critical code and a certified operating system for the rest. This is the focus of the Hercules project, which has been funded by the European Union.

For the safety-critical side of the system, Hercules has chosen Erika Enterprise, a system that is licensed under GPLv2+. It has been designed explicitly for automotive electronic control units, and carries a number of the relevant certifications. Erika is used in production in some cars now; it supports a range of CPUs and can run under a variety of hypervisors.

On the hypervisor side, the system of choice is Jailhouse, which is also available under GPLv2. Jailhouse, too, has been designed for safety-critical applications with certification in mind; to that end, the project has a goal of not exceeding 10,000 lines of code on any architecture that it supports. The plan is to use Jailhouse to run realtime, safety-critical tasks on multicore platforms alongside Linux. It should be able to provide strong and clean isolation between the two while performing at "bare-metal levels".

Jailhouse has the concept of a "root cell" which, while being in control of the system as a whole, is not in full control of the hardware it is running on. The root cell will be running Linux. Other "cells" can run (as an "inmate") any kernel that has Jailhouse support; unlike full virtualization systems, Jailhouse is not able to run unmodified kernels. There is no scheduling built into Jailhouse; each core in the system is given over fully to one system for its use. There is no overcommitting of hardware, and no hardware emulation. Memory is partitioned between the cells, with some set aside for Jailhouse itself.

Linux systems with Jailhouse support have a special device (/dev/jailhouse) used for configuration of the root cell and the loading of inmate systems into the other cells. It uses a rather long and intimidating configuration file, written as a set of C data-structure definitions, that fully describes the hardware and its partitioning; this file can be automatically generated on x86 systems, but must be written by hand for Arm systems.

Cells in Jailhouse are isolated from each other, but they are still likely to need to communicate; Jailhouse provides a virtual PCI device for that purpose. There is no multicast capability, which is a bit of a shortcoming, so the Hercules developers have added their own communications library on top. It provides both blocking and non-blocking calls and dynamic message sizes; this code should be released soon.

One of the biggest problems that needs to be solved is avoiding interference between the cells, which can happen even with hard partitioning. Memory bandwidth and cache space can be particularly problematic. One solution for cache contention is to use cache coloring — assigning virtual addresses so that each cell uses a different portion of the cache. Another approach is to use the system's performance counters to monitor the use of memory bandwidth and cache space; tasks can then be throttled if need be to keep them within their limits. Coscheduling, wherein processes are scheduled so as to avoid contending with each other for memory, is also under development. This code, too, is expected to be released soon.

As the session ended, Thomas Gleixner observed from the audience that Hercules is "a nice fairy tale", an embodiment of the design that everybody seems to want. He also said that it is "a pipe dream", though, for a simple reason: there are no CPUs large enough to run this kind of system that have been certified for safety-critical tasks. Without a certified CPU, the system as a whole cannot be certified. Scordino responded that the vendors are working on this problem. Once they have a solution, it would appear that Hercules will be ready to run on it.

[Thanks to LinuxLab and to the Linux Foundation, LWN's travel sponsor, for supporting my travel to the event.]