The state of the realtime union

The 2017 Realtime Summit was held October 21 at Czech Technical University in Prague to discuss all manner of topics related to realtime Linux. Nearly two years ago, a collaborative project was formed with the goal of mainlining the realtime patch set. At the summit, project lead Thomas Gleixner reported on the progress that has been made and the plans for the future.

The Real-Time Linux (RTL) project has the goal of getting the patch set into the upstream kernel, but there is more to it than that, Gleixner said. Other facets of the project's work include documentation, establishing a community, and long-term maintenance. He often gets asked about the "community" goal; it is one of the more important pieces, actually.

Gleixner has talked with Linus Torvalds about the realtime patches along the way; Torvalds is adamant that he will not merge them if there is no community of users and developers that can maintain the code going forward. Drivers for different kinds of hardware can be merged into the mainline without impacting other parts of the kernel much or at all. The realtime patches touch many places in the core of the kernel itself, so it will be important to have a community that can handle its maintenance over the long haul.

Funding for the project for 2017 is secured, from ten member companies at three different membership levels. There are five paid developers on the project working on code, testing infrastructure, documentation, and the like. For many years, Gleixner worked on the project as something of a hobby, but it is "much more fun to get paid for things", he said.

Work on the project started around 21 months ago, he said, and much has been accomplished. The two main things that have occupied the most amount of time were the CPU hotplug rework (much of which he also detailed in a talk at Kernel Recipes) and the hotplug locking rework. He went into a bit more detail about the latter piece at the summit.

Hotplug locking

Hotplug locking caused a lot of headaches for realtime Linux. It used a counting semaphore that was mostly not covered by the lockep lock validator. Since it "evaded lockdep almost completely", it is not a surprise that there were various bugs and deadlocks hidden in that code. Gleixner removed the counting semaphore and replaced it with a per-CPU reader/writer semaphore.

That moved the hotplug locking under the purview of lockdep, which resulted in complaints that were also addressed as part of the realtime work. There are various subsystems outside of hotplug that take the hotplug lock (e.g. tracing, watchdog), which is generally where deadlocks could occur. There were some kernel developers who complained about false positives from lockdep once that change was made, which was similar to the complaints made with the recent crossrelease feature for lockdep (and those from when lockdep was first introduced); annotating the code to teach it about false positives is the right way forward. Peter Zijlstra noted that he got his start at Red Hat by doing lockdep annotations to avoid false positives, which is "how I got sucked into all of this".

The locking rework is mostly done with its "curing" now, Gleixner said. There are some watchdog problems that still need to be dealt with, however. He learned some lessons in the process, which sounded strangely familiar—unsurprisingly, no new "lessons learned" arose in the short time between his talk at Kernel Recipes (linked-to above) and the summit.

Accomplishments

He then went on to list some other accomplishments beyond the hotplug work. That includes the timer wheel rework, which allowed the CONFIG_NO_HZ and CONFIG_NO_HZ_FULL modes to start working with the realtime patches again. There is some reworking of the high-resolution timers that was posted and reviewed; there are plans to post another version soon. An overhaul of the futex and RTmutex code was done as well.

The realtime patches have long had a way to independently disable preemption and the handling of page faults, which the s390 architecture needed in mainline. So the RTL project helped s390 get it upstream. There was also a large cleanup of abuses of the task affinities; there was a lot of code doing so for "obscure reasons", Gleixner said. Lastly, the project extended the debug facilities in the kernel to catch the "abuse of various things in the kernel".

700 patches have been merged or queued for 4.15 and there are 50 patches under review. The project has fixed 40 real and 80 latent bugs. The latter are "impossible to trigger", he said, but the real ones are actually relatively easy to trigger. They just have either not been hit, due to luck, or not been reported.

He then listed the various releases that are being maintained with the realtime patch set. There are several 3.x releases (3.2, 3.10, 3.16) that are rarely updated; the 4.x series (4.1, 4.4, 4.9, and 4.14 is planned) mostly tracks the long-term stable releases of the mainline. Much of that work is done by Steven Rostedt, "and Steven does not scale", Gleixner said. Figuring out what to do about that would be one topic for the project meeting that was held on October 23.

The current development version is based on 4.13; it "sort of works". It will be abandoned in favor of the 4.14 version sometime soon.

Remaining work

The most complex remaining development tasks are locking for the dentry cache (dcache), some softirq modifications ("still some rough edges" to work out there), interaction with the memory management subsystem (pretty straightforward), and handling local locks and annotations, which is something he has wanted to get into the mainline for a long time.

Of those, the dcache locking issue is the most complex. Zijlstra pointed out that the code in question is in an area that Linus Torvalds "really cares about", so he suggested taking any proposed solutions to Torvalds first. The problem for realtime is that the dcache uses trylock loops; in the mainline, trylock is "pretty cheap", Gleixner said, but it doesn't work for realtime. If the code holding the lock gets preempted by a task on the same CPU that needs the lock, it will lead to a livelock.

Right now, the realtime patches are using a "butt-ugly workaround" of just sleeping for one millisecond if the trylock fails, which will allow the preempted task to make progress and eventually release the lock. It is the "cheapest solution for the problem" and if realtime-relevant tasks are doing "filesystem operations using the dcache, I can't help them anyway". Rostedt suggested using his "trylock and boost" approach, which Gleixner admitted worked, but said was even uglier. Rostedt replied that at least it was deterministic, thus it was "deterministically ugly", which elicited some laughter.

Gleixner said that he had been talking to some filesystem developers who think that the trylock loops may not be needed. There has been some experimentation using RCU; those experiments "kind of work" but there are corner cases that are, as yet, unsolved. This is the most challenging issue facing realtime right now, he said, so if anyone has any ideas that don't involve trylock boosting (which he does not believe can be sold to Torvalds), he would love to hear them.

He concluded his talk with a call for help. There are needs for testing and documentation. He has also recently updated the task list for the project for any developers who want to help. He asked that people "grab the code and fix it" and then to send him the patches.

Gleixner has presented various roadmaps for the realtime patch set over the years. This year's edition was textual: "Due to the evolutionary nature of Linux the roadmap will be published after the fact, but it will be a very precise roadmap." As with the others, this roadmap gives some insight into Gleixner's feelings about regularly being asked to provide one.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Prague for the Realtime Summit.]