LWN: Comments on "Preparing for the realtime future"

Preparing for the realtime future

broonie — Tue, 15 Sep 2020 14:57:46 +0000

To be clear (with my KernelCI hat on) Greg's assessment of where KernelCI is at for testing is a bit optimistic - there's still a bunch of work to do on the reporting side for things that aren't pass/fail and integrating testsuites is not trivial.

Preparing for the realtime future

broonie — Tue, 15 Sep 2020 14:50:58 +0000

They are, it's part of what people currently look at, part of the issue here is that the test isn't something that Greg could consume directly but needs some interpretation.

Preparing for the realtime future

cemde — Mon, 14 Sep 2020 18:18:27 +0000

>> Apollo Lake that Intel equipped with special RT properties
> Where can I learn about these properties and their recommended values?
Depends on the BIOS. You may wish to contact your board and/or your BIOS manufacturer.

Preparing for the realtime future

xz — Mon, 14 Sep 2020 06:02:32 +0000

> Apollo Lake that Intel equipped with special RT properties

Where can I learn about these properties and their recommended values?

Preparing for the realtime future

cemde — Sun, 13 Sep 2020 23:04:18 +0000

Thank you for taking up the cudgels for us!

Yes, everybody wins. The continuous integration of the kernel developers on the one hand and the RT system engineering that we provide on the other hand are two different activities with a well defined handover. We start our work when a new RT kernel is available, compiles flawlessly and runs for a while with expected RT capabilities. But whether this kernel behaves as expected on a particular hardware and firmware combination in an industrial device over long periods of time is another matter. Skylake, for example, took us more than a year to get a suitable BIOS and to find out appropriate settings of BIOS variables. And even Apollo Lake that Intel equipped with special RT properties took us about six months to convince BIOS and board manufacturers to actually implement the recommended variables and let us select and configure them. Or take the RPi as another example. It didn't work with RT until OSADL took care of it and published a patch (https://www.osadl.org/Single-View.111+M5c03315dc57.0.html). You cannot expect this work to be done by the RT kernel developers, this is why you need OSADL.

We know that some of the imperfection of the current RT patches will only be solved after they will be completely merged to mainline, because this is the only way that subsystem maintainers can test their upgrade changes against RT early enough. Until then we try to do our best to support the merging activities and not to jeopardize them by submitting our "exotic errors". Instead we are searching for workarounds by not using certain configurations and functionalities. People may then use the configuration of the respective QA Farm system to create a state-of-the-art Linux RT kernel.

When some of the diagnostic and auxiliary functions that we need for our assessments but may require extra work to merge to mainline were thrown out of the official RT patches we decided to take over their maintenance (https://www.osadl.org/?id=2943).

A number of systems of our QA Farm run in so-called shadow mode. These are twin systems with identical hardware and software that are ideally suitable to test the effect of changing a single variable. We use these shadow systems often to study the effect of a particular setup on RT capabilities such as hyperthreading, graphical mode, various kernel configurations etc. and document the findings in technical assessments. Some of them have been made available to the LF Real Time Linux collaborative project on request.

In conclusion, there is nothing wrong with having separate organizations to take care of various stages and aspects of the RT patches in the interest of all of us. We are glad that users of the RT patches continue to join OSADL as members and help us grow and, thus, do even more and better for the Linux RT community.

Preparing for the realtime future

cemde — Sun, 13 Sep 2020 21:22:35 +0000

Why are you considering that we might run RT systems in a VM? All our RT systems under test run on native hardware. Anything else would not make sense. However, there is one RT system that runs a non-RT kvm guest to ensure that the guest does not interfere with the RT capabilities of the host. We do these test, since it was an OSADL funded project (executed long ago by tglx) to fix kvm induced latencies of the host.

BTW: Even if the host kernel is RT, a fully virtualized guest cannot have RT capabilities irrespective of what kernel you use.

The article(s)

corbet — Fri, 11 Sep 2020 21:20:54 +0000

That's not just one article, though...that's 16 years of history, much of which has been lovingly chronicled on these pages. Look over here and enjoy...:)

Preparing for the realtime future

ppisa — Fri, 11 Sep 2020 20:58:24 +0000

Thomas has taken part in many discussions and presented the final RT miniconf PREEMPT_RT: status and Q&A contribution.

Preparing for the realtime future

jezuch — Fri, 11 Sep 2020 20:53:31 +0000

Exactly! It'd be great to read an article describing how that incredible feat was achieved :)

Preparing for the realtime future

alison — Fri, 11 Sep 2020 16:27:34 +0000

This article alone justifies my subscription to LWN.

I see no mention of tlgx in the article! I hope that he is well.

Preparing for the realtime future

weberm — Fri, 11 Sep 2020 05:59:45 +0000

Just as glenn noted, they do run on native hardware. I was at an OSADL RT workshop and got an in-depth introduction to the farm, they actually do quite a decent job to test the RT under various workloads and introducing error conditions as well. The 3D plots of the latencies are quite nice, and regressions can easily be seen as well. I think they deserve more visibility, and with their long-term involvement in RT, I am surprised that their test-bed hasn't come up during this session. I've also notified them of maybe volunteering for some more visible testing. Which would benefit their corporate members (who, at a certain level, may have THEIR manufactured hardware tested on the QA Farm with RT) as well as the community as well as OSADL as a whole. Win-win-win.

Preparing for the realtime future

clump — Fri, 11 Sep 2020 00:16:39 +0000

I'm also happy to see this news. I'd been reading about the difficulty in getting real time merged into mainline for well over ten years.

Preparing for the realtime future

glenn — Thu, 10 Sep 2020 17:23:53 +0000

I believe that the OSADL farm is all on native hardware. Here is a list of what is currently under test: https://www.osadl.org/Linux-kernels-under-test.qa-farm-ke...

Click on a system link in the "Box" column to view a detailed report about the hardware, interrupt configuration, .config, and a latency histogram (click the link "Display most recent latency plot").

There is very little documentation on best-practices in configuring a PREEMPT_RT kernel. I frequently use the OSADL configs as a point of reference.

Preparing for the realtime future

pbonzini — Thu, 10 Sep 2020 15:35:42 +0000

Does it run in a VM? If it does, it might not pass the real-time criteria unless the host kernel is also RT.

Preparing for the realtime future

glasserc — Thu, 10 Sep 2020 14:27:09 +0000

I know this doesn't add much to the conversation, but I'm really excited to see realtime in the mainline kernel! Thanks everyone who has been working on this!

Preparing for the realtime future

weberm — Thu, 10 Sep 2020 08:02:42 +0000

How are these folks - seemingly - not aware of the OSADL tests, i.e. the QA Farm Realtime?

https://www.osadl.org/QA-Farm-Realtime.linux-real-time.0....

Preparing for the realtime future

metan — Thu, 10 Sep 2020 07:47:18 +0000

There are also some long forgotten realtime tests in LTP see https://github.com/linux-test-project/ltp/tree/master/tes...

These are mostly unmaintained and rotting, it would be nice if anyone would have a look to see if we should keep these or not, or even better help to find these testscases a new home, in a case they are useful.