Scenes from the Real Time Linux Workshop

Real time or real fast?

There is a certain amount of confusion surrounding realtime systems; most commonly, people think that it is concerned with speed. The real focus of realtime computing, though, is determinism: the fastest possible response is far less important than knowing that the system will respond within a bounded time period. In fact, realtime is often at odds with speed, especially if speed is measured in system throughput; this conflict was driven home by Paul McKenney's talk titled "Real fast or real time: how to choose." Paul concluded that one should choose the "real fast" option in a number of situations, including those where throughput is the primary consideration, virtualization is in use, or hard deadlines are not present. In other words, if realtime response is not needed, a realtime kernel should not be used - not a particularly surprising conclusion.

Interestingly, though, the "real fast" option may sometimes be best in hard-deadline situations as well. In particular, if the amount of processing which must be done within the deadline is large enough, the performance costs associated with hard realtime systems may become more of an impediment to getting the work done in time than the non-deterministic nature of general-purpose systems. The number Paul put out was 20ms; if the system must do more computing than that within each deadline cycle, it is likely to perform better on "real fast" machines. In other words, after 20ms of computation, a throughput-optimized system will have caught up enough to make up for any extra latency which might delay the start of that computation.

See Paul's paper [PDF] for more details.

Non-deterministic hardware

Determinism is generally seen as a software issue; it is expected that hardware always behaves in a consistent way. Some research [PDF] presented by Peter Okech, though, makes it clear that contemporary hardware is not as deterministic as one might think. Today's computers incorporate a great deal of complexity from many sources: multiple processors, multiple levels of caching, long instruction-processing pipelines, instruction reordering, branch prediction, system management interrupts, etc. From complexity, says Peter, comes randomness. As a demonstration of this fact, his group did extensive timings of simple instruction sequences; even after long "warmup" cycles and with interrupts disabled, these sequences never did reach a point where they would execute in a constant or predictable time.

For added fun, Peter's group coded a random number generator based on hardware non-determinism. The resulting random number sequences were then subjected to all of the tests they could come up with, from basic mean-calculation and compression tests through to full entropy computation. The results came out the same each time: instruction timings on contemporary systems are truly random. There is no real need to buy special-purpose hardware for random number generation; we are already running on such hardware. Needless to say, there are implications for anybody looking for strict determinism from their systems, especially on very small time scales.

Developers and academics

The closing event of the conference was a panel session on the disconnect between academia and the development community; the panelists were James H. Anderson, Thomas Gleixner, Hermann Härtig, Jan Kiszka, Doug Niehaus, Ismael Ripoll, and Peter Zijlstra. The problem statement asked: why are there dozens of papers on deadline schedulers, but no implementation in Linux? How can somebody get a computer science degree without learning about the problems posed by multicore processors? The actual discussion was relatively unstructured, involving numerous members of the audience, and it did not answer those specific questions. But it was interesting nonetheless.

The session opened with an invitation to the panelists to make wishes, with no real concern for practicality. Developers and academics both wished that professors could receive recognition and credit for patches which get merged into an upstream project. The current system rewards the publication of papers while ignoring practical contributions (including little details like teaching) altogether. Without an incentive to get their work upstream, researchers tend to stop working once their research reaches a publishable state.

It was noted that in some companies (Siemens was cited), employees get credit for accepted patches in much the same way they get credit for more traditional publications.

Another wish which was well received on both sides was the idea that developers and researchers should attend each others' conferences. The two groups tend to speak very different languages; for example, academics talk about "deadlines" (a set period after which the work must be done) while developers worry about "latency" (how long it takes the system to respond to an event). Given fundamental concepts that differ in this way, it is not entirely surprising that the two groups do not always communicate well. Going over to the other side and being immersed in the concerns and language found there would be helpful for everybody working in this field.

Developers asked for the publication of papers which are more easily read on their own. It is hard for busy developers to make time to read academic papers; if they have to go look up a dozen other papers to make sense of one, they are likely to just give up. The publication of more survey papers was suggested as one way to help in this area. Another was to read recent dissertations, which tend to start with relatively complete summaries of the current state of academic understanding. The hosting of summary tutorials at conferences was also suggested.

There was a request from academia for more example problems and tasks that students could take on. Also requested was an easier way to hook research code into the kernel and play with it. That might make it easier for academics to push code upstream, but not all developers are convinced that's a good idea. Instead, they say, it may be better if academics remain focused on long-term problems, with the development community adapting the best ideas for implementation and upstream merging.

[PULL QUOTE: The best thing that could happen would be that Linus Torvalds suddenly falls in love with microkernels. END QUOTE] If one gives academics the green light to be impractical, they will rarely miss the opportunity. So, it was suggested, the best thing that could happen would be that Linus Torvalds suddenly falls in love with microkernels. Thomas Gleixner could then become the maintainer of the L4 microkernel system. The underlying motivation here was not just that academics still think microkernels are better (many certainly do); it's also the simple fact that the Linux kernel has become so complex that it's getting hard for researchers to play with.

There was some lamentation that the academic community is not really producing students who are able to work with the development community. They don't know how to get code upstream. Increasingly, it seems, they don't really even know how to program - especially at the operating systems level. The academic system was charged with churning out armies of Java programmers who have little understanding of how computers actually work and have no clue of the costs of things. The result is that they go forth and create no end of highly bloated systems. The really good developers, it was claimed, tend to come from an electrical engineering background - though the prevalence of hardware engineers who churn out bad code was also noted.

Some universities have experimented with "real-world programming" courses. One of the things they have found is that registrations tend to be low - there is not a great deal of interest in taking that kind of class. There was also some special criticism directed toward the "Bologna process," which is trying to harmonize educational offerings across Europe. That process calls for reducing the standard undergraduate program to three years, which is not at all sufficient to teach people what they really need to know.

A suggestion for students who are interested in learning community development was to simply start with mailing list archives and spend some time watching how things are done. Then dive in. The community is making a real effort to avoid flaming people to a crisp these days, so jumping in is safer than it once was. But, in the end, people join the development community because they are interested in doing so; offering netiquette lessons is unlikely to inspire more of them. There are very few students who have the interest and the ability to become competent system-level programmers. It has always been that way; things have not really changed in that regard.

Internships at open source companies were suggested as a way to build both interest and experience. Such internships exist at a number of companies, though they tend to be fairly severely limited in number. What does exist, though, is the Google Summer of Code program, which is, for all practical purposes, an internship program on a massive scale. The problem here is that the kernel and realtime communities are not really organized in a way that lets them sign up to mentor summer of code students - this problem should certainly be solvable.

But none of that will help if students do not want to learn to do real development in the community. As strange as it seems, it appears to not be an entirely attractive profession. It takes years of work to become a competent engineer; many are simply unwilling to put in that time. Whether things have gotten worse because people expect instant gratification now, or whether it has always been this way was a matter of debate. One panelist suggested that things will only get better when good engineers make more money than good lawyers.

Another complaint was that universities have a certain tendency to actively block free software users. Some use proprietary virtual private network technology which is not available to Linux users. Homework submission sites which only work with Internet Explorer were also mentioned.

The session ended with little in the way of specific action items, but there was one: researchers requested a means by which they could easily experiment with new scheduling algorithms in the kernel. It was agreed that some sort of pluggable schedule technology would be added to the realtime tree, which has long served as a sort of playground for interesting new approaches. A pluggable scheduler seems unlikely to make it upstream, but presence in the realtime tree should make it sufficiently available for researchers to make use of.

The conference adjourned with the announcement of the venue for next year's event. The Real Time Linux Workshop has tended to move around more than most conferences; past events have been held all over Europe as well as China, Mexico, and the US. The 2010 Workshop will continue that practice by moving to Nairobi, Kenya, in the latter part of October. That should be an interesting place to discuss what's happening in the rapidly developing realtime Linux area.