Real Time Linux Workshop
was held in Dresden, Germany, at
the end of September; it was attended by some 200 researchers and developers
working in that area. RTLWS was a well-organized event, with engaged
participants, interesting topics, and more than adequate amounts of German
beer. This article will be concerned with three sessions from that event;
other topics (deadline schedulers in particular) will be looked at separately.
Real time or real fast?
There is a certain amount of confusion surrounding realtime systems; most
commonly, people think that it is concerned with speed. The real focus of
realtime computing, though, is determinism: the fastest possible response
is far less important than knowing that the system will respond within a
bounded time period. In fact, realtime is often at odds with speed,
especially if speed is measured in system throughput; this conflict was
driven home by Paul McKenney's talk titled "Real fast or real time: how to
choose." Paul concluded that one should choose the "real fast" option in a
number of situations, including those where throughput is the primary
consideration, virtualization is in use, or hard deadlines are not
present. In other words, if realtime response is not needed, a realtime
kernel should not be used - not a particularly surprising conclusion.
Interestingly, though, the "real fast" option may sometimes be best in
hard-deadline situations as well. In particular, if the amount of
processing which must be done within the deadline is large enough, the
performance costs associated with hard realtime systems may become more of
an impediment to getting the work done in time than the non-deterministic
nature of general-purpose systems. The number Paul put out was 20ms; if
the system must do more computing than that within each deadline cycle, it
is likely to perform better on "real fast" machines. In other words, after
20ms of computation, a throughput-optimized system will have caught up
enough to make up for any extra latency which might delay the start of that
See Paul's paper [PDF] for
Determinism is generally seen as a software issue; it is expected that hardware
always behaves in a consistent way. Some research [PDF]
presented by Peter Okech,
though, makes it clear that contemporary hardware is not as deterministic
as one might think. Today's computers incorporate a great deal of
complexity from many sources: multiple processors, multiple levels of
instruction-processing pipelines, instruction reordering, branch
prediction, system management interrupts, etc. From complexity, says
Peter, comes randomness. As a
demonstration of this fact, his group did extensive timings of simple
instruction sequences; even after long "warmup" cycles and with interrupts
disabled, these sequences
never did reach a point where they would execute in a constant or
For added fun, Peter's group coded a random number generator based on
hardware non-determinism. The resulting random number sequences were then
subjected to all of the tests they could come up with, from basic
mean-calculation and compression tests through to full entropy computation.
results came out the same each time: instruction timings on contemporary
systems are truly random. There is no real need to buy special-purpose
hardware for random number generation; we are already running on such
hardware. Needless to say, there are implications for anybody looking for
strict determinism from their systems, especially on very small time scales.
Developers and academics
The closing event of the conference was a panel session on the disconnect
between academia and the development community; the panelists were James
H. Anderson, Thomas Gleixner, Hermann Härtig, Jan Kiszka, Doug
Niehaus, Ismael Ripoll, and Peter Zijlstra. The problem statement
asked: why are there dozens of papers on deadline schedulers, but no
implementation in Linux? How can somebody get a computer science degree
without learning about the problems posed by multicore processors? The
actual discussion was relatively unstructured, involving numerous members
of the audience, and it did not answer those specific questions. But it
was interesting nonetheless.
The session opened with an invitation to the panelists to make wishes, with
no real concern
for practicality. Developers and academics both wished that professors
could receive recognition and credit for patches which get merged into an
upstream project. The current system rewards the publication of papers
while ignoring practical contributions (including little details like
Without an incentive to get their work upstream, researchers tend to stop
working once their research reaches a publishable state.
It was noted that in some companies (Siemens was cited), employees get
credit for accepted patches in much the same way they get credit for
more traditional publications.
Another wish which was well received on both sides was the idea that
developers and researchers should attend each others' conferences. The two
groups tend to speak very different languages; for example, academics talk
about "deadlines" (a set period after which the work must be done) while
developers worry about "latency" (how long it takes the system to respond
to an event). Given fundamental concepts that differ in this way, it is not
entirely surprising that the two groups do not always communicate well.
Going over to the
other side and being immersed in the concerns and language found there
would be helpful for everybody working in this field.
Developers asked for the publication of papers which are more easily read on
their own. It is hard for busy developers to make time to read academic
papers; if they have to go look up a dozen other papers to make sense of
one, they are likely to just give up. The publication of more survey
papers was suggested as one way to help in this area. Another was to read
recent dissertations, which tend to start with relatively complete
summaries of the current state of academic understanding. The hosting of
summary tutorials at conferences was also suggested.
There was a request from academia for more example problems and tasks that
students could take on. Also requested was an easier way to hook research
code into the kernel and play with it. That might make it easier for
academics to push code upstream, but not all developers are convinced
that's a good idea. Instead, they say, it may be better if academics remain
focused on long-term problems, with the development community adapting the
best ideas for implementation and upstream merging.
The best thing that could
happen would be that Linus Torvalds suddenly falls in love with
If one gives academics the green light to be impractical, they will rarely
miss the opportunity. So, it was suggested, the best thing that could
happen would be that Linus Torvalds suddenly falls in love with
microkernels. Thomas Gleixner could then become the maintainer of the L4 microkernel system. The
underlying motivation here was not just that academics still think
microkernels are better (many certainly do); it's also the simple fact that
the Linux kernel has become so complex that it's getting hard for
researchers to play with.
There was some lamentation that the academic community is not really
producing students who are able to work with the development community.
They don't know how to get code upstream. Increasingly, it seems, they
don't really even know how to program - especially at the operating
systems level. The academic system was charged with churning out armies of
Java programmers who have little understanding of how computers actually
work and have no clue of the costs of things. The result is that they go
forth and create no end of highly bloated systems. The really good
developers, it was claimed, tend to come from an electrical engineering
background - though the prevalence of hardware engineers who churn out bad
code was also noted.
Some universities have experimented with "real-world programming" courses.
One of the things they have found is that registrations tend to be low -
there is not a great deal of interest in taking that kind of class. There
was also some special criticism directed toward the "Bologna process,"
which is trying to harmonize educational offerings across Europe. That
process calls for reducing the standard undergraduate program to three
years, which is not at all sufficient to teach people what they really need
A suggestion for students who are interested in learning community
development was to simply start with mailing list archives and spend some
time watching how things are done. Then dive in. The community is making
a real effort to avoid flaming people to a crisp these days, so jumping in
is safer than it once was. But, in the end, people join the development
community because they are interested in doing so; offering netiquette
lessons is unlikely to inspire more of them. There are very few students
who have the interest and the ability to become competent system-level
programmers. It has always been that way; things have not really changed
in that regard.
Internships at open source companies were suggested as a way to build both
interest and experience. Such internships exist at a number of companies,
though they tend to be fairly severely limited in number. What does exist,
though, is the Google Summer of Code program, which is, for all practical
purposes, an internship program on a massive scale. The problem here is
that the kernel and realtime communities are not really organized in a way
that lets them sign up to mentor summer of code students - this problem
should certainly be solvable.
But none of that will help if students do not want to learn to do real
development in the community. As strange as it seems, it appears to not be
an entirely attractive profession. It takes years of work to become a
competent engineer; many are simply unwilling to put in that time. Whether
things have gotten worse because people expect instant gratification now,
or whether it has always been this way was a matter of debate. One
panelist suggested that things will only get better when good engineers
make more money than good lawyers.
Another complaint was that universities have a certain tendency to actively
block free software users. Some use proprietary virtual private network
technology which is not available to Linux users. Homework submission
sites which only work with Internet Explorer were also mentioned.
The session ended with little in the way of specific action items, but
there was one: researchers requested a means by which they could easily
experiment with new scheduling algorithms in the kernel. It was agreed
that some sort of pluggable schedule technology would be added to the
realtime tree, which has long served as a sort of playground for
interesting new approaches. A pluggable scheduler seems unlikely to make
it upstream, but presence in the realtime tree should make it sufficiently
available for researchers to make use of.
The conference adjourned with the announcement of the venue for next year's
event. The Real Time Linux Workshop has tended to move around more than
most conferences; past events have been held all over Europe as well as
China, Mexico, and the US. The 2010 Workshop will continue that practice
by moving to Nairobi, Kenya, in the latter part of October. That should be
an interesting place to discuss what's happening in the rapidly developing
realtime Linux area.
to post comments)