Weekly Edition
Return to the Front page
| |||||||||||
|
| |||||||||||
Enterprise realtime and cooperative development
At the end of November, LWN posted a pointer to Novell's
announcement for its SUSE Linux Enterprise Realtime offering. The
resulting comments were surprisingly negative. Some readers took exception
to the language of the release - though it really is just the standard
tortured English which is seemingly required for press releases. But
others question the need for realtime response in "enterprise" settings.
Anybody who is still wondering about the value of that product will be
doubly confused now that Red Hat has announced
a realtime distribution service of its own. Clearly somebody sees a need
for deterministic response in big corporate deployments.
What's going on here is that corporate operations are, increasingly, being run by automated systems. One immediate example is in the financial trading field, where automated systems execute customer trades and, often, make the decisions to perform the trades in the first place. Often the conditions that make a particular trade advantageous last for very short periods of time - perhaps only as long as it takes for the first interested party to arrive on the scene. So predictably fast response to trading decisions is an absolute requirement. Losing too many milliseconds in the execution of an order can cost real money. It does not take much imagination to see that, as these systems become more capable, more corporate dealings will happen via automatic agents which require lightning-fast response. So enterprise realtime has the look of a growth industry. It's not surprising that the two companies most interested in selling Linux-related services into the enterprise market have announced offerings within a week of each other. What is surprising is the amount of silly sniping which has come with these releases. Consider this quote from the Red Hat side:
"Till last week, Novell sold a Real Time system that forked their
kernel," said Scott Crenshaw, VP of Red Hat's infrastructure
business unit, on a visit to London today. Last week, Novell
announced SUSE Linux Enterprise Real Time (SLERT), using open
source Linux code that Crenshaw says was "80 percent" written by
Red Hat people. "We welcome Novell to the real-time community," he
said, slightly stingingly: "We look forward to them making
contributions to it."
Note to Red Hat: this is open source, remember? Novell is shipping
tested and enterprise-hardened Linux with real time
capabilities. Just because Red Hat is again late to market (see
enterprise Linux desktop, Xen virtualization, etc.) doesn't mean
Linux contains "beta code."
Despite their competitive relationship, Linux distributors have traditionally dealt with each other in a friendly, even cooperative manner. At the development level, things are still that way: developers for a given project work together and only very rarely does anybody care about who a given developer's employer is. Developers, it seems, are more polite than managers and PR people. So who is forking the kernel? In fact, both distributors will be shipping something which is pretty far from the mainline. Back in October, LWN looked at the contents of the realtime tree, finding some 400 patches which have not yet made it into the mainline. Anybody who is shipping a true realtime kernel will have to include the bulk of those patches, and probably some others as well. In recent years, much work has been done to enable distributors to ship kernels which are much closer to the mainline, but these realtime offerings are a step in the opposite direction. They are, for all practical purposes, forked kernels.
That statement should not be taken as a criticism; there is no other way to
ship realtime Linux at this point. While much of the realtime code has
been merged, some of the deepest, most necessary components remain outside
of the mainline. The process of getting those patches merged has taken
quite a bit longer than anybody would have expected; among other things,
some of the core realtime developers have been distracted by small side
projects like the i386/x86_64 architecture merger. Until the process of
[PULL QUOTE:
Every attempt to take Unix and add hard real time to it has been a failure.
More than almost any other area of kernel development, the realtime code has been the subject of recurring debates over who deserves the credit for the work. See this LWN article from 2005 for an example. This time around, Red Hat would appear to be claiming ownership of the realtime work. In fact, much of this work, including the crucial low-level preemption work which got the current realtime effort going, was done at Red Hat. But other components have come from companies like MontaVista, Linutronix, TimeSys, and, yes, Novell (and others, of course). For these two companies to be arguing about credit is a little silly; both are clearly significant contributors to the kernel (and beyond). We may see more of this kind of talk, though. This market looks like it could be big, so the companies working in that area are going to make a serious effort to be successful there. The result may well be that Linux ends up as the dominant system in the fast-moving, agent-driven world where much of corporate operations appears to be heading. That cause will be helped, though, if the relevant managers and spokespeople take a clue from the developers who are making all of this work actually happen. We are all building this system together; pointless mud slinging can only get in the way. (Log in to post comments)
Enterprise realtime and cooperative development Posted Dec 5, 2007 14:14 UTC (Wed) by troyhebe (subscriber, #22084) [Link] Why do people always assume real-time = "lightning-fast response" and "Losing too many milliseconds in the execution of an order can cost real money.". real-time simply means deterministic worst case time. A real-time system could have a worst case time of 3 hours and still be real time. Real-time does not necessary mean fast.
Enterprise realtime and cooperative development Posted Dec 5, 2007 14:41 UTC (Wed) by nix (subscriber, #2304) [Link] Yeah, but a system with a 3 hour hard realtime bound can in practice be implemented with pencil and paper in many cases: virtually any normal OS could handle bounds like that (OK, so it couldn't *guarantee*, but it could push the probability down arbitrarily far, farther down than, say, the likelihood of hardware failure in that timespan). Hell, I've implemented hard-realtime hardware control systems with 3s bounds using mlock()ed processes on ordinary Unix platforms. (Your only problem then is priority inversion, and the easiest solution to that is to dictate what is running on the box, which is generally 'not a lot' for such systems, and to take care with the locking design of one's own program, if it includes processes running at different priorities.) The only time realtime stuff is interesting for the OS is when priority inversion really is an issue or when the realtime bounds are quite tight.
Enterprise realtime and cooperative development Posted Dec 5, 2007 20:27 UTC (Wed) by kingdon (subscriber, #4526) [Link] Well, deterministic worst case is the traditional definition, but as soon as you have memory caches, fancy CPUs (so you can't just count the cycles for each instruction and add them up), etc, life is rather fuzzier and even knowing what the worst case is can be hard. One of the things which makes the marketing (and the engineering) of this so complicated is that real time has a fairly wide variety of definitions (likewise for "embedded" and some other terms).
Enterprise realtime and cooperative development Posted Dec 5, 2007 21:47 UTC (Wed) by sobdk (subscriber, #38278) [Link] Well, because while it is true that real time means "deterministic worst case time" it is also true that real time customers want that worst case time to be as small as possible. This may mean that a RTOS may have a slower response time than a non-RTOS, but it also means that you have a guarantee you won't loose "too many milliseconds in the execution of an order" which you simply can't get from a non-RTOS.
realtime vs fast Posted Dec 6, 2007 17:52 UTC (Thu) by giraffedata (subscriber, #1954) [Link] it is also true that real time customers want that worst case time to be as small as possible I don't think that's particularly true. While it's true that computer users in general want worst case time to be as small as possible, real time people are probably less demanding than non-RT people. Say you're running an automobile's cruise control. You have to make adjustments every 100 milliseconds. It's essential that your worst case response time be less than 100 milliseconds, but every bit faster than that is just a waste of money. But in a web server, every bit faster you can serve a page is worth something.
Enterprise realtime and cooperative development Posted Dec 6, 2007 12:05 UTC (Thu) by adesharatushar@gmail.com (guest, #49449) [Link] As per my little understanding about RealTimeOs, it is more about starting the process as early as possible rather then speed of the process, isn't that the case?
realtime vs fast Posted Dec 6, 2007 18:04 UTC (Thu) by giraffedata (subscriber, #1954) [Link] As per my little understanding about RealTimeOs, it is more about starting the process as early as possible rather then speed of the process, isn't that the case? Well, almost. It's about finishing the process as early as possible. And that's true only in the big picture. In an individual system, it's not ending it as early as possible, but just ending it before some externally imposed (i.e. real time) deadline. While the OS doesn't completely control when the process finishes, it has plenty of influence beyond just controlling when it starts.
Enterprise realtime and cooperative development Posted Dec 5, 2007 15:04 UTC (Wed) by dsaxena42 (subscriber, #47380) [Link] I develop for a distro and we're all forking the kernel. No distro ships stock kernel.org as there is always some feature or HW support that is not yet upstream at the time a release needs to happen. It is completely idiotic for any distro to point fingers at another about the non-pistrine nature of their source but we're talking marketing here. Their job is to spread FUD and they will find every angle they can to attack their competitors.
Enterprise realtime and cooperative development Posted Dec 6, 2007 1:08 UTC (Thu) by aleXXX (subscriber, #2742) [Link] > No distro ships stock kernel.org I think that's not completely true, What about Slackware, and maybe Debian or Gentoo ? Alex
Enterprise realtime and cooperative development Posted Dec 6, 2007 3:26 UTC (Thu) by Velmont (subscriber, #46433) [Link] Arch Linux also ships vanilla stock kernel. And I wouldn't call «putting in a few new patches» forking.:-)
Enterprise realtime and cooperative development Posted Dec 6, 2007 4:31 UTC (Thu) by jonth (subscriber, #4008) [Link] Debian patches its kernel: http://packages.debian.org/etch/linux-patch-debian-2.6.18
Enterprise realtime and cooperative development Posted Dec 6, 2007 4:48 UTC (Thu) by terber (subscriber, #3311) [Link] > > No distro ships stock kernel.org > I think that's not completely true, What about ... Gentoo ? While Gentoo offers "vanilla-kernel" as alternative, it uses by default a patched kernel. See http://dev.gentoo.org/~dsd/genpatches for details.
Enterprise realtime and cooperative development Posted Dec 6, 2007 21:35 UTC (Thu) by dberkholz (subscriber, #23346) [Link] Not bad, only 4 patches against 2.6.24-pre
Enterprise realtime and cooperative development Posted Dec 6, 2007 19:10 UTC (Thu) by jmmc (subscriber, #34939) [Link] I've recently brought up Slackware 12.0 on my laptop. I've found no evidence of patched kernel sources anywhere with any of the provided kernels. AFAIK, Slack has been and remains 'pure' kernel.org with it's releases.
Enterprise realtime and cooperative development Posted Dec 5, 2007 15:39 UTC (Wed) by marduk (subscriber, #3831) [Link] This could potentially break Linux in the same way that it "broke" Unix. One of the problems with Unix venders is that they really tried too hard to distinguish themselves from the competition, claim to be the only "true" Unix, and basically segregate and, IMO, weaken Unix as a whole.
Enterprise realtime and cooperative development Posted Dec 5, 2007 16:52 UTC (Wed) by JoeBuck (subscriber, #2330) [Link] The analogy to the old Unix days is wrong, because in those days the extensions and forks were proprietary and incompletely documented.What we're seeing now not going to break anything. Because all of the patches are GPL, we're seeing a constant process of forking and re-merging. Tools like git make this easier to manage. Right now both Red Hat and Novell have to provide extensive patches because the fundamental stuff isn't in the kernel, but this will change. As everyone gets more experience with this code based on actual deployment, it will be easier to get the upstream kernel to accept it, and the Linus kernel will get the hard real-time capabilities now seen only in vendor kernels.
Enterprise realtime and cooperative development Posted Dec 6, 2007 4:38 UTC (Thu) by IkeTo (subscriber, #2122) [Link] > This could potentially break Linux in the same way that it "broke" Unix. Unfortunately for those Microsoft guys, this is not going to happen. Vendors successfully broken Unix because Unix is "traditionally" licensed (even though a few rather untraditional license conflicts did occur). Each can have a different version, each have no way knowing how other vendors achieve the effect they were touting, each implemented their version incompatibly, and there is no way to have the "fork" rejoin except when a vendor acquires the assets of another, which didn't happen very often really. Linux is protected against this scenario by its license. Vendors can do whatever they want to fork their own version of Linux, extend whatever, etc. But apart from very early adopters, most interested developers will tell them that it is unacceptable to use their solution until all solutions of all forks have the same interface. At that point vendors are forced to join their interface. And also, GPL gives a very easy way for this to be done, since everybody know what everybody else is doing. Somebody is going to pick the greatest features in each of them, perhaps reimplement them, and put them in mainline. Then the distinguished advantage of the vendors disappears, and they will find new things to compete on. In other words, while Unix competitions tends to hurt everybody, Linux competition tends to benefit everybody. As is written by RMS in 1992, that "Competition becomes combat when the competitors begin trying to impede each other instead of advancing themselves", and GPL stopped that.
Enterprise realtime and cooperative development Posted Dec 6, 2007 5:04 UTC (Thu) by drag (subscriber, #31333) [Link] Having the code open source has prooven this a dozen times in the past. The major example of this in kernel-land is late in the life of the 2.4 kernel and the 2.5 development series. In order to meet the demands of customers Redhat and other enterprise-ish companies backported massive amounts of code from 2.5 series back to the 2.4 and added many of their own patches. That was much more of a fork then any thing else in the kernel's lifespan. So what ended up happenning is that the kernel developers modified how the development proccess worked in order to avoid situations like that in the future. The truth of the matter is that real-time performance is not required for the vast majority of use cases for Linux. In fact having a strong real-time kernel can actually degrade performance for server and desktop workloads, as well as the normal break-stuff that large patches tend to do. Realtime stuff has been in the works for a long long time and as patches can be applied and changes made to the kernel they are done. Much of the work is actually already in the vanilla kernel and always bits and peices are incorporated into it as time and reality permits. 2.6.23 is much more realtime-ish then something like 2.6.8 in this respect, offering better performance and desktop response if configured correctly.
Enterprise realtime and cooperative development Posted Dec 6, 2007 3:36 UTC (Thu) by skitching (subscriber, #36856) [Link] I still don't understand why financial systems care about real-time. Assume two systems ready to fire a "buy" offer once a stock price drops to exactly the same value ($10.00). One is a "real time" system, and one is a standard-one with the "buy" process running at high priority. The real-time one will guarantee that the buy program runs within a fixed period (eg 100ms). But the non-realtime one will *usually* win the race. Real-time is useful for systems where leaving things too long *even once* leads to physical system damage (eg plant control), or data loss (eg realtime audio/video processing). But in system where the one who completes a process *first* *on average* wins, and where the occasional miss can be tolerated, why use real-time?
Enterprise realtime and cooperative development Posted Dec 6, 2007 5:20 UTC (Thu) by stephend (subscriber, #8045) [Link] The problem is you can't really work on averages. Trading activity isn't distributed evenly throughout the day. Chances are on a non-real-time system your latencies would increase when the markets were busiest -- just the time you can't afford to lose opportunities.
Averages DO matter, just in different place Posted Dec 6, 2007 14:28 UTC (Thu) by khim (subscriber, #9252) [Link] The keyword here is "grid". As long as rules were as simple as "buy if less then $10" non-realtime computers worked great. Today we have 1000 computers which calculate complex models. So we have something like this: 990 computers have lighting-fast response (10-50ms, for example), 5 have the same 100ms as realtime computers and the last 5 have latency 200ms. Realtime grid wins. Next round: 995 computers finished in 50ms, 3 in 100 ms, one in 150ms and one in 200ms. Still realtime grid wins. Once per blue moon you have situation when non-deteministing-but-usually-fast response produces answer in less then 100ms and non-realtime grid wins, but in general - you NEED soft realtime there...
Enterprise realtime and cooperative development Posted Dec 6, 2007 7:56 UTC (Thu) by niv (subscriber, #8656) [Link] "The real-time one will guarantee that the buy program runs within a fixed period (eg 100ms). But the non-realtime one will *usually* win the race." Just out of curiosity, why do say that the non-realtime one will usually win? Are you assuming real-time guarantees come at the _expense_ of latency? Isn't it usually the other way around?
Enterprise realtime and cooperative development Posted Dec 6, 2007 10:55 UTC (Thu) by lmb (subscriber, #39048) [Link] Lower latency guarantees usually come at the cost of lower throughput. However, you can scale up throughput by increasing the CPU power you have available (within Amdahl's Law, etc); the same is not, necessarily, true for latency.
Enterprise realtime and cooperative development Posted Dec 7, 2007 16:43 UTC (Fri) by Tet (subscriber, #5433) [Link] I still don't understand why financial systems care about real-time.They don't. As someone that writes trading systems for a living, I can assure you that none of them are real time. The trick is to be as fast as possible, not to have a bounded worst case. As you say, the occasional miss can be tolerated.
Enterprise realtime and cooperative development Posted Dec 8, 2007 4:20 UTC (Sat) by nix (subscriber, #2304) [Link] And some of them are *way* slower than being discussed here. A goodly number of organizations are still expecting people to be impressed that they can get trades out on the same *day* they went in. (I boggle, too, but that's the way it is.)
Enterprise realtime and cooperative development Posted Dec 11, 2007 18:31 UTC (Tue) by da4089 (subscriber, #1195) [Link] They don't care about real real-time systems (or they'd all use VxWorks or QNX or whatever). They care about low latency, which has come to be described as "enterprise real-time". Some of the concerns are similar, but trading platforms care a lot more about minimising latency, rather than just bounded latency.
Realtime is also needed for multimedia Posted Dec 6, 2007 11:27 UTC (Thu) by ssavitzky (subscriber, #2855) [Link] I'm not an enterprise, I'm an amateur musician with a bedroom studio. If I don't have a low-latency kernel (obtainable from, e.g., 64studio or ubuntustudio) I can't record audio without dropouts.
Realtime is also needed for multimedia Posted Dec 6, 2007 18:22 UTC (Thu) by giraffedata (subscriber, #1954) [Link] If I don't have a low-latency kernel (obtainable from, e.g., 64studio or ubuntustudio) I can't record audio without dropouts. s/low-latency/bounded-latency These kernels have greater latency on average than kernel.org. They just have better worst-case latency.
Realtime is also needed for multimedia Posted Dec 6, 2007 21:13 UTC (Thu) by ssavitzky (subscriber, #2855) [Link] That would be more nearly correct, though whether they're better or worse on average probably depends on the exact workload. I believe that, originally, they really did have lower latency; many of those patches have made it into the mainstream kernels because they also improved performance on SMP systems by making more of the kernel pre-emptable.
Realtime is also needed for multimedia Posted Dec 7, 2007 9:50 UTC (Fri) by niv (subscriber, #8656) [Link] Do you futz around with the priority of your application, though? I'd be interested in hearing what your experience was in terms of latencies.
Realtime is also needed for multimedia Posted Dec 7, 2007 21:16 UTC (Fri) by ssavitzky (subscriber, #2855) [Link] I don't normally tweak priorities, but I don't usually have anything running when I'm recording except Audacity, an xterm, and X. I've found that you want to use a separate video card rather than one that shares main memory, and you want to turn off the DRI (Direct Rendering Interface) in the X server config file. Recording onto an NFS-mounted filesystem is dicey.
Realtime is also needed for multimedia Posted Dec 10, 2007 18:55 UTC (Mon) by RobertBrockway (subscriber, #48927) [Link] Perhaps you don't want to run X on the box at all. X is network transparent so you can display Audacity to another box on the network. Yes this means involving another box (or a thin client) but it _might_ give better performance.
Realtime is also needed for multimedia Posted Dec 11, 2007 7:31 UTC (Tue) by ssavitzky (subscriber, #2855) [Link] I actually recorded about 2/3 of my CD that way -- it worked, but there was a noticable lag that made it less responsive for editing and overdubbing. My current recording box is a dual-core CPU, which gives me the best of both worlds.
Enterprise realtime and cooperative development Posted Dec 10, 2007 18:52 UTC (Mon) by RobertBrockway (subscriber, #48927) [Link] I disagree with the claim that these are forked kernels. They would only be forked kernels if there was no intention or attempt to have the patches wrapped back into mainline. People run kernels with lots of custom patches all the time but they aren't kernel forks.
What is real-time? Posted Dec 13, 2007 7:47 UTC (Thu) by dmag (subscriber, #17775) [Link] "Real-time" is ambiguous. Either your task is HARD real-time or SOFT real-time. Everyone wants to think their task is hard real-time, but most likely they're wrong. Hard Real-time
Soft Real-time
|
|||||||||||