Con Kolivas returns with a new scheduler

I said it was polite... :-)

"Configure your kernel with 1000Hz, preempt ON and disable dynamic ticks."

Any scheduler that fails to take issues like NUMA, SMP, dynamic ticks, etc. in to account while claiming to be "looking forward" will remain nothing but a toy scheduler for an insular workload. All of these have effectively become common place, to the extent that simply discounting them out of hand reads a lot more like looking back than forward, especially given that many low memory systems depend on all of these capabilities.

In any event there is anything wrong with out of tree pet projects, especially for trying out new things. If this new scheduler improves things for certain workloads, then hopefully someone will step up to work with upstream and improve things there incrementally.

What's changed is nowadays most of us have enough RAM to keep the whole kernel source and object files in cache, so there's no I/O delay after the first compile of the day.

I'm surprised Con's scheduler compiles more quickly than CFS, and that has to be worth looking into.

Linux has a number of serious deficits in terms of video drivers and API support on it's desktop that makes what Adobe does on Linux very very difficult. It's much harder for Adobe to get good and consistent performance on Linux then it is on Windows or even OS X due to a number of reasons.

-------------------------------

Here is a example.

Look at DirectX vs OpenGL + SDL. SDL and OpenGL is about the closest we have to DirectX.

As you know gaming development is now dominated by DirectX. Now.. why would developers choose DirectX over OpenGL? OpenGL is fine and can take advantage of hardware just as well when you add vendor extensions and OpenGL is cross platform. So there has to be other things that the developers find valuable over DirectX.

Now you can point out developer tools and hype/marketting and that sort of thing, which is true. However I am convinced that one of the major reasons why DirectX displaced OpenGL as the API of choice for gaming developers is because of _constistancy_.

Even in Windows each vendor has it's own OpenGL stack. Sure you get a generic one from Microsoft, but when you install proprietary drivers then you end up using their proprietary stack.

Intel, ATI, Nvidia.. all have different OpenGL implementations. And vendor extensions are different from one to another. They have different performance profiles, different effects, and so on and so forth.

So in effect when your program for OpenGL on Windows your not targetting 'windows'. Windows is a like the Genus. Each video card provides you a different species of platform.

Were as with DirectX, Microsoft forces each vendor to have a consistent hardware platform and provides the unified software stack. When you program for DirectX there is not going to be much difference between end user's computers. This dramatically lowers the cost, effort, and support overhead of making games.

So in this way one of biggest reasons DirectX wins is not because of the techology, it's because of the developers have a easier time with it.

Now fast forward to Linux....

Linux is extremely schizophrenic. It's MUCH worse then anything you'd ever face in Windows.

Open source drivers vs nvidia drivers vs ati drivers vs open source nvidia drivers vs open source ati drivers. For a huge part of the target audiance you can't even depend on there being _ANY_ acceleration at all. And what acceleration is present, all except for proprietary nvidia users, is very inconsistant, buggy, and universally slow.

API support is spotty and incomplete. The only people have any sort of constistancy and performance approaching what you can get in Windows is proprietary Nvidia drivers... and even people using nvidia hardware they have to go through extra steps to install the drivers otherwise (until very recently) they lacked any sort of 3D acceleration at all.

Look at the discusson that went on here with Gnome 3.0 and Gnome shell. You have half of everybody bitching that requiring any sort of acceleration at all is just going to break their machines and make everything perform like shit, make it buggy, use to much resources, etc etc. When in fact when you take advantage of acceleration it _SHOULD_ things _faster_ and more efficient... A composited desktop is more efficient and faster then one that is not, for example. (it's just a better design.) Yet the common user experience on Linux is almost exactly opposite.

Any computer made in the last 13 years or so has relatively decent acceleration. Enough for many tasks. Any computer made in the last 7 years or so should have the capabilities to run simple and older OpenGL games and whatnot with accepetable performance. Even IGP stuff.

Yet it's very likely that a machine sold _yesterday_, when Linux is installed, will lack the ability to play Quakelive/Quake3 with good performance.. which is based on a _open_source_ game engine that was originally released for Linux 10 years ago.

This makes doing any sort of graphics work, if your goal is to support anybody other then just proprietary Nvidia users, just pure hell on Linux.

This is primarially why Adobe Flash on Linux sucks and is why you'll see the majority of open source and indie game development is happenning on Windows.

It's interesting that Linux users like to complain about the poor quality of proprietary software implementations for their platform. What vendor actually makes *good* proprietary software for Linux? All I ever see are complaints about Nvidia, complaints about VMware, complaints about Adobe, etc.

It's possible that all of the developers at all these companies are stupid or incompetent, but I'd put forth the claim that it's simply really hard to do quality *proprietary* software for Linux. Things change way too fast, APIs break all the time, and you pretty much have to rely on the community being able to fix your code, which they can't do, because it's proprietary.

It's at least a more reasonable and less offensive position to come out and just say that proprietary software sucks (for all the usual reasons/ethos/ideology etc.), but it's annoying to see the complaints directed at these vendors with the implicit assumption that they could somehow do better while retaining their current software distribution, development, and revenue models. I don't think they can.

When I read through them at the time I thought the discussions over the scheduler were badly handled by some developers and that the outcome for Con was not really deserved. The regression I remember him arguing over seem like a rather justified thing to discuss further since it was effectively a case of whether policy should be encoded in the kernel. On the other hand, my understanding of the swap prefetch situation was that there was a lack of hard evidence in favour of it, so the handling of that situation from upstream seemed far more balanced. I think it's fair enough not to tamper with a tricky subsystem unless there's a compelling motivation for it. I'm surprised that it wasn't possible to concoct useful benchmarks for the situation, really, since it didn't actually sound that esoteric.

I did remember the scheduler thing as the straw that broke the camel's back, though, rather than the swap prefetch patch?

Maybe this is all academic anyhow. If Con's doing stuff that's interesting to him and may benefit some users, there's no harm in that - plurality like this is a major benefit of the open source model, so I guess there's not really anything to be unhappy about!

I had very good results when I used to run a Linux laptop by memlocking some critical daemons into RAM. Stuff like X and some Gnome daemons. I used a GDB script to attach to the process and force it to call memlock with ALL and FUTURE flags.

This scenario is very real - just last week I managed to crash our master
server by accidentally running about 100 Valgrind-debugged web services at
the same time. The machine went completely unresponsive for a long time, so
finally we went to the server room to reset it the hard way (even the
console would not respond at all). Linux still doesn't handle OOM situations
properly, it seems.

See http://lwn.net/Articles/350072/ and in particular the link to the University of Wisconsin paper linked within the thread - they looked at reiserfs, ext3 and JFS, and how they cope with disk failures.

Or is this a conventional hard drive model?