LWN: Comments on "Con Kolivas returns with a new scheduler"

Con Kolivas returns with a new scheduler

daenzer — Thu, 17 Sep 2009 10:05:27 +0000

Do you have any form of kernel preemption enabled? IME that helps a lot for interactivity vs. swap. But of course, there's only so much it can do in the face of a runaway process which eats memory like there's no tomorrow.

Con Kolivas returns with a new scheduler

jbh — Fri, 11 Sep 2009 15:45:40 +0000

Good question! The desired property here is mainly "not CFQ", since the
default CFQ scheduler seems to behave badly (wrt IO latency) with cheap
SSDs. See for example http://forum.eeeuser.com/viewtopic.php?id=23580 .

The noop scheduler is also sometimes recommended, but the linked thread
suggests it may suffer from IO starvation since it doesn't do any balancing
between processes.

[ it also recommends the following, which I haven't tried:
echo 1 > /sys/block/sda/queue/iosched/fifo_batch
]

Con Kolivas returns with a new scheduler

SEMW — Fri, 11 Sep 2009 14:54:59 +0000

> - barrier=0 mount option
> - elevator=deadline boot option
>- tune2fs -o journal_data_writeback

Regarding the Deadline i/o scheduler, according to Wikipedia, "The kernel docs suggest this is the preferred scheduler for database systems, especially if you have TCQ aware disks, or any system with high disk performance.". Since an ssd on an eeepc is neither a database system, nor TCQ aware (TCQ being meaningless for SSDs), nor has high disk performance, what property makes it good for that workload?

Con Kolivas returns with a new scheduler

realnc — Wed, 09 Sep 2009 20:28:06 +0000

Ingo posted a new version of latt.c that when used, reports better values than BFS.

Normally I'd say this smells fishy since it comes from Ingo, but I'm sure other devs would have objected if he would modify latt.c in any malicious way.

Con Kolivas returns with a new scheduler

kmike — Wed, 09 Sep 2009 15:19:49 +0000

I'm kind of late to the party, but I was just reading the linked thread at LKML. Just check out these numbers (post by Jens Axboe):

BFS210 runs on the laptop (dual core intel core duo). With make -j4
running, I clock the following latt -c8 'sleep 10' latencies:

-rc9

        Max                17895 usec
        Avg                 8028 usec
        Stdev               5948 usec
        Stdev mean           405 usec

        Max                17896 usec
        Avg                 4951 usec
        Stdev               6278 usec
        Stdev mean           427 usec

        Max                17885 usec
        Avg                 5526 usec
        Stdev               6819 usec
        Stdev mean           464 usec

-rc9 + mike

        Max                 6061 usec
        Avg                 3797 usec
        Stdev               1726 usec
        Stdev mean           117 usec

        Max                 5122 usec
        Avg                 3958 usec
        Stdev               1697 usec
        Stdev mean           115 usec

        Max                 6691 usec
        Avg                 2130 usec
        Stdev               2165 usec
        Stdev mean           147 usec

-rc9 + bfs210

        Max                   92 usec
        Avg                   27 usec
        Stdev                 19 usec
        Stdev mean             1 usec

        Max                   80 usec
        Avg                   23 usec
        Stdev                 15 usec
        Stdev mean             1 usec

        Max                   97 usec
        Avg                   27 usec
        Stdev                 21 usec
        Stdev mean             1 usec

One thing I also noticed is that when I have logged in, I run xmodmap
manually to load some keymappings (I always tell myself to add this to
the log in scripts, but I suspend/resume this laptop for weeks at the
time and forget before the next boot). With the stock kernel, xmodmap
will halt X updates and take forever to run. With BFS, it returned
instantly. As I would expect.

So the BFS design may be lacking in the scalability end (which is
obviously true, if you look at the code), but I can understand the
appeal of the scheduler for "normal" desktop people.

Amazing difference. I'm tempted to try this thing out.

Con Kolivas returns with a new scheduler

jbh — Wed, 09 Sep 2009 08:47:35 +0000

Ah, then we're in full agreement. An SSD will in fact *cause* the problems
described in this thread.

;-)

Con Kolivas returns with a new scheduler

realnc — Wed, 09 Sep 2009 08:41:16 +0000

This was related to the original comment by kragil you replied to. An SSD won't fix the problems described in this thread.

Con Kolivas returns with a new scheduler

jbh — Wed, 09 Sep 2009 07:43:15 +0000

Huh? I'm not sure what you mean, I already have an SSD. That's the problem: it's very slow, random writes <100kB/s. And I strongly doubt that BFS can magically fix that.

What CAN fix it is a combination of: (i) write less, (ii) don't wait for writes. That's the point of the tuning. Nothing to do with process scheduling.

Con Kolivas returns with a new scheduler

realnc — Tue, 08 Sep 2009 20:11:13 +0000

You don't need an SSD. You need BFS. Cheaper and works wonders.

Con Kolivas returns with a new scheduler

jlokier — Mon, 07 Sep 2009 10:09:59 +0000

The real reason we used to use "make -j MORE_THAN_NR_CPUS" was to get useful work done when a process waits on I/O. Con's scheduler doesn't help with that: if gcc foo.c needs to read foo.c from disk, you must have more jobs than CPUs, or you have an idle CPU.

What's changed is nowadays most of us have enough RAM to keep the whole kernel source and object files in cache, so there's no I/O delay after the first compile of the day.

I'm surprised Con's scheduler compiles more quickly than CFS, and that has to be worth looking into.

Con Kolivas returns with a new scheduler

jbh — Mon, 07 Sep 2009 08:56:39 +0000

Right! You probably have most things covered. Make sure you use the fast
ssd for the system of course, and save the 8GB for data. Three more things:
- barrier=0 mount option
- elevator=deadline boot option
- tune2fs -o journal_data_writeback

The first one (mount -o barrier=0) is subjectively the one that makes the
most real difference. Ext4 has barriers enabled by default, as opposed to
ext3.

I have a slow-only 16GB 900. With your setup, it might be effective to put
all journals on the fast disk (external journal), but I guess that's if you
actually enjoy tweaking :)

Finally it's possible to replace that slow SSD with a faster one from
RunCore for example. But that doesn't explain why windows does better
(which surprises me a bit as I've heard others complain that windows is
very bad at least on the 16BG --- but maybe it's ok if it's installed on
the fast 4GB disk).

Con Kolivas returns with a new scheduler

kragil — Sun, 06 Sep 2009 21:43:30 +0000

Thank you.

It is a 901 with a fast 4GB SSD drive and a slow 8GB SSD. I have done some tweaking (Firefox ramdisk, ext4, noatime,noload) but maybe you have the right answers for my problems (I am using KDE4 on Fedora 11)

I will try 2.6.31 as soon as it is ready. I guess it might also help.

Con Kolivas returns with a new scheduler

jbh — Sun, 06 Sep 2009 17:14:12 +0000

If your eeepc has an SSD, skips/freezes are more likely to come IO stalls
than from the CPU scheduler. Particularly if it's the 16GB variant of
900/900a, those are known to have very poor performance. In that case there
are some mount options and other tunables that can make a large difference
(disabling barriers and so on, let me know if you need a full list).

Or is this a conventional hard drive model?

Con Kolivas returns with a new scheduler

trasz — Sun, 06 Sep 2009 11:46:27 +0000

z/OS keeps full compatibility only for applications that don't mess with system internals. If your program does something strange (although supported by IBM), it's possible that it will cease working in few releases from now. Of course there is a whole process of phasing out features, so it won't be a nasty surprise to you or your customers - but still, they are definitely not keeping 30 years of backward compatibility for that.

Con Kolivas returns with a new scheduler

kragil — Sun, 06 Sep 2009 10:50:13 +0000

I have to agree completely!

On my EeePC Linux (Fedora) hangs, skips and freezes. Really annoying. XP and OSX86 don't. I really wish some Netbook distro would integrate this scheduler ASAP or Linus would come to his senses and make the scheduler pluggable. I don't need fair behavior or 4096 CPU support on my netbook. I want it to react to my commands and play my media like it was supposed to be played, but I guess servers don't need that and so most of the kernel devs just don't give a shit.

Con Kolivas returns with a new scheduler

realnc — Sat, 05 Sep 2009 03:28:34 +0000

"BIG NUMA machines will probably suck a lot with this because it pays no deference to locality of the NUMA nodes when deciding what cpu to use. It just keeps them all busy. The so-called "light NUMA" that constitutes commodity hardware these days seems to really like BFS."

The point is that big NUMA machines aren't used for Desktops ;)

Con Kolivas returns with a new scheduler

realnc — Fri, 04 Sep 2009 11:10:23 +0000

I've just installed a bfs (204) patched kernel on my Core 2 Duo. I fired up LMMS and started a CPU hungry project and there's not a single sound skip/pop/crackle. Smooth. It was giving lots of sound artifacts (due to under-runs) before. At the same time I started playing notes in the Fluidsynth synthesizer (utilizing a 1.8GB sound font!), and there's virtually no lag between the key press and when the note actually plays anymore.

Then I started a kernel build (-j2), fired up mplayer on a 1080p HD video, and continued working in LMMS.

Still smooth. That was totally killing LMMS before (too much lag). I felt like I was using the MS Windows version of LMMS. I now realize that the current scheduler totally sucks for Desktops.

IMHO, what Desktop Linux needs desperately right now are more people like Kolivas. Dude, if you're reading this: thank you. I hope people will contribute to BFS. But I guess the kernel devs will not even take a look at it since it doesn't "scale". (The sound cracks/pops and lag in my music authoring tool won't "scale" either, but it seems they don't care very much; you see, LMMS is not MySQL. A 1% regression there and they start running to fix it. But who cares about me making music on Linux, right?)

Con Kolivas returns with a new scheduler

DavidG — Fri, 04 Sep 2009 09:31:35 +0000

FWIW: for AMD, only Opterons have NUMA and AFAIK Intel i7 don't have NUMA either.
So you're save to use BFS.

Input pauses

Cato — Fri, 04 Sep 2009 08:17:40 +0000

I've had these pauses for several seconds sometimes in Firefox 3.5.2, but they seem to be associated with large writes e.g. un-tarring a huge file at the same time - so I suspect it's the well known "Firefox 3.0 + ext3 + fsync" issue, reported in http://lwn.net/Articles/284126/ and elsewhere.

Con Kolivas returns with a new scheduler

shane — Fri, 04 Sep 2009 07:49:15 +0000

From the linked-to text:

GIT repository?

Sorry, it's not the right tool for me so it's not worth me investing the time
in setting one up.

Input pauses

jzbiciak — Fri, 04 Sep 2009 04:32:56 +0000

I know Firefox's stalls are Firefox's fault, not the Linux scheduler's. I was begging Con to do interactivity work on Firefox. ;-)

(As in, contribute to the Mozilla team.)

Input pauses

ncm — Fri, 04 Sep 2009 04:09:25 +0000

I see input-event stalls for up to six seconds in Firefox (Debian Iceweasel, actually), routinely. To get it I just have to leave it running for a week or so, with a few dozen pages, some of which self-update.

I doubt the scheduler would help; during those six seconds, one CPU is pegged at 100%. This is with nothing swapped. I suspect it's doing a garbage collection scan during each pause. I'd welcome suggestions for how to discover what's going on.

Google Chrome is coming along too slowly.

Stuck keys.

ncm — Fri, 04 Sep 2009 02:47:57 +0000

I've seen on 2.6.29. Unplugging and re-plugging the (USB) keyboard didn't fix it. Only switching to console and back did it. I was inclined to blame the new X input system. It hasn't happened since I upgraded X and also went to 2.6.30, so who knows?

Con Kolivas returns with a new scheduler

Tronic — Fri, 04 Sep 2009 02:38:44 +0000

Except that it is not really possible to prevent OOM with ulimits. The
limits are per-process and thus a larger number of processes can easily eat
all your RAM. Setting nproc and memory limits so that running out of RAM is
impossible is otherwise unfeasible.

This scenario is very real - just last week I managed to crash our master
server by accidentally running about 100 Valgrind-debugged web services at
the same time. The machine went completely unresponsive for a long time, so
finally we went to the server room to reset it the hard way (even the
console would not respond at all). Linux still doesn't handle OOM situations
properly, it seems.

Con Kolivas returns with a new scheduler

jzbiciak — Thu, 03 Sep 2009 18:14:15 +0000

That sounds more like a dropped event somewhere such as dropped interrupt or the like. I haven't experienced that on any Linux box so far.

Con Kolivas returns with a new scheduler

charris — Thu, 03 Sep 2009 17:31:34 +0000

I also see "stuck" keys where a letter will repeat until the buffer is full. I see this on various hardware with various keyboards, so it isn't an actual stuck key. Do you see that also?

Con Kolivas returns with a new scheduler

smadu2 — Thu, 03 Sep 2009 17:02:16 +0000

OK.

Con Kolivas returns with a new scheduler

jzbiciak — Thu, 03 Sep 2009 13:39:56 +0000

I've only gotten multi-second pauses like that when I had a hard drive going bad, or I was swapping rather furiously.

With that much RAM, I wonder if VM housekeeping itself could cause such lags. 12Gb would have 3 million 4K pages. If you are running at 3GHz and did something that averaged 3000 cycles/page across all 3 million pages, that's 3 seconds.

Con Kolivas returns with a new scheduler

zlynx — Thu, 03 Sep 2009 13:28:08 +0000

I was just commenting on how they do it. I am not saying anything bad about it.

Con Kolivas returns with a new scheduler

nix — Thu, 03 Sep 2009 13:15:54 +0000

It's certainly not the bluetooth driver or X, because I see this on the console of a one-die four-core Nehalem (Core i7) with no X running at all and a PS/2 keyboard. It's lightly loaded by these standards (load average 8.1, which as it's hyperthreaded is pretty much equivalent to 'everything loaded but not much'), and surely isn't swapping (12Gb RAM, 6Gb *free*, not even used for cache). Yet I see three-second pauses in keyboard activity.

I'll turn on latencytop and see what it says, but the pauses are fairly rare so it might be interesting to interpret.

Desktop Debian = Ubuntu

nix — Thu, 03 Sep 2009 13:10:09 +0000

Former DPLs need not apply ;}

(of course I'm running Debian on the desktop too, and adminning it remotely for my mum, who's in the same boat, and all she wants is something that Just Works and isn't virally infestable...)

Welcome To The Jungle

gouyou — Thu, 03 Sep 2009 13:09:14 +0000

It reminds me of this blog post from the Adobe/Linux/Flash guy: Welcome To The Jungle.

Con Kolivas returns with a new scheduler

hppnq — Thu, 03 Sep 2009 11:53:31 +0000

From what I've read about your problems with filesystem X eating your data on platform Y, you are likely to be a victim on any combination of X and Y.

Just make proper backups and do not assume that any filesystem is going to keep your data, especially if you stress it enough.

Con Kolivas returns with a new scheduler

Cato — Thu, 03 Sep 2009 10:17:40 +0000

I don't agree the filesystems are currently 100% fine - I've lost a lot of data on ext3, most likely due to the default use of write caching in the drive (not a filesystem issue but the default is dangerous for use with most filesystems) combined with a lack of journal checksumming (added in ext4). Also, until btrfs is stable, we won't have disk block level checksums, as in ZFS, which is the one thing that really makes me want to use OpenSolaris.

See http://lwn.net/Articles/350072/ and in particular the link to the University of Wisconsin paper linked within the thread - they looked at reiserfs, ext3 and JFS, and how they cope with disk failures.

Con Kolivas returns with a new scheduler

k8to — Thu, 03 Sep 2009 05:02:57 +0000

Please don't complain about that which you don't understand.

The Sound and Graphics have arguable unpleasantries (though all platforms do).

The scheduler isn't optimal for some workloads, but it's also not bad. Compare it to the windows or Mac scheduler both of which regularly do really unpleasant things for both server and workstation workloads.

The filesytems are 100% fine.

Con Kolivas returns with a new scheduler

k8to — Thu, 03 Sep 2009 04:56:04 +0000

Isn't that kind of aside from the point? I mean how they do it is pretty much their choice. They do it.

Con Kolivas returns with a new scheduler

charris — Thu, 03 Sep 2009 02:37:58 +0000

I find that gnome terminals will ignore my bluetooth keyboard for up to several seconds. I don't know if this is because of the bluetooth driver, X, or the scheduler, but it sure is annoying. And this on a 3GHz quad core system.

Con Kolivas returns with a new scheduler

smadu2 — Thu, 03 Sep 2009 00:07:59 +0000

New addition to the list of things that are not so good in Linux kernel/OS, 1. Sound 2. Graphics 3. Filesystem and now 4. Scheduler.

Desktop Debian = Ubuntu

alex — Wed, 02 Sep 2009 21:01:19 +0000

Hey I suggested my other half install Debian on her old laptop as she wanted experiment with free software. I was slightly worried she would have issues with the wireless but it all installed and ran fine. I think Debian makes a fantastic desktop OS if your not concerned about the latest whizz bang graphics effects or media players. It's stable and lightweight and very developer friendly.

Desktop Debian = Ubuntu

zlynx — Wed, 02 Sep 2009 20:47:15 +0000

But do I know you?

I mean, when I said it, I meant people I actually know in person and I know what they're running.

Now, as I don't make a habit of going around asking people if they're running Ubuntu or Debian or what, my sample size is about 5 people.

One of those I know used to run a Debian laptop and I *think* he is running Ubuntu now but possibly not. I know the other 4 are running Ubuntu.