LWN: Comments on "OLS: On how user space sucks"

OLS: On how user space sucks

MKallas — 2010-11-11T09:57:54+00:00

Yes, this would be something worth of a follow-up!
Besides, here's the updated link to the slides of Dave Jones' presentation "Why userspace sucks" (Magicpoint format).

OLS: On how user space sucks

Randakar — 2010-11-10T10:11:43+00:00

(Sorry for raising this particular article from the grave)

How much of this has been changed and/or fixed today?

Infinite bloat

barrygould — 2006-08-09T01:28:35+00:00

True was changed from a shell script to an executable due to the fact that running a shell for a user that wasn't supposed to be able to login (e.g. an ftp-only user) created a security hole (ctrl-c could get you a shell).

Barry

gtk file selector dialog

Duncan — 2006-08-06T16:07:59+00:00

> Does [...] gnome/kde has a "open with application"
> dialog like the one windows uses

KDE (3.5.4) does. It pops up a 3-section window, textbox file chooser on
top (browse button to the right), a tree-view copy of the K-Menu in the
center, and two checkbox options on the bottom, run in terminal (with a
don't close after exit suboption that's dimmed out until run in terminal
is selected), and remember application association (similar to the
MSWormOS dialog option). Below that are the usual OK/Cancel buttons.

Duncan

gtk file selector dialog

kelvin — 2006-08-03T09:17:35+00:00

If you're unfortunate enough to navigate to /usr/bin using it, it will probably lock up for a good 2 minutes as it reads 10+ mb of data and makes tens of thousands of system calls.

This was all too true on older versions of GTK+, but the file selector (among other GTK-things) has been heavily profiled and improved in the latest versions. On this P4/3GHz Ubuntu install with GTK+ 2.8.20, a cold-cache opening of /usr/bin takes roughly 2 seconds (including display of the "undecipherable icons").

Test, with "test" vs /usr/bin/test

kreutzm — 2006-07-29T15:43:50+00:00

Where do you see that "60x"? ALso I'd advise you to use time(1) next time ;-)

OLS: On how user space sucks

sandmann — 2006-07-28T17:45:15+00:00

> Wow. I'm not a particularly good coder, but my understanding is that if it polls like a large chunk of the desktop programs do, generally, you've done it wrong. I know of the sleep() and usleep() functions, and I'm pretty sure could figure out callbacks.

Uh, are you saying that using poll() is wrong?

gtk file selector dialog

bjornen — 2006-07-28T15:03:13+00:00

> So... for now I mostly use opera, at least on slow machines.

Ouch! opera 9.0-20060616.6 is one of the worst offenders on my computer (a
750MHz P4 laptop (dell i8k)).

top's TIME+ column says it used 24 CPU seconds by the time it's done
loading. After this it continues to steal ~3% CPU by itself and increasing
Xfree86's CPU usage to ~7%.

And this on a fresh install, only 9 tabs opened, the window minimised and
javascript, plugins, et al turned off.

There some surprising blame and constructive criticism in Federico Mena
Quintero's blog -
http://primates.ximian.com/~federico/news-2005-11.html#mo...

Profiling before you optimize

Thalience — 2006-07-27T17:26:54+00:00

Ok. Those are reasonable numbers to work with. Thanks for making the effort!

I still think, however, that ditching auto-detection would be to throw the baby out with the bathwater. By focusing more attention on hot-plug style auto-detection, we can have both fast startup and reliable, effortless hardware support.

As an aside, I've always thought that Kudzu was a very appropriate name for that particular peice of software. :)

OLS: On how user space sucks

lockhart — 2006-07-27T17:15:56+00:00

Or see the individual paper at http://ols2006.108.redhat.com/

Sometimes userspace is made to suck, I mean poll

gdt — 2006-07-27T16:34:02+00:00

linux-2.6.17/Documentation/feature-removal-schedule.txt says:

What: mount/umount uevents
When: February 2007
Why: These events are not correct, and do not properly let userspace know when a file system has been mounted or unmounted. Userspace should poll the /proc/mounts file instead to detect this properly.

Also, try and detect if an unrelated process is running without polling (hint: *notify doesn't work on procfs).

gtk file selector dialog

emj — 2006-07-27T14:33:43+00:00

Here is te bug report in bugzilla all you have to do is fix it. Does any one know if gnome/kde has a "open with application" dialog like the one windows uses

gtk file selector dialog

emj — 2006-07-27T14:21:50+00:00

The best part is you can't just enter a command to execute having firefox look it up in the path.. So this bug of GTK fileselector only show up because a misfeature in Firefox.. ;-(

gtk file selector dialog

jbh — 2006-07-27T11:13:13+00:00

Yes! 2 minutes to choose an application sounds about right (even if I know the exact path and write it in the Ctrl-L location dialog).

I guess it wouldn't be too hard to work around either, but I really don't want to wade into the flame fest that surrounds the gtk file chooser. So... for now I mostly use opera, at least on slow machines.

inotify

nix — 2006-07-27T09:35:11+00:00

... except of course that if you're a poor sod whose home directory is mounted over the network (perhaps from a centralized RAIDed fileserver), then, oops, the damn thing falls back to polling (over a network!)

Apparently its inability to send notifications to other copies of itself over the network is a *feature*, but given that you're using NFS or a similar fs in any case, I can't imagine what extra security threats could be opened by sending notifications around. (FAM could do this.)

Profiling before you optimize

ekj — 2006-07-27T07:54:01+00:00

Not scientifically valid profiles, no.

But bootup-time (measured from GRUB loads the kernel until the last startup-script finishes) improved from 1:48 to 1:32 simply by disabling kudzu (which does detection of new hardware).

Uninstalling packages that where auto-installed without question, and that support hardware I don't have shaved another 10 seconds off that to aproximately 1:23.

1:23 and 1:48 ain't that hugely different, but it *does* mean a 30% increase in bootup-time for trying to detect hardware I don't have on every bootup.

Some hardware is regularily plugged in an out. It's reasonable (and good) to try to detect such. But that should happen on the fly, and not as part of a bootup-script. Afterall, the user may very well plug in a usb-stick or whatever *after* logging in.

Other hardware (like for example a pcmcia-slot) is rather unlikely to suddenly appear. I'm guessing that 99.9% of the computers that don't have it when the distro is installed, will *never* have it.

Infinite bloat

nix — 2006-07-25T12:05:35+00:00

That is all GNU libc-version-dependent, and happens before main() is entered.

Test, with "test" vs /usr/bin/test

Richard_J_Neill — 2006-07-25T03:59:12+00:00

"if [ a = b ] " and "if test a = b " are both shell-builtins. If you want the other one, you have to call /usr/bin/test

$ date; for ((i=0; i<10000; i++)) ; do if [ a = b ] ; then c=d ; fi ; done ;date
Tue Jul 25 04:51:16 BST 2006
Tue Jul 25 04:51:16 BST 2006

$ date; for ((i=0; i<10000; i++)) ; do if test a = b ; then c=d ; fi ; done ;date
Tue Jul 25 04:51:27 BST 2006
Tue Jul 25 04:51:27 BST 2006

$ date; for ((i=0; i<10000; i++)) ; do if /usr/bin/test a = b ; then c=d ; fi ; done ;date
Tue Jul 25 04:51:33 BST 2006
Tue Jul 25 04:51:50 BST 2006

It's a huge difference! The script with the builtins runs 60 times faster.

gtk file selector dialog

joey — 2006-07-25T01:49:38+00:00

My favorite example at the moment is the gtk file selector dialog, as seen in firefox. Each time you change directories, it first uses getdents to get all the files in the directory, then stats each individual file, then _reads_ 4k of each file to determine the file type. That information is used to put undecipherable tiny useless icons next to the files indicating their file type.

If you're unfortunate enough to navigate to /usr/bin using it, it will probably lock up for a good 2 minutes as it reads 10+ mb of data and makes tens of thousands of system calls. Compare with ls /usr/bin which takes about 0.2 seconds even in inefficient sort and colorise mode.

Infinite bloat

joey — 2006-07-25T01:40:25+00:00

Shell coders who are really interested in being efficient don't use external commands like /bin/true anyway, when the shell builtin ":" will do the same thing.

(true is also a builtin in bash and dash, but I prefer ":" for space-efficiency also.)

However, your version of gnu true doesn't seem to match mine, which opens only /usr/lib/locale/locale-archive, and which is faster than the zero-byte version.

Infinite bloat

zlynx — 2006-07-25T01:11:50+00:00

Not mine, but I remembered seeing it before.

Here you go :)
http://www.muppetlabs.com/~breadbox/software/tiny/true.as...

Infinite bloat

jonabbey — 2006-07-25T01:00:43+00:00

But invoking a complete /bin/sh process to evaluate the empty shell script file would have been worse, surely?

OLS: On how user space sucks

cjwatson — 2006-07-24T23:27:49+00:00

On Ubuntu, pcmciautils is installed by default because - aside from its init script - it's pretty small and lightweight (compared to the monster that was pcmcia-cs/cardmgr) and it simplifies the installer, debugging the resulting system, etc. if we just install it all the time.

Per Olofsson simplified that init script a fair bit in Debian; Edgy has that simplification, and e.g. no longer calls laptop-detect.

It's also worth noting that 'case' in shell scripts doesn't spawn a subprocess, unlike 'if [ ... ]', so simply counting conditionals doesn't always give you a fair picture.

OLS: On how user space sucks

bluefoxicy — 2006-07-24T23:18:21+00:00

You know all you people are gathering nice profiling data and all, I'd like to know all the stuff that normally gets scanned for and why it's relevant. Seems to me the lot of you are running strace and finding a djillion gettimeofday()s et al; wouldn't it be nice to have a small shell script that does this and spits out timing information?

Unfortunately oprofile tends to suck and eat up gigs of hard disk space in a few days, else it'd be interesting to have a method of automatic profiling and reporting. Of course we then sacrifice cycles to do runtime profiling, which is no good in a production environment; but I'd volunteer some CPU time to gather data like that, it's not that much of a drag for light usage like typical desktop stuff. I'm not sure of the security implications here ... runtime timing data may be useful for getting parts of passwords or encryption keys, if it's detailed enough; but that's black magic to me so far.

Infinite bloat

aegl — 2006-07-24T21:01:23+00:00

A long time ago in a Unix version far, far away /bin/true was an empty shell script (since it was executable, the shell would run it as a shell script when the kernel failed the exec(2), with nothing in the script, the shell returned a 0 exit code).

$ ls -l /bin/true
-rwxr-xr-x 1 root root 21969 2004-04-05 21:32 /bin/true

Infinite bloat!

But it is worse. Run strace on /bin/true, and you'll see it open and mmap a dozen locale files (and try and fail to open a dozen more).

The problems here seem to have arisen because someone decided to add "--version" and "--help" arguments. Aaargggghhhh!

inotify

NAR — 2006-07-24T12:27:43+00:00

My hardware is supported out of the box on new kernels, which is wasn't for older.

Your mileage may vary, but I never managed to boot my old 486 with 2.6 kernel - fortunately it worked with 2.4. It didn't worked well, the TCP connection tracking code kept tracking connections that were long gone, so the system ran out of memory, but it still worked. On the other hand, one of the two reasons I use 2.6 on my other computer is that with 2.6 I dont' have to reboot between watching a DVD and burning a CD-R.

Stability has improved. Wireless support has improved. Udev makes things easier for me

Again your mileage may vary, but my computer locks up hard with every single 2.6 if I make a larger I/O operation while watching TV with xawtv - and this wouldn't make a useful bug report. I don't have wireless cards and never felt the need for dynamic /dev, so these features do not make me happy.

WHY 2.6.15, 2.6.16, 2.6.17 series kernels are unusable

Recording audio from TV doesn't work with mplayer. I've reported the bug and it's supposed to be in mplayer and supposed to be fixed, yet it still didn't work when I tried last time. So I stick with 2.6.14.

Bye,NAR

inotify

drag — 2006-07-24T06:42:24+00:00

Each kernel gets better for me. 2.4 series was better then 2.2. 2.6 is better for me then 2.4

Lower latencies, more usable desktop. Better responsiveness. My hardware is supported out of the box on new kernels, which is wasn't for older. ALSA sound drivers are a huge improvement over OSS for me. With dmix I can have, get this, more the _one_sound_ at a time and it doesn't sound like crap. Multimedia performance has improved.

(of course I am still taking about the kernel here.. it's desktop scedualing options makes life better)

Stability has improved. Wireless support has improved. Udev makes things easier for me now that I just tell the computer what /dev files I want vs having to dig around and finding the stupid major minor numbers for everything.

Maybe if the other person was to post WHY 2.6.15, 2.6.16, 2.6.17 series kernels are unusable maybe they would have receive more sympathy.

inotify

cventers — 2006-07-23T18:00:17+00:00

If 'cheap' is a function including n (the rate of change) rather than a
constant, then I think the kernel is about as 'cheap' as you can get.

It's unfortunate that you've had problems since 2.6.14. What sort of
problems are you having?

After having seen the survey conducted here on kernel quality, it would
seem like most users are pleased (I'm one of them).

inotify

NAR — 2006-07-23T17:55:49+00:00

Look at the Linux 2.6 process - I would call that "Fast, cheap, good". It's not perfect, but it's damn fast, it's still F/OSS and it's still /very/ good.

I wouldn't call the 2.6 process "fast, cheap and good". It might be fast, but it's certainly not good (the last usable kernel for me was 2.6.14) and definitely not cheap - I'd like to know how many kernel developers are funded for their work on the kernel. I think it's not a particulary low number.

Bye,NAR

inotify

cventers — 2006-07-23T17:55:06+00:00

True, but this is something that should be part of the desktop
infrastructure, not a part of every application. Most application
programmers wouldn't have a lot of fun with Xlib either, but someone's got
to do it...

inotify

NAR — 2006-07-23T17:51:27+00:00

in practice you must build something which tests for inotify at runtime and falls back to dnotify or even polling.[...]However, this is perfectly doable.

Yes, but I'm afraid this is way above the avarage application programmer's level.

Bye,NAR

inotify

cventers — 2006-07-23T17:10:56+00:00

Most of what you say about "Fast, cheap, good. Pick two" is fine and good.
But all I'm really trying to say is that we, the F/OSS community, have the
capacity to do better. Look at the Linux 2.6 process - I would call
that "Fast, cheap, good". It's not perfect, but it's damn fast, it's still
F/OSS and it's still /very/ good.

You could twist the definition of fast, cheap, and good enough to make
the "Pick two" argument apply to any project. The problem I have
with "Pick two" and the earlier optimization quote is simply that most of
the time I've heard an engineer saying one, it's being invoked as an
excuse for shoddy design. And I've personally witnessed that when you
simply let a passion for your art drive your work, and sprinkle on a
little bit of experience in the environment you're working in, you can
deliver "fast, cheap, good" all at once.

F/OSS is getting more and more industrialized, but depending on the
project, the majority of the code still comes from people with that
passion -- people just scratching their itch. I hope our projects don't
erode into the same corporately-managed disasters as are so commonplace to
the proprietary software engineer. But since engineers have the power in
F/OSS, I think if we focus on passion and rejecting ideas like "fast,
cheap, good -- pick two," we'll be entirely successful in breaking the
traditional rules of development once again.

This is free software. The traditional rules of corporate development
don't apply; please leave them at the door.

inotify

pizza — 2006-07-23T13:08:22+00:00

"Fast, cheap, good. Pick two" is a reflection of the reality that nothing is without cost.

If you want your software to be developed "good and fast", then it's not going to be cheap. If you want it "fast and cheap" then it's not going to be all that good. If you want it "good and cheap" then it won't happen particularly quicky.

"fast and cheap" is usually where software ends up when someone is directly footing the bill (and hence, there is an upper bound on cost, aka budgets/deadlines, and "good" tends to suffer). "Good and cheap" is where F/OSS software traditionally lies, where the "it'll be done when it's done" attitude is the norm. Then we end up with the likes of NASA (or other life-critical situtations), where the requirement of "good" is so important that it happens neither quicly nor cheaply.

The problem with the above generalization is that many larger F/OSS projects (including Gnome) actually fall into the first category, as the majority of the "work" is done by people required to do so, with formal goals, deadlines and budgets. F/OSS has gone up and been corporatized!

(Another glaring hole in this generalization is that "good" means different things to everyone -- In the end, only the one who is footing the bill gets to make that call -- but that is the nature of generalizations..)

And finally, I would agree with you and chalk up the problems that Dave raised to (A) and (B), although they both are symptoms of (C) -- which is usually due to inexperience, not idiocy. Subsequently, with better awareness of (A) and (B), (C) is lessened as the programmer presumably will learn from their mistakes.

Dave's ("spot on", as you put it) paper was a direct result of the idea embodied by the "no premature optimization" blurb that you took so much of an issue with. Without that instrumentation, this handful of bugs/mistakes wouldn't have likely come to light, and we wouldn't have been able to learn from them.

Some more observations: xfce4-panel, firefox

hein.zelle — 2006-07-23T00:09:08+00:00

Inspired by the above article, I went to check the first programs in the cpu ordered top-list on my system. Although I realize the point of the article was more generic, it may be an interesting excercise to try this on your own system. My system is running a recently updated debian unstable with kernel 2.6.17-1-k7.

xfce4-panel: average 2% cpu usage. A strace of approximately 10 seconds shows 267 function calls to gettimeofday, and 141 to ioctl and poll.

firefox: average just over 2% cpu usage. A strace of approximately 10 seconds shows 594 calls to gettimeofday, 151 calls to poll, read and ioctl, 282 calls to futex. I have no idea why it calls gettimeofday 4 times in a row in every cycle.

Apparently cpu-unintensive polling is a more common problem than I thought it would be. Is using gettimeofday, ioctl and poll the common way to do this?

inotify

drag — 2006-07-22T09:51:44+00:00

I am sure that it's worth complaining about... but it's a lot better then the 'famd' it replaced!

For instance with famd if I had mount point or something like that in my home directory then it would crap out if I tried to go more then 2 directories deep. And basicly cause anything to do with gnome that concernes files (nautilus mostly)

With gamin there is no problem.

I think that a huge part of the problem we have with performance on Linux desktop nowadays is that everybody was scrambling to get just the basics in place and everything more or less working.

Hal/Dbus/X.org/inotify(and it's userspace helpers)/desktop search stuff/udev.. etc etc. All of it is thrown together and made to 'make it just work'.

Now it seems that the push is going towards making 'make it work well'. Filling out the blanks, improving performance. That sort of thing.

inotify

dvdeug — 2006-07-22T06:07:23+00:00

As far as I know, none of the modern distros build for a stock i386. You can't; the C++ libraries depend on i486 opcodes to properly implement threading, and while there are i386 alternatives, they're slow and unreliable.

Pointers to the relevant bugs filed

davej — 2006-07-21T21:39:48+00:00

I don't have pointers to bugs, because in a lot of cases, I just mailed/IRC'd the relevant developers.

Lots of the examples I covered are already fixed, but there are still a number of outstanding issues. I'll be rerunning the tests some time soon, and see what's left that sticks out, but based on data I collected not so long back, we're now doing a *lot* better than we used to.

The initial tests the paper was based on were done on Fedora Core 5 test1 iirc, and I did some quick stats on an FC5 final release, and the amount of reads/stats/exec's were pretty much halved, even though we had added more functionality, and a few extra daemons etc.

I'll be looking at this stuff again as we get closer to FC6.

inotify

cventers — 2006-07-21T20:14:15+00:00

> Most of your response is tangental to the argument I submitted.

Really? I'm not sure I see how. It seems to me like you were listing
counterpoints to my complaint about programming to the least common
denominator, and I was systematically addressing them (including your
quote about optimization)

> Here's the bottom line -- we're not all "above average" programmers.
> Even when we know what "the right way" is, we usually don't have that
> luxury due to externally-imposed constraints.

What does "average" have to do with it? It doesn't take oodles of talent
to build a model capable of using different implementations. Sometimes,
it's even more trouble to try and come up with something generic!

You allude to constraints but never mention what some of them might be.

> "Cheap, fast, good. Pick two"

Why pick just two? One of the greatest things about free software
development is that it's usually not the requirements-driven,
oh-my-the-deadline-is-yesterday-and-the-customer-is-complaining-style
development uncomfortably familiar to programmers working in the
corporate world. And if our projects are being run that way (which I
don't think they are), we should move further up the chain and ask why
we're adopting policies and procedures that impose external constraints
on our code quality.

This stuff isn't actually all that complicated. The problem is either

*A) No one had pointed out ways in which apps misbehave, so no one knew
there was a problem (glad we have this paper to enumerate some examples!)
*B) Developers did what they thought was 'good enough' and just didn't
realize that their implementation didn't make their expectation
*C) We're less than average programmers and we can't figure this stuff
out for the life of us (doubt that, there's oodles of awesome free
software from all of the major projects out there, which demonstrates
competency)

So I think Dave's paper was spot-on. We should skip the 'apologizing'
step and move on to 'making it better'.

inotify

pizza — 2006-07-21T19:18:58+00:00

Most of your response is tangental to the argument I submitted.

Here's the bottom line -- we're not all "above average" programmers. Even when we know what "the right way" is, we usually don't have that luxury due to externally-imposed constraints.

"Cheap, fast, good. Pick two"

inotify

nix — 2006-07-21T18:31:40+00:00

fam 2.7.0 uses dnotify in any case, if it's available. It may not be as nice as inotify but it's a hell of a lot better than polling.

Hm. Looking at the sources, gamin has had an inotify backend since v0.0.8, Aug 26 2004, *long* before inotify hit the kernel proper. It is enabled by default.

Looks like this might be an out-and-out bug. I'll have a look this weekend and see if I can reproduce and fix it.