| |||||||||||||
Still waiting for swap prefetchStill waiting for swap prefetchPosted Jul 25, 2007 23:04 UTC (Wed) by elanthis (guest, #6227)In reply to: Still waiting for swap prefetch by mgb Parent article: Still waiting for swap prefetch
_Linux_ has been getting worse, or the shit that people use on their systems is getting worse?
I run 2GB in my home machine, but I don't think I've ever seen the memory peak over 1GB of actual usage. Over 60% of that is just file caches and the like - unnecessary but nice performance enhancements, basically.
Now, there are some things that we use these days that eat up a shit load of memory. My mail client has to track some 40,000 messages among all my folders. Firefox has to deal with Youtube videos. I have Tracker indexing my file system. I have more data in a single file than a hard-drive 15 years ago could even hold.
There is a lot of fat in the current desktop system, that is for sure. Ubuntu and Red Hat both have this obsession with Python, which is about as bloaty a language as you can get, and then they write crappy apps in Python that would be slow as molasses even in C because they use crappy algorithms.
(Log in to post comments)
Still waiting for swap prefetch Posted Jul 25, 2007 23:20 UTC (Wed) by xorbe (subscriber, #3165) [Link] When I have 2GB of ram in my home desktop system, I never ever want to see Linux drop binaries from ram, and swap them back in from the binary text file. Or see it drop data pages to increase hdd cache in memory.
Still waiting for swap prefetch Posted Jul 25, 2007 23:40 UTC (Wed) by johnkarp (subscriber, #39285) [Link] Even when the disk-cache-pages are more frequently used than theprogram-text-pages? If I understand things correctly, a program has to wait ~9ms when the data it needs is on the disk, whether its program text or data. Also, programs often have pages that are *never* used after startup; the sooner those pages are evicted in favor of useful things the better.
Still waiting for swap prefetch Posted Jul 25, 2007 23:47 UTC (Wed) by sjlyall (subscriber, #4151) [Link] When I have 2GB of ram in my home desktop system, I never ever want to see Linux drop binaries from ram, and swap them back in from the binary text file. Or see it drop data pages to increase hdd cache in memory.Turn off swap then if you feel that way. I've got no problem with my system swapping out the getty copies which will never be used and instead using the RAM to cache my mail files. I have a bunch of stuff that includes running programs, program data and files that the CPU has to deal with. I want everything to go as fast a possible so whenever possible if the CPU wants some of that data it should be able to get it as quickly as possible which means it should be in RAM if at all possible. The better the kernel does this, the faster everything will run.
Still waiting for swap prefetch Posted Aug 7, 2007 2:33 UTC (Tue) by xorbe (subscriber, #3165) [Link] Even with swap off, kernel can drop the program text, and swap it in from the backing binary on disk -- all without a swap file.
OpenOffice comes with a "pagein" utility. Posted Jul 26, 2007 0:14 UTC (Thu) by dmarti (subscriber, #11625) [Link] Why should the kernel have to guess what to page back in when you could use the "pagein" from OpenOffice, or something like it? Here's a simple example.
OpenOffice comes with a "pagein" utility. Posted Jul 26, 2007 0:39 UTC (Thu) by corbet (editor, #1) [Link] Interesting. It's doing something different, though: it's indiscriminately bringing in pages from the program's text area. I can't see how it could do anything about process anonymous pages which have been shoved into swap. Somebody who really wanted to enjoy their coffee hot would want to use both pagein (for text) and prefetch (for data) - and find some, currently unknown way to balance the two.
IWFM Posted Jul 26, 2007 16:13 UTC (Thu) by dmarti (subscriber, #11625) [Link] Subjectively, for me, running that shell function seems to get rid of the lag on alt-tab to Firefox, Gimp, or OpenOffice first thing in the morning.
What would be better, for a desktop system, would be a "goodnight" script that would run housekeeping tasks such as updatedb and logrotate, then page back in whatever the user had been working on, then suspend.
explicit pagein Posted Jul 27, 2007 22:12 UTC (Fri) by giraffedata (subscriber, #1954) [Link] To avoid the manual step (either good morning or good night), I run this in a cron job every hour all day. It doesn't matter that the interactive programs get paged in uselessly several times during the night; even if done right in the middle of updatedb, they just get paged right out again and the cost is insignificant.I also send messages through some response-time-critical daemons regularly, to "loosen them up." I.e. page in executables, anonymous pages, etc.
Still waiting for swap prefetch Posted Jul 27, 2007 0:05 UTC (Fri) by jschrod (subscriber, #1646) [Link] Then you will be surprised.
When you get 2GB (I have them), just start Eclipse on a larger project, while Firefox and Thunderbird are running; and watch your memory getting used. And you didn't even start any VM (Xen or VMware)...
Still waiting for swap prefetch Posted Jul 25, 2007 23:50 UTC (Wed) by kune (guest, #172) [Link] If Firefox is really faster on Windows than on Linux, then it should be easy to hack some benchmarks in Javascript to prove it. Nobody will be able to fix it, if he cannot measure the performance. And Internet network bandwidth has be taken out of the equation.
Sure, you can write bad code in any language, but in my experience Python is not particularly slower than other script languages. Here is a benchmark that appears to validate that experience: http://www.timestretch.com/FractalBenchmark.html Another one is here: http://acker3.ath.cx/wordpress/archives/7
Lua might be something to look at.
Of course you will always be able to write faster code in C, but this will take you some more time.
Still waiting for swap prefetch Posted Jul 25, 2007 23:53 UTC (Wed) by JoeBuck (subscriber, #2330) [Link] Besides, for a GUI program "written in Python" all the expensive code is actually in libraries that are written in C or C++, with a thin binding to allow calls from Python. yum isn't slow because of Python, but because it does way too many system calls and I/O.
Still waiting for swap prefetch Posted Jul 26, 2007 0:09 UTC (Thu) by briangmaddox (subscriber, #39279) [Link] "Of course you will always be able to write faster code in C, but this will take you some more time."
Ya had me until you said this. Why stop at C, when it could be written in ASM? And heck, how do you know the assembler will generate fast code, better do it in hex instead.
I would have thought that after all these years that people would learn more about computer science and programming than to troll the "C is always faster than everything else" line.
Still waiting for swap prefetch Posted Jul 26, 2007 5:15 UTC (Thu) by wilreichert (subscriber, #17680) [Link] "Ya had me until you said this. Why stop at C, when it could be written in ASM? And heck, how do you know the assembler will generate fast code, better do it in hex instead."
Hex? No thanks, i prefer to hack the macro assembler and control the logic gates on my cpu directly.
Still waiting for swap prefetch Posted Jul 26, 2007 9:46 UTC (Thu) by nix (subscriber, #2304) [Link] The thing to do is to generate the assembler and then munge it with a horrible perl script.
(Hey, ghc does it, it must be good! :) )
Still waiting for swap prefetch Posted Jul 26, 2007 12:04 UTC (Thu) by dcoutts (guest, #5387) [Link] Shh! You're not supposed to tell people about that, it's called the Evil Mangler for a good reason.
Still waiting for swap prefetch Posted Jul 26, 2007 13:51 UTC (Thu) by nix (subscriber, #2304) [Link] Having just watched it use fifteen minutes of CPU time (on the ghc lexer, natch) I think it needs bringing into the light so it can be optimized by some hardier soul than I :)
program speed vs programming language Posted Jul 27, 2007 23:06 UTC (Fri) by giraffedata (subscriber, #1954) [Link] Why stop at C, when it could be written in ASM? That's not a natural progression. Code compiled from C is often faster than that compiled from assembly language, for the same reason that a computer can land an airplane more smoothly than a human. Even code compiled from C by a naive compiler (e.g. gcc -O0) is unlikely to be slower than code compiled from assembly language. C is that low-level a language. how do you know the assembler will generate fast code, better do it in hex instead We do know that. The assembler will generate code that is not only the same speed as that generated by the hex editor, but is actually the same code. That's the definition of assembly language. I would have thought that after all these years that people would learn more about computer science and programming than to troll the "C is always faster than everything else" line. The only line I saw was, "C is always faster than Python." And it is, isn't it?
program speed vs programming language Posted Jul 31, 2007 12:08 UTC (Tue) by liljencrantz (subscriber, #28458) [Link] Depends on how you define your scope. I've seen situations where people solve the same problem in different languages, and because they have to spend so much time to do _anything_ in a low-level language, they are forced to chose a dumb algorithm, whereas people coding in a high level language can spend more time on the high level logic and can therefore chose a fast algorithm.
Still waiting for swap prefetch Posted Jul 26, 2007 9:44 UTC (Thu) by nix (subscriber, #2304) [Link] Benchmarks in Javascript will mostly show the performance of the JS interpreter and things it can block on, i.e. it's not a complete monitoring tool by any means.
Still waiting for swap prefetch Posted Jul 26, 2007 7:46 UTC (Thu) by pointwood (guest, #2814) [Link] "Ubuntu and Red Hat both have this obsession with Python, which is about as bloaty a language as you can get, and then they write crappy apps in Python that would be slow as molasses even in C because they use crappy algorithms."
Could you elaborate on that? I'm not much of a coder, but it would be interesting to hear what the problem with python is since I have considered playing with it.
In regards to Ubuntu and Redhat, well I bet they are open for improvements to their code though you don't say what apps it is. Your comment lacks facts.
Still waiting for swap prefetch Posted Jul 26, 2007 8:28 UTC (Thu) by rsidd (subscriber, #2582) [Link] There's nothing wrong with python (or other high-level languages), and it worked fine on linux systems 10 years ago. C is faster only for CPU-bound tasks, which hardly any system tasks are. C may be less memory intensive (it depends on what libraries you're using, what your coding style is, and so on), but with C you have to be careful about all kinds of bugs and security holes (buffer overflows, memory leaks, etc) that even experts get bitten by, but don't occur with high-level languages.
Still waiting for swap prefetch Posted Jul 26, 2007 12:23 UTC (Thu) by rwmj (subscriber, #5474) [Link] Oh dear no there's _plenty_ wrong with Python. It's dynamictyping nature means that simple objects have huge amounts of baggage that they have to carry around, mostly never used.
C sucks for programming too, for the reasons you outline.
But guess what folks! C and Python are not the only programming
My personal fave at the moment is OCaml. About as fast as C,
Rich.
Still waiting for swap prefetch Posted Jul 27, 2007 11:56 UTC (Fri) by IkeTo (subscriber, #2122) [Link] > C and Python are not the only programming> languages in the world!! You won't believe it, but other > programming languages have been invented.
> My personal fave at the moment is OCaml. About as fast as C,
Two suggestions. Suggestion 1: start lobbying people all around to start using OCaml: universities, companies, etc. If you are successful you have a bunch of people who *know* what it is (currently the people with that characteristics are so few that it doesn't matter). Or choose suggestion 2: start implementing some *real cool* thing in OCaml, making sure that developing the equivalent thing (e.g., with same performance, flexibility, etc) is so expensive that nobody can do *because of a choice of different language*. Then you serve as an example showing others the real benefit of the language.
Before you do so, accept that languages behaving the same way as in C or Python are those who know by those who work on creating applications. Problem is, money talks. If the industry do need C and Python, that's what's university courses teach, that's what's people know, and that's what applications will be written in.
Still waiting for swap prefetch Posted Aug 2, 2007 7:40 UTC (Thu) by renox (subscriber, #23785) [Link] My problem with Ocaml is its syntax and its functional mindset: I tried once to learn Ocaml and I disliked the syntax plus the PDF book I used insisted on using functional way to solve everything which is strange as Ocaml is said to support both imperative and functional style, why the book insisted so much on the functional style is beyond me, bleach.
So to be successful, Ocaml would need 2 things:
Somehow I doubt that will happen, so Ocaml is bound to stay in the limelight..
Still waiting for swap prefetch Posted Jul 26, 2007 15:15 UTC (Thu) by arjan (subscriber, #36785) [Link] > There's nothing wrong with python
... except that even a simple "hello world" seems to take 40 megabytes of memory. It's not about a few cpu cycles that kills you in performance, it's the enormous overhead that even simple programs get....
Still waiting for swap prefetch Posted Jul 26, 2007 17:11 UTC (Thu) by vmole (guest, #111) [Link] That is so much crap.$ cat hello.py import time print "hello, world" time.sleep(30) % python hello.py & [1] 25243 hello, world $ ps l -C python F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 0 1000 25446 2688 17 0 3844 2256 - S pts/0 0:00 python hello.py So that's 2M resident, 4M virtual. And it exagerates a simple "hello world" program, because I had to import the time module for sleep() so I could run ps. Most of it is the interpeter itself, so multiple python programs don't actually take 4M each. Python isn't perfect - what language is? But don't make stuff up, it just ruins your argument.
Still waiting for swap prefetch Posted Jul 26, 2007 19:01 UTC (Thu) by nevyn (subscriber, #33129) [Link] Let's try longer than 30 secs, eh? % ps ax -o cmd:50,vsz,rss,stime | egrep '[p]ython' /usr/bin/python -E /usr/sbin/setroubleshootd 808024 12104 Jul06 python /usr/share/system-config-printer/applet.py 232764 6436 Jul06 /usr/bin/python -E /usr/bin/sealert -s 411728 18524 Jul06 python /usr/libexec/revelation-applet --oaf-activa 458328 55056 Jul06 python /usr/lib64/gdesklets/gdesklets-daemon 453980 65768 Jul23 python ./hpssd.py 171220 1176 Jul11 ...that's 2_536_044 KB VSZ and 159_064 KB RSS, and as you can see I've rebooted gdesklets recently (it was roughly double that, I think). Plus I'm not running pupplet/yum-updatesd atm. And for instance "revelation-applet" is just a text entry box 95% of the time. I appreciate that the huge VSZ numbers are (hopefully) mostly unused shared libs. etc. but half a GB is still a lot for the OS to manage for a text box, and 40+MB of RSS for a simple GUI is far from "so much crap". For comparison my webserver uses two processes with a VSZ of about 12MB each and RSS of about 1MB each, and I'd prefer that to be smaller.
Still waiting for swap prefetch Posted Jul 26, 2007 19:24 UTC (Thu) by vmole (guest, #111) [Link] You said "a simple hello world" program, and that's what I tested. Comparing GUI programs to a webserver is irrelevant. What you've mostly demonstrated is that that GNOME/GTK is huge. Blaming that on Python doesn't seem to make sense. I'd bet a Perl implemenation of the same programs would be equally huge.
Still waiting for swap prefetch Posted Jul 26, 2007 9:37 UTC (Thu) by nix (subscriber, #2304) [Link] The page cache is not an `unnecessary but nice performance enhancement'. The text pages of all your binaries are sitting in there while they run, as is all your other mmap()ed stuff. If your page cache was empty you'd not be able to run any userspace code to speak of.
Still waiting for swap prefetch Posted Jul 26, 2007 13:22 UTC (Thu) by cate (subscriber, #1359) [Link] _Linux_ has been getting worse, or the shit that people use on their systems is getting worse?Check the size of a Linux kernel, and you will see that the is a bigger increment (in percent) in 5 years than the memory used by mozilla. We have more memory, so we can afford to use it more inefficiently.
Still waiting for swap prefetch Posted Jul 26, 2007 16:19 UTC (Thu) by khim (subscriber, #9252) [Link] I was barely able to boot Linux on 2MB system back then. Today I need 8Mb for that. Netscape was quite useful on 12MB Linux system (8MB was too slow), today you can barely even start Firefox on 48MB system. Huge size of Linux kernel in real life is mostly page tables so I think Linux and Firefox grew in unison...
Still waiting for swap prefetch Posted Jul 30, 2007 8:35 UTC (Mon) by gouyou (subscriber, #30290) [Link] We have more memoryNot really take a look at the OLPC or at the new development on the embeded device scene (Nokia N770/N800, OpenMoko) ...
Still waiting for swap prefetch Posted Aug 3, 2007 4:04 UTC (Fri) by bersl2 (subscriber, #34928) [Link] "We have more memory, so we can afford to use it more inefficiently."That's badly worded. It would be more correct to say that memory efficiency has been sacrificed for time efficiency.
|
|||||||||||||