Weekly Edition
Return to the Front page
| |||||||||||||||||||||||
Our bloat problem
Andy Oram's report from the
Ottawa Linux Symposium notes that OpenOffice.org took some grief there:
Already, two speakers have made wisecracks about OpenOffice.org,
tagging it as a bloated memory hog. I have the suspicion that some
attendees see Linux as something to run for its own intrinsic
value, rather than as a platform for useful applications that can
actually help people accomplish something.
As one of those speakers, your editor will plead guilty to taking a cheap shot for an easy laugh (and people did laugh). But the remark had nothing to do with the value of OpenOffice.org as an application. It was about bloat. In a private conversation at the same conference, an engineer working with a services company in a developing country mentioned a valuable line of business for his employer. It seems that there are customers with large numbers of older desktop computers running legacy operating systems; they would like to extend the life of those computers by putting Linux onto them. But Linux does not run as well on these systems as anybody would like; it is simply too big. OpenOffice.org is especially problematic on smaller systems, but the problem does not stop there. Not that long ago, Linux was a relatively small and fast system which could run well on a wide variety of older hardware. That may still be true in some specific cases - Linux-based firewall/routers, for example - but, as a general-purpose operating system, Linux has become just as bloated as its proprietary competition. Your editor just looked at his desktop system, with two days of uptime, to see where the memory went. A few examples:
It is a sad world when 10MB of memory is required to display a clock, and 21MB to run a terminal emulator. Developers who have taken a class in data structures have probably heard all about time-space tradeoffs. Programs can often be made faster at the expense of higher memory usage. The truth of the matter, however, is that these tradeoffs are often illusory. Big code is slow code. From inferior processor cache usage through to virtual memory thrashing, large code slows things down across the entire system. On contemporary systems, the way to faster code is often by using less space, not more. There are signs that more developers are beginning to understand the costs of bloat. There is a GNOME memory reduction project underway, for example, though it does not appear to be progressing rapidly. But a more serious effort will be required if the Linux desktop is going to lose some significant weight. And it should lose that weight. Some growth is to be expected from the development of the software itself - Linux systems can do much more than they could a few years ago. But it seems clear that much of our development has been aimed at the addition of new features, and relatively little attention has been paid to memory usage. At this point, Linux need not feel insecure about the features it offers; maybe the time has come to put some more effort into implementing those features with fewer resources. Otherwise, Linux is inflating itself out of a number of possible applications and losing the leanness which used to be one of its best attributes. (Log in to post comments)
Our bloat problem Posted Aug 3, 2005 19:28 UTC (Wed) by jg (guest, #17537) [Link] Jonathan,
The address space you are seeing in modern desktop programs are generally all shared libraries, which are shared across all applications of that desktop family.
The other problem, which I believe is getting fixed, is that ld is referencing even libraries that aren't actually used, and the build systems typically are referencing all the libraries when linking. There is an ld option that will prevent that from happening. This is worth doing not only to reduce the apparent memory being used, but also to avoid some work being done on a per library basis, even if no symbols are actually referenced.
So while we have bloat, it isn't as bad as it (currently) appears.
Our bloat problem Posted Aug 3, 2005 20:15 UTC (Wed) by hp (subscriber, #5220) [Link] I don't know how to get very useful numbers out of top myself - what one would want for a bloat metric is "malloc'd RAM unique to this process" or something, perhaps "plus the size of each in-use shared page divided by number of apps currently sharing it," perhaps "plus resources allocated on the X server side on behalf of this app." Instead top has your choice of arcane numbers that aren't too useful. What you want is a number that will go down when genuine improvements are made, go up when things are genuinely made worse, and show in a fair way how each app contributes to the total.
memprof is pretty good for getting better numbers, when it hasn't been busted by yet another ABI change (it mucks around in internals that aren't in the public ABI). But it only works for a single process.
Maybe this is similar to the boot speed problem, where Owen's suggestion to create a good visualization resulted in http://www.bootchart.org/ which in turn led to lots of truly useful enhancements.
Anyone want to make a "bloat chart"?
If it included X resources and mallocs and accounted for shared libs somehow that would be pretty good, though it still wouldn't be handling other daemons that allocate resources on behalf of apps (e.g. gconfd uses more memory as more apps ask it to load more settings).
There are some real limits on bloat improvements, though. Things like terminal scrollback buffer, emails, web pages, icons, background images are going to be big, and they're going to be bigger the more you have of them.
Our measurement problem Posted Aug 3, 2005 23:07 UTC (Wed) by JoeBuck (subscriber, #2330) [Link] One of the more embarrassing problems is that we don't really have tools that can give us an accurate picture of the problem. Everyone will tell you that "top" doesn't give an accurate picture, and everyone is right. The difficulty is that there is at present no tool that really shows what is going on.Before we can really improve things, we need better measurement tools. Remember how much faster the boot times got after some guy did bootchart? Back in the early 80s, VMS had a "graphical" (termcap-style) tool that would show a real-time animation of every page in memory, tracing it to the corresponding executable or shared library or kernel. The modern extension would show memory-mapped files, cached files, etc. as well. If we could just have those pictures, people's attention would quickly focus on the hot spots.
Our measurement problem Posted Aug 4, 2005 1:13 UTC (Thu) by freddyh (guest, #21133) [Link] I do agree that there is not really a tool that *easily* shows the memory problem. Atkins however can teach you exactly what your application is doing, including its memory usage.Unfortunately you'll have to dig in yourself, and the first time will cost you quite some time.
FreddyH
Our measurement problem Posted Aug 4, 2005 6:45 UTC (Thu) by MathFox (subscriber, #6104) [Link] Well "cat /proc/<pid>/maps" allready gives you a lot of information:
$ cat /proc/self/maps
This shows a quite lean program, in order of the lines:
(each page on this machine is 4 kb)
Most of the apparent bloat for cat (850 pages * 4 k = 3.4 Mb) is in localisation and the shared C library. All but 16 pages (64 k) are shared with other processes.
Who writes a nice graphical frontend to /proc/*/maps?
Our measurement problem Posted Aug 4, 2005 8:59 UTC (Thu) by Yorick (subscriber, #19241) [Link] It is not possible from /proc/$pid/maps to find out how many pages of a mapped file have beenprivately modified (.data sections etc). This would be a useful addition, but because of the lack of forethought it can be difficult to add it to this pseudo-file without breaking existing programs.
It is also not possible to find the set of actually shared pages from a mapped file - just because
We definitely need better memory usage metrics from the kernel, or bloat like this will be difficult
Our measurement problem Posted Aug 6, 2005 8:00 UTC (Sat) by sandmann (subscriber, #473) [Link] The kernel actually has a system call - mincore - that tells you which pages of a mapped file are in memory. I wrote a small program a long time ago that uses it to print more details about memory usage:
http://www.daimi.au.dk/~sandmann/freon.c
You have to be root to run it.
Unfortunately a similar thing can't be done for malloc()d memory because mincore doesn't work for anonymous pages.
Our bloat problem Posted Aug 4, 2005 1:15 UTC (Thu) by eru (subscriber, #2753) [Link] There are some real limits on bloat improvements, though. Things like terminal scrollback buffer, emails, web pages, icons, background images are going to be big, and they're going to be bigger the more you have of them.Actually this "payload data" is often minuscule. Take those terminal scrollback buffers. Assuming each line contains 60 characters on the average (probably an over-estimate) and you have them in a linked list with 8 bytes for links to the previous and next line, storing a 1000 lines needs just 66.4 Kb. Where does the rest of the 21 Mb of gnome-terminal go? Similarly in emails, a single piece of mail might typically need of the order of 10 Kb for one textual message. Images and sound files are of course inevitably large, but not most applications don't deal with them.
8 byte characters? Posted Aug 4, 2005 2:25 UTC (Thu) by davidw (subscriber, #947) [Link] 8 byte characters are becomming a thing of the past, in user-facing applications... Still though, your point is taken - 66K multiplied by a factor of 4 still isn't that much.
8 byte characters? Posted Aug 4, 2005 8:33 UTC (Thu) by smitty_one_each (subscriber, #28989) [Link] > 8 byte characters are becomming a thing of the past, in user-facing applications...I think you meant 8-bit, but isn't that what UTF-8 is about? Why not just use that throughout the system? Footprint may grow, but simplicity is often worth the fight. R, C
8 byte characters? Posted Aug 12, 2005 7:45 UTC (Fri) by ringerc (guest, #3071) [Link] Many apps use UCS-2 internally, because it's *MUCH* faster to work with for many things than UTF-8 . With utf-8, to take the first 6 characters of a buffer you must decode the UTF-8 data (you don't know if each character is one, two, or four bytes long). With UCS-2, you just return the first 12 bytes of the buffer.
That said - it's only double. For text, that's not a big deal, and really doesn't explain the extreme memory footprints we're seeing.
8 byte characters? Posted Aug 12, 2005 20:53 UTC (Fri) by hp (subscriber, #5220) [Link] Unicode doesn't fit in 16 bits anymore; most apps using 16-bit encodings would be using UTF-16, which has the same variable-length properties as UTF-8. If you pretend each-16-bits-is-one-character then either you're using a broken encoding that can't handle all of Unicode, or you're using UTF-16 in a buggy way. To have one-array-element-is-one-character you have to use a 32-bit encoding.
UTF-8 has the huge advantage that ASCII is a subset of it, which is why everyone uses it for UNIX.
8 byte characters? Posted Aug 20, 2005 0:24 UTC (Sat) by miallen (guest, #10195) [Link] Many apps use UCS-2 internally, because it's *MUCH* faster to work with for many things than UTF-8 .I donno about that. First, it is a rare thing that you would say "I want 6 *characters*". The only case that I can actually think of would be if you were printing characters in a terminal which has a fixed number of positions for characters. In this case UCS-2 is easier to use but even then I'm not convinced it's actually faster. It your using Cyrillic, yeah, it will probably be faster but if it's 90% ascii I would have to test that. Consider that UTF-8 occupies almost half the space of UCS-2 and that CPU cache misses account for a LOT of overhead. If you have large collections of strings like from say a big XML file the CPU will do a lot more of waiting for data with UCS-2 as opposed to UTF-8. In truth the encoding of strings is an ant compared to the elephant of data structures and algorithms. If you design your code well and adapt interfaces so that modules can be reused you can improve the efficiency of your code much more than petty compiler options, changing character encodings, etc.
Our bloat problem Posted Aug 4, 2005 7:35 UTC (Thu) by elanthis (subscriber, #6227) [Link] That isn't generally how a terminal scrollback buffer works, however. You generally work with blocks of memory, so even blank areas on lines are filled in with data. That's required due to how terminals work in regarding to character attributes. Which also brings up the point that you have more than just character data per cell, you also have attribute data. And then let's get to the fact that in 2005, people use more than just ASCII, and you actually can't use only a byte per character, but have to use something like 4 bytes per character in order to store UNICODE characters.
So if you have an 80 character wide display with 100 lines of scrollback, and we assume something like 8 bytes per character (4 for character data, 4 for attributes and padding) we get 8*80*100 = 640000. And that's just 100 lines. Assuming you get rid of any extraneous padding (using one of several tricks), you might be able to cut down to 6 bytes per character, resulting in 6*80*100 = 480000. Almost half a megabyte for 100 lines of scrollback.
More features requires more memory. If you want a terminal that supports features that many people *need* these days, you just have to suck it up and accept the fact that it'll take more memory. If you can't handle that, go find a really old version of xterm limited to ASCII characters without 256-color support and then you might see a nice reduction in memory usage. The default will never revert to such a terminal, however, because it flat out can't support the workload of many people today, if for no other reason than the requirement for UNICODE display.
Our bloat problem Posted Aug 4, 2005 7:44 UTC (Thu) by tialaramex (subscriber, #21167) [Link] You don't need 4 bytes per character for Unicode in most places. A brief examination of the unicode xterm shows that, as expected it doesn't actually store everything as 32-bit ultra-wide characters. Most strings can be stored as UTF-8, a few places might deal with the actual code point and have a 32-bit integer temporarily, but certainly not huge strings of them.
Effect of Implementation Choices Posted Aug 4, 2005 10:21 UTC (Thu) by eru (subscriber, #2753) [Link] That isn't generally how a terminal scrollback buffer works, however. You generally work with blocks of memory, so even blank areas on lines are filled in with data.But that is a very wasteful implementation choice. There are several other ways of doing it (like the linked list I proposed) that are not much more complex to program. I forgot about attributes in my original post, but they, too can easily be represented in ways that average much less than 4 bytes per character. And as another poster pointed out, you can store Unicode with less than 4 bytes per character. In today's computers the CPU is so much faster than the memory that it may not pay to optimize data structures for fast access at the cost of increased size. I think this difference illustrates a major reason for the bloat problem: using naive data structures and code without sufficient thought for efficiency. Maybe OK for prototypes, but not after that. I am not advocating cramming data into few bits in complex ways (as used to be common in the days of 8-bit microcomputers), but simply avoid wasting storage whenever it can be easily done. Like, don't store boolean flags or known-to-be small numbers in full-size ints, allocate space for just the useful data (like in the scroll-back case), don't replicate data redundantly. I wonder if the well-known GNU coding guidelines (see Info node "standards" in Emacs installations) may be partly to blame for bloat problems in free software... To quote: Memory Usage Right, but what when you have lots of programs open at the same time, each using "just a few meg of memory"? (I recognize Stallman wrote that before GUI's became common on *nix systems).
Our bloat problem Posted Aug 4, 2005 13:29 UTC (Thu) by berntsen (guest, #4650) [Link] If people malloc like you multiply, I see where the bloat is comming from, you have a factor of 10 wrong ;-)
/\/
Our bloat problem Posted Aug 7, 2005 14:56 UTC (Sun) by oak (guest, #2786) [Link] I don't know how to get very useful numbers out of top myself - what one would want for a bloat metric is "malloc'd RAM unique to this process" or something, Malloc => heap. If the program has written to the allocated heap page, it's private. I don't see why program would allocate memory without writing to it, so in practice all of heap can be considered private to the process. You can already see heap usage from /proc and with Valgrind you can actually get a graph where it goes. perhaps "plus the size of each in-use shared page divided by number of apps currently sharing it," During his Guadec 2005 speech, Robert Love mentioned a kernel patch which will produce the information about how much memory is private (dirty = allocated heap that has been written to, shared library relocation tables etc.) to a process. He promised to add a link to it on to the Gnome memory reduction page. perhaps "plus resources allocated on the X server side on behalf of this app." XresTop tells this. Some programs can push amazing amounts of memory to the Xserver (the huge number number shown in 'top' for Xserver comes from memory mapping the framebuffer though I think).
Our bloat problem Posted Aug 7, 2005 15:59 UTC (Sun) by hp (subscriber, #5220) [Link] Of course you can figure this out painstakingly. What you can't do though is get all that info aggregated for 1) each process and 2) a current snapshot of all processes, in an easy way.
Aggregating this stuff into a snapshot of the whole system at a point in time would let you really point fingers in terms of bloat and figure out where to concentrate efforts.
It's not easy enough now, which is why people just use "top" and its misleading numbers.
Even better of course would be to take multiple snapshots over time allowing observation of what happens during specific operations such as log in, load a web page, click on the panel menu, etc.
A tool like this would probably be pretty handy for keeping an eye on production servers, as well.
Our bloat problem Posted Aug 18, 2005 14:10 UTC (Thu) by oak (guest, #2786) [Link] > Aggregating this stuff into a snapshot of the whole system at a point in> time would let you really point fingers in terms of bloat and figure out > where to concentrate efforts. If the patch Robert mentioned is the same which results I have seen (post-processed), you can already get that information from Linux kernel by patching it a bit. The results I saw included machine wide statistics and process specific stats of how many pages reserved for the process were ro/rw/dirty, how much each linked library accounts for the process memory usage (relocation table sizes etc., not how much each library allocates heap for the process[1]). This is quite useful for system overview, whereas valgrind/Massif/XResTop/top tell enough of individual applications. [1] that you can get just with a simple malloc wrapper that gets stack traces for each alloc and some data post-processing heuristics to decided which item in the stack trace to assign the guilt. The hard part is actually the post-processing, deciding who in e.g. the allocation chain of App->GtkLabel->Pango->Xft2->FontConfig->Freetype should be blamed for the total of the allocations done at any point in the chain as you don't know whether the reason for the total allocations is valid or not without looking at the code... Best would be an interactive allocation browser similar to Kcachegrind with which one could view also the source along with accumulated allocation percentages.
Our bloat problem Posted Aug 11, 2005 6:24 UTC (Thu) by anton (guest, #25547) [Link] I don't see why program would allocate memory without writing to itI do this when I need contiguous memory, but don't know how much. Then I allocate lots of space; unused memory is cheap. The result looks like this: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND anton 17100 0.0 0.2 6300 1196 pts/0 S+ 14:16 0:00 gforthThe large VSZ is caused mainly by unused allocated memory. So if you want to know the real memory usage, counting private anonymous mappings is not good enough.
Our bloat problem Posted Aug 3, 2005 20:29 UTC (Wed) by omez (subscriber, #6904) [Link] "...ld is referencing even libraries that aren't actually used, and the build systems typically are referencing all the libraries when linking."
Michael Meeks wrote about some of his Open Office and toolchain related work: http://go-oo.org/~michael/OOoStartup.pdf
Our bloat problem Posted Aug 4, 2005 0:39 UTC (Thu) by emj (guest, #14307) [Link] Recuding relocations at startup? I'm not sure it will help alot, well it's a start but... He says However doing this will chop ~1 second off the warm-start time of OO.o, and substantially improve both Mozilla and KDE startup performance, now this is very nice it goes down from 6s to 5s startup on my machine.But cold startup takes 23s, and this is what mater to me and my fellow users. The time when you feel OO.o startup time the most is when you have to wait for the computer to boot and then OO.o to start, it seems like forever. What can you do about this, is it just ineffienct loading of 21MB of OO.o process. It takes 46s to cold start WinME (from bios) and 6s to to start Word (while still loading the OS). And then there's no read from the disk at all..
OO,o starts Posted Aug 4, 2005 1:52 UTC (Thu) by eru (subscriber, #2753) [Link] The time when you feel OO.o startup time the most is when you have to wait for the computer to boot and then OO.o to start, it seems like forever.Indeed. I recently found that if I want to just view a single PowerPoint file on Linux, it is about 10x faster to start wine + Microsoft's PowerPoint viewer, than to start OO.o to view the file. Sad.
Unshared memory usage Posted Aug 4, 2005 10:27 UTC (Thu) by ajross (subscriber, #4563) [Link] Looking only at the writable, unshared memory of my clock-appletafter boot on FC4:
% pmap 2430 | fgrep rw | sed 's/K rw.*//' | sed 's/.* //' | awk '{total+=$1};END{print total}'
4304
So maybe the complaint should be amended to "It is a sad world
Unshared memory usage Posted Aug 4, 2005 13:49 UTC (Thu) by pphaneuf (subscriber, #23480) [Link] I use wmcalclock, and the same command-line you get tells me 288. ps' vsz is 2868.
Maybe I'm just too old school or something.
Unshared memory usage Posted Aug 6, 2005 23:45 UTC (Sat) by komarek (guest, #7295) [Link] I ran the same commandline on gkrellm2, which includes a clock among many other monitors. I get 3436, more than wmaclock but still less than the clock applet. Crazy stuff.
Unshared memory usage Posted Aug 8, 2005 22:22 UTC (Mon) by daniel (subscriber, #3181) [Link] pmap 2430 | grep rw | awk '{total += $2}; END {print total}'
Bloat must be hunted down and exterminated wherever it is found :-)
Unshared memory usage Posted Aug 10, 2005 23:16 UTC (Wed) by sholden (guest, #7881) [Link] pmap 2430 | awk '/rw/ {total += $2}; END {print total}'
It's about time... Posted Aug 3, 2005 21:41 UTC (Wed) by jabby (guest, #2648) [Link] It's about time someone started making more noise about the bloat on the Linux desktop. It's sad when non-profits who can't afford newer computers to run a Microsoft OS more recent than Win98 look to Linux to reuse their older hardware and I have to tell them "Well, theoretically you can, but..."
Even after setting up LTSP on a reasonably powerful machine, it can only power about 4-5 terminals with KDE or GNOME and Firefox before the RAM on the LTSP server is exhausted. And LTSP is less than ideal because sound and local removable storage don't really work, yet (at least I couldn't get them working).
This is a sad commentary on the supposedly inclusive nature of Linux. Even though we all have access to the code, no one is spending time so that we can *all* run it!
Efforts like the RULE Project have sought to create streamlined installs of major distributions, choosing only lightweight apps and desktops. Other efforts include specialized distributions, which thereby isolate the user into a smaller, niche community. Both approaches can be successful, but the applications they use lack the depth of community support of their mainstream (bloated) cousins.
Nevertheless, these apps prove that the bloat is not necessary. Anyone who has used the ROX-Filer file manager knows that it is possible to write a GUI filemanager that runs blazingly fast on an old 586 box and uses very little of that precious RAM.
I agree with Jon's observation about the temporary rush for features to the detriment of performance. It is the bloated toolkits and all of the supporting libraries for the major desktops that have both allowed a sudden proliferation of apps and features as well as saddled those apps with unrealistic hardware requirements. My key example is KWrite. It's a simple text editor with a few extra features. It takes *seconds* to load on my 1GHz Athlon machine with 256MB of RAM. I can't imagine trying to use KDE 3.4 on a 586 with 32MB...
I like the suggestion in the first thread of creating a "bloat chart" so developers can measure and remedy their bloat. But I think developers and distribution maintainers really need to look at dependencies a lot closer and try to remove as many as possible. Rather than linking to a library because it's there and you can, really evaluate the trade-off between the functionality you're getting and the bloat you're dragging along to get it. If jg is right and most of those libraries don't need to be loaded, then breaking those false dependencies would go a long way toward solving the immediate logjam.
It's about time... Posted Aug 3, 2005 22:36 UTC (Wed) by hp (subscriber, #5220) [Link] It's significant that most people don't _use_ the "lightweight" apps, though, unless they really genuinely don't have the compute power.
http://www.joelonsoftware.com/news/20020407.html
Solid element of truth there.
It's about time... Posted Aug 3, 2005 23:09 UTC (Wed) by JoeBuck (subscriber, #2330) [Link] Unfortunately, we lack lightweight apps that can deal with Microsoft file formats.
It's about time... Posted Aug 3, 2005 23:53 UTC (Wed) by hp (subscriber, #5220) [Link] Not sure you could have them - the MS file formats require all the MS application features. The hard part of implementing the formats is coding the features that use the stuff in the file.
anti-bloating microsoft docs Posted Aug 12, 2005 9:55 UTC (Fri) by astrophoenix (subscriber, #13528) [Link] my friend was writing her Ph.D. dissertation in microsoft word. now that she has to get it formatted to pass the university, I'm helping her convert it to LaTeX (we found a LaTeX class for her university to handle the formatting). the first step was to filter the .doc through antiword, producing a nice ascii file. even the tables came out nice. her 400 page .doc file was converted to ascii so fast that I wasn't sure it worked at first!
It's about time... Posted Aug 12, 2005 17:28 UTC (Fri) by obi (guest, #5784) [Link] Well, unlike OpenOffice Abiword and Gnumeric feel very light. Opening an Excel file in Gnumeric has _never_ failed for me (in my personal experience, 100% compatible - maybe I haven't used difficult documents enough), and on my 800mhz machine startup is nearly instant. Abiword is a bit less compatible with Word than I'd like it to be, but it's definitely usable, produces very nice output in XHTML, is kept simple - nice!
(BTW Inkscape/GIMP start up in a fraction of the time needed for their commercial counterparts - which might have a bit more features, but I seriously doubt a lot of people use them)
Just to mention that it's not all bad.
Exactly Right Posted Aug 4, 2005 10:37 UTC (Thu) by GreyWizard (subscriber, #1026) [Link] On my desk I have an old iMac from 1999 with 32MB of RAM. Last week I tried to install Fedora Core 4 to on the poor thing, only to discover that it took thirty seconds from the time I moved the mouse to the time anything happened on the screen. That was just in anaconda (the installation program)! I'll have to try again in text mode when I get the chance, but I doubt I'll be running Firefox on that machine.
Meanwhile I'm typing on a 1.7 GHz laptop with 1GB of RAM where FC4 is quite snappy providing innumerable features I've come to think of as essential. Going back to RedHat 6 is simply not an option, even supposing it could be tricked into supporting the internal wireless card. Doing more with less is a worthy goal and I would be thrilled if some genius worked magic that made my iMac useful again, but until then I vote for bloat.
It's about time... Posted Aug 4, 2005 14:47 UTC (Thu) by jrigg (subscriber, #30848) [Link] >It's significant that most people don't>_use_ the "lightweight" apps, though, >unless they really genuinely don't >have the compute power.
I tried KDE recently after using WindowMaker and rxvt for the last few years. It was irritatingly slow even on a dual Opteron, so I reverted to my old setup. Maybe I'm impatient, but I prefer the speed of lightweight apps running on fast hardware.
It's about time... Posted Aug 4, 2005 0:55 UTC (Thu) by NAR (subscriber, #1313) [Link] it can only power about 4-5 terminals with KDE or GNOME and Firefox before the RAM on the LTSP server is exhaustedAs far as I remember, KDE was never a lean fast software, even the 1.0 version was considered to be a slow memory hog, so it's not surprising at all that it's still a bloat. Fortunately it's not compulsory to use KDE - on my desktop I have 512 MB RAM and it's very rarely swapping, if at all.
KDE and bloat Posted Aug 4, 2005 5:20 UTC (Thu) by pointwood (subscriber, #2814) [Link] Since KDE is a desktop environment, it's not surprising that it is not as lightweight as a "simple" windowmanager. There is no such thing as a free lunch. KDE uses more memory because it provides you with a helluva lot more than a windowmanager does, but that doesn't mean KDE is bloated.
I'm just a user, not a KDE developer, so won't claim to know much about the issue, however I do know that it is an issue the KDE developers have been working on for quite some time and if I recall correctly, the current KDE 3.X releases doesn't really use more memory than KDE 2 did, even though it provides a lot more.
I did a little google search and found these items as proof that the KDE developers are very much aware of this and continue to work on making KDE use as few ressources as possible:
I'm sure there are KDE developers that are reading LWN, so I I hope they'll chime in sooner or later with more accurate info about the work that's been done and is still going on.
KDE and bloat Posted Aug 4, 2005 7:44 UTC (Thu) by NAR (subscriber, #1313) [Link] KDE developers are very much aware of this and continue to work on making KDE use as few ressources as possibleI'm aware of this - I seem to remember that when KDE2 was released, the application startup was supposed to be faster in KDE2 than KDE1. Unfortunately my experience showed that KDE2 was slower than KDE1 because of the memory usage. I think this is really a feature issue: I use Opera and Firefox on both Linux and Windows and the Windows versions didn't seem to be faster.
It's about time... Posted Aug 4, 2005 9:35 UTC (Thu) by cdmiller (subscriber, #2813) [Link] Most folks who run thin clients or LTSP use a lightweight window manager rather than gnome or kde.
It's about time... Posted Aug 4, 2005 11:11 UTC (Thu) by sbergman27 (subscriber, #10767) [Link] > Even after setting up LTSP on a reasonably powerful machine, it can only power about 4-5 terminals with KDE or GNOME and Firefox before the RAM on the LTSP server is exhausted.
I have a client that has a 1266MHz PIII w/1GB running Fedora Core 2 which runs 16+ terminals, 8 of which are remote and use Xvnc making them much heavier since a virtual X server has to run on the FC2 box, plus 70+ telnet sessions of a curses based point of sale/accounting package that they use, plus a web server and postgresql database server. (The web and db servers are lightly loaded.)
It runs Gnome 2.6 and OpenOffice and does just fine.
To be honest, we did up it to 2GB recently, but that was more of a luxury than a necessity.
I just checked and with the 2GB, disk buffers (cache+buffer) is 783MB, with 64MB in swap and 48M free.
LTSP Posted Aug 12, 2005 7:55 UTC (Fri) by ringerc (guest, #3071) [Link] My experience with LTSP at work is that the KDE and XFCE desktops both use minimal RAM. KDE seems to use a scarily small amount of additional RAM per instance after the first one - good use of static read only data etc I suspect.
OO.o, firefox, and Evolution, however, are lethal. Evolution in particular uses awe-inspiring amounts of RAM when working with large IMAP mailboxes. I've clocked it at > 250MB.
GNOME doesn't seem to be too bad with RAM use either. It looks a lot like the real issue is the apps.
Our bloat problem Posted Aug 4, 2005 3:04 UTC (Thu) by rakoch (guest, #4666) [Link] On Windows one usually specifies which symbols get exported while on Linuxthe default is that everything gets exported. Reducing the number of symbols in the library can make loading an application significantly faster since fewer symbols need resolving. I wonder how much faster OpenOffice will load when using this patch to GCC: http://www.nedprod.com/programs/gccvisibility.html and explicitly specify in the code which class is private to the library and which is public. -Rudiger
.so vs DLL Posted Aug 4, 2005 3:59 UTC (Thu) by eru (subscriber, #2753) [Link] On Windows one usually specifies which symbols get exported while on Linux the default is that everything gets exported.Linux (or rather the linker and run-time loader) works hard to make shared library programming and usage as similar as possible to using static libraries. If I remember correctly from past life, on Windows writing and building a shared library ("DLL") requires special care ,and you can easily share only functions. This makes the run-time mechanism much simpler and faster to load.
Our bloat problem Posted Aug 6, 2005 16:30 UTC (Sat) by nix (subscriber, #2304) [Link] KDE can already use the visibility attributes (available in GCC 4 without the need to patch).
I'm not sure if OOo can use them, but if it can't, it should. :)
gcc4's -fvisibility=hidden Posted Aug 12, 2005 7:58 UTC (Fri) by ringerc (guest, #3071) [Link] Support for this should be *REALLY* easy to add to a library that works under win32, or an app that already supports plug-ins under win32. You can piggyback the required attribute directives on the win32 export macros.
It does help, and is well worth doing since in many cases it won't require much more than tweaking the export macros in a header, then scratching together some more m4 macros for your configure scripts.
Our bloat problem Posted Aug 4, 2005 5:28 UTC (Thu) by dmantione (guest, #4640) [Link] Some sources of bloat on a Linux system:
Our bloat problem Posted Aug 4, 2005 11:54 UTC (Thu) by error27 (subscriber, #8346) [Link] Perl runtime takes less than 2M. Python takes slightly over 2M. I doubt you have tested Mono or TCL. I'm willing to believe that some JVMs take a lot of RAM.
Our bloat problem Posted Aug 4, 2005 13:15 UTC (Thu) by man_ls (subscriber, #15091) [Link] I'm willing to believe that some JVMs take a lot of RAM.That is probably why the most important efforts in Java on Linux (e.g. Red Hat's) are improvements in gcj: compiled Java, to do away with the JVM. Well, to be honest free Java is also a strong motivation. gcj-Java shared libraries still carry a lot of bloat and take up many megabytes of RAM, but again I'm not sure it is actually used.
Our bloat problem Posted Aug 5, 2005 3:37 UTC (Fri) by dmantione (guest, #4640) [Link] 2M for a script. Try to measure again for a real app, say Mandrake'surpmi and you'll see the Perl overhead starts to cost. I'm coding a lot in TCL by the way, but Pascal is my main language, as I'm developer for Free Pascal. Incidentally Free Pascal is a very good tool for fighting bloat.
Our bloat problem Posted Aug 4, 2005 13:32 UTC (Thu) by henning (subscriber, #13406) [Link] The problem with qt (one big lib for small tasks) is identified, and resolved in qt-4 . Now you must only load the needed parts for the gui, or the network specific lib for example.
Our bloat problem Posted Aug 5, 2005 13:56 UTC (Fri) by roelofs (subscriber, #2599) [Link] Also don't forget the small ones, libpng is over 200kb on my system, just to interpret some metadata grouped into fourcc chunks. This exclusive decompression,That's a rather simplistic view. libpng supports 19 pixel formats, depth-scaling, a moderately complex 2D interlacing scheme, alpha compositing, CRC verification, and various other transformations and ancillary bits. It also includes row-filtering, which is a fundamental component of the compression codec. I won't defend everything that's gone into libpng, but it's highly misleading to refer to all of it as bloat. If libpng (or a higher-level library built on top of it) didn't include it, all of your PNG-supporting applications would have to. that is in libz (better, 80kb, but I remeber Pkzip being 30kb on my MS-Dos system). Your memory is slightly faulty there. PKZIP 2.04g was 42166 bytes, and if all you care about is compressing your files, I can do far better with my 3712-byte trunc utility--and at 100% compression, too! But if you'd actually like to decompress them again someday, you'd better add PKUNZIP 2.04g, which rang in at 29378 bytes. IOW, the PKWARE codec was 71544 bytes (plus a number of other standalone utilities), while my copy of libz.so.1.2.3 is 71004 bytes--and 1.2.2 was 66908 bytes. Keep in mind also that significant parts of PKZIP and PKUNZIP were written in assembler, which, though capable of producing much smaller binaries, is generally not considered by most of us to be the most productive or maintainable of programming languages. (And, btw, libpng includes additional MMX assembler code for decoding row filters and for expanding interlace passes.) I'm sure the Commodore Amiga was able to read iff files (on which png is based) in less code. PNG was not based on IFF. The gross structure may have been suggested by someone familiar with IFF, but IFF itself was considered and rejected as a basis for PNG. Greg
Our bloat problem Posted Aug 6, 2005 5:22 UTC (Sat) by dmantione (guest, #4640) [Link] In other words, it is bloat. All that people want fom libpng is read andwrite png files and I doubt it is being used for more than that in the majority of situations. I know the differences between iff and png. I'd say png is even easier to read than iff.
libraries Posted Aug 12, 2005 8:03 UTC (Fri) by ringerc (guest, #3071) [Link] Actually, as far as I know a well written lib won't have much non-read-only static data (so it can be shared efficiently), and should incur only a very small memory overhead for any unused portions. If I recall correctly unused parts of the library aren't even read from disk.
There are many things to complain about with shared libraries, but their on-disk size is not one of them unless you're building embedded systems. If you are, you can build a cut down version of most libraries quite easily.
Our bloat problem Posted Aug 6, 2005 16:46 UTC (Sat) by nix (subscriber, #2304) [Link] I'm sorry, but much of this post is just plain wrong.String operations become more expensive. Processing an UTF-8 string is usually n times more expensive than processing an ASCII string, because there is no match anymore between byte numbers an character numbers.Actually, only some operations (random access, basically) become more expensive. Random access inside strings is rare, and apps that do it a lot (like Emacs and vim) have had to face this problem for years. We can't make the world speak English, no matter how much we might like to. Half the world's population speaks languages that require Unicode. o Conversion tables need to be in memory. Most applications load their own conversion tables, which means they are n times in memory. These tables are also loaded on startup, decreasing application startup times.I think you meant `increasing' there. :) but yes, this is a potential problem. It would be nice if there were some unicode daemon and corresponding library (that talked to the daemon) which could cache such things... but then again, the extra context switches to talk to it might just slow thigns right back down again. * Java, Mono, Python, Perl, TCL - These programming languages require runtime environments. The runtime environment needs to be loaded, can be slow itself and most importantly can use quite a bit of memory. It especially becomes bad if one multiple runtime environents get loaded on one desktop. Script languages can be good for scripts, but are bad for the desktop. The popularity of Java and Mono is propably a bad thing regarding the bloat on our machine.Actually, Python, Perl and Tcl have very small runtime environments (especially by comparison with the rampaging monster which is Sun's JRE). The problem with these languages is that their data representations, by explicit design decision, trade off size for speed. With the ever-widening gulf between L1 cache and RAM speeds, maybe some of these tradeoffs need to be revisited. * C++ - Even the de facto standard C++ is sometimes a problem. Especially if templates are being used C++ compilers output large amounts of code behind a programmers back. This can cause huge libraries and executables.Now that's just wrong. The use of templates in C++ only leads to huge code sizes if you don't know what you're doing: and you can write crap code in any language. The size problem with C++ at present is the (un-prelinkable) relocations, two per virtual method table entry... this can end up *huge*, even more so in apps like OOo which can't be effectively prelinked because (for reasons beyond my comprehension) they dlopen() everything and so most of them goes unprelinked. There was a paper recently on reducing OOo memory consumption which suggested radical changes to the C++ ABI. Sorry, but that isn't going to fly :) * Shared libraries - Many programmers are under the impression that use of shared libraries is free. WRONG. They need to be loaded, resolved and even if you only use part of them a large amount of code is executed within them before you know it.Wrong. Most shared libraries contain no, or very few, constructors, and so no code is executed within them until you call a function in them. (Now many libraries do a lot of initialization work when you call that function, but that'd also be true if the library were statically linked...) The dynamic loader has to do more work the more libraries are loaded, but ld-linux.so has been optimized really rather hard. :) Oh, and it doesn't have to load and resolve things immediately: read Ulrich Drepper's paper on DSO loading. PLT relocations (i.e. the vast majority, that correspond to callable functions, rather than those which correspond to data) are normally processed lazily, incurring a CPU time and memory hit only when the function is (first) called. o Libc is just a C runtime library but has unfortunately grown to several megabytes.That's because it also implements all of POSIX that isn't implemented by the kernel, and passes the rest through to the kernel. Oh, and it also supports every app built against it since glibc2 was released, which means that old interfaces must be retained (even the bugs in them must be retained!) This nceessarily costs memory, but most of it won't be paged in (and thus won't cost anything) unless you run apps that need it. You do rather seem to have forgotten that binaries and shared libraries are demand-paged!
Gnumeric Posted Aug 4, 2005 7:57 UTC (Thu) by tialaramex (subscriber, #21167) [Link] Gnumeric is if anything more useful for loading an MS Excel file than OpenOffice.org's spreadsheet OOCalc. Yet it loads at least an order of magnitude faster.
So the OO.org developers don't like us to say “It's bloated”, but what else should we say? Congratulations on your enormous application, it's very, um, well it's a, er, it's a good test of CPU and disk performance ? People who “actually want to accomplish something” are too busy to wait while OpenOffice.org loads.
Gnumeric Posted Aug 4, 2005 8:41 UTC (Thu) by dmantione (guest, #4640) [Link] I'd say that OpenOffice will be because of its bloat a dead piece ofsoftware. Why? Because programmers that have a choice choose better architected office suits. Take Koffice. Beatifully architected, too little features. Its code size is reasonable, and it is coded in C++, a language understood by many programmers. It should be possible to get started with it. OpenOffice, on the other hand requires full knowledge about both C++ and Java, is nightmare to develop on because simply compiling it takes ages because it is so much code. You only want to develop on it if you're paid to do so, which is exactly what is happening, OpenOffice is still coded mostly by Sun employees.
Wrong trade-off. The time is DEVELOPMENT time, not PERFORMANCE Posted Aug 4, 2005 8:07 UTC (Thu) by dwheeler (subscriber, #1216) [Link] Yes, there's a time vs. memory trade-off. But you're measuring the wrong time. The time that is getting traded off is development time.On the desktop, GNU/Linux systems are basically in "catch-up" mode, trying to implement a complete GUI desktop with highly honed, mature, and featureful applications faster than the current dominant vendor. Clearly, OSS/FS permits a radically different process that seems to enable faster development, but people still want things faster still. So how do you do it? Other trades are possible; you could sacrifice quality (but no one wants that) or reduce features (many users don't accept that). The obvious thing to do is to use approaches that save development time (use higher-level languages, pre-integrated massive libraries, etc.) at the expense of memory use. That doesn't mean you should ignore memory use. Indeed, any general solutions that reduce memory use significantly, without vastly slowing down all development, should definitely be employed. If people will track down the primary "memory hogs", and fix them, the world would be a better place. But let's be clear about the reasons for the memory use. It's not that people are stupid; smart people are making conscious decisions that, at least at the time, other issues were more important.
Wrong trade-off. The time is DEVELOPMENT time, not PERFORMANCE Posted Aug 4, 2005 10:01 UTC (Thu) by thompsot (guest, #12368) [Link] This is pretty much what I was thinking as I was reading through all this. Isn't the now common use of OO RAD tools which often spit out huge, unoptimized executables the biggest root of the problem? At one time most apps were written in C and the speed was incredible.This is definitely an issue of "speed to market" driving the efficiency (or lack thereof) of the code. Commercial vendors who compete to see who can release the next cool feature suffer from this trend, and so does OSS if we make speed to market a primary driver in decision making. I am inclined to believe that no/low cost, high quality, "slower to market" will eventually beat out high cost, low quality, "faster to market" over time. That's where the latest OSS movement started and consumer adoption has continually been on the rise from the beginning.
shared libraries [ab]use Posted Aug 4, 2005 12:29 UTC (Thu) by dann (subscriber, #11621) [Link] It seems like there's some abuse of share libraries going on.For example for the clock-applet that was mentioned in the article on a FC4 system:
ldd /usr/libexec/clock-applet | wc -l
Some of the libraries don't look like should be needed for a clock:
shared libraries [ab]use Posted Aug 4, 2005 13:33 UTC (Thu) by hp (subscriber, #5220) [Link] The talk about the clock is sort of bogus; the clock isn't just a clock, it's a small evolution frontend with all the calendar stuff in there. Those deps are coming from libedataserver (the evo backend)
The size of the calendar on the screen is small but it still has to have the same data available as a full calendar app and be able to talk to calendar servers and all that crap.
shared libraries [ab]use Posted Aug 4, 2005 16:16 UTC (Thu) by dann (subscriber, #11621) [Link] OK, ignore the clock then, let's look at any other applet, just as an example:ldd /usr/libexec/null_applet | wc -l 63 and it does link to the krb, ssl, gss*, gnome-keyring libraries (this seems to be true for all the applets that I have tried, and in general all the Gnome applications seem to link to a huge number of libraries).
shared libraries [ab]use Posted Aug 4, 2005 19:51 UTC (Thu) by hp (subscriber, #5220) [Link] Those specific ones must be gnome-vfs then, for https and authentication.
It's simple enough to cut down library deps for certain apps, but I doubt it makes much real performance difference. (unused code isn't mapped anyway)
Maybe it does, but that's why real data is needed.
shared libraries [ab]use Posted Aug 5, 2005 7:11 UTC (Fri) by dmantione (guest, #4640) [Link] No, that's the wrong assumption. Shared libraries are less expensive thanstatically linked ones, but not free.
shared libraries [ab]use Posted Aug 5, 2005 14:25 UTC (Fri) by hp (subscriber, #5220) [Link] Avoiding anything "not free" isn't a useful guideline.
The question is what are the tradeoffs. In the shared lib case, for example, avoiding them often means efficiency loss:
So if shared libs aren't a significant percentage of the overall problem, then "not free" is irrelevant; if they are a significant percentage of the overall problem then "not free" is probably more an argument for fixing shared libs (or coming up with some less-broken module system than ELF shared libs) rather than an argument for static linking and hand-rolled broken code.
Simplistic answers like "just don't use libs" are not useful. You don't want to warp the UI and implementation of every app in extensive ways just to work around a module system that needs optimizing.
shared libraries [ab]use Posted Aug 7, 2005 14:45 UTC (Sun) by oak (guest, #2786) [Link] I think the argument was that each *redundant* (unused) shared librarylinked to the binary has a cost. For example: Just by linking a library (that your program doesn't use), you've increased your program's memory usage by at least 4KB. At least one private (non-read-only) page is reserved for symbol resolving of each linked shared library. The new linker has a command line option to not to link dynamically libraries that are not used. I think it was discussed on the Gnome mailing lists, but globally enabling it had broken some program, so it's not enabled by default.
shared libraries [ab]use Posted Aug 5, 2005 10:38 UTC (Fri) by dann (subscriber, #11621) [Link] Well, using lots of shared libraries affects the startup time.Here's an example:
echo 'int main (void) {return 0;}' > t.c
gcc t.c -lgnomevfs-2
Obviously the above is only a small part of the startup time.
Maybe a kernel person can tell us what are the kernel overheads from having lots of shared libraries...
shared libraries [ab]use Posted Aug 6, 2005 11:59 UTC (Sat) by rlrevell (guest, #23596) [Link] Heh, really? It doesn't do any of that fancy stuff for me.
Right clicking only lets me "Copy Date", "Copy Time", "Adjust Date & Time" or set the "Preferences".
Left clicking brings up a view of the current month, that only lets me scroll back and forth to select a different day and month. AFAICT there's nothing you can DO with it.
Are you sure you did not mean "in the FUTURE this will provide access to evolution calendaring functionality?"
shared libraries [ab]use Posted Aug 6, 2005 19:02 UTC (Sat) by hp (subscriber, #5220) [Link] It has a read-only display of your evolution calendar and todo list, if you haven't put anything into evolution then you won't see anything.
shared libraries [ab]use Posted Aug 12, 2005 17:53 UTC (Fri) by dag- (subscriber, #30207) [Link] Every Gnome user interested in a clock was secretly running a complete evolution client ? I don't think this is funny. Really. Who's bright idea was this ? And everybody agreed to that ?
Our bloat problem Posted Aug 4, 2005 13:14 UTC (Thu) by ortalo (subscriber, #4654) [Link] I agree fully!!! (I like and use OOo Writer a lot, but that's not a reason to occult its weaknesses.)
Emacs? Posted Aug 4, 2005 15:00 UTC (Thu) by iabervon (subscriber, #722) [Link] I'm surprised that your emacs is so large. My experience with it is that it could actually be an acronym for "eight megs and constantly swapping", because it's always been about eight megs big (which I can barely remember being enough to constantly swap).
If you're getting that size by using a bunch of emacs processes, you should probably look into emacsclient, which acts like an editor but edits the file in an existing emacs process. This also makes it practically instantaneous to start.
It's the CPU too... Posted Aug 6, 2005 12:22 UTC (Sat) by rlrevell (guest, #23596) [Link] Whenever Linux desktop bloat comes up everyone wants to talk about memory wastage. But I think excessive CPU suckage is just as bad of a problem.
Recently I found a horrible performance bug in Evolution where it repeatedly goes over every message header with an empty search pattern if the message search bar is empty (rather than short circuiting the entire search process).
I distributed my quick hack solution to a few people on LKML and most people said it made browsing their mail 10x faster.
Unfortunately it took weeks and much flamage for me to even convince the Evolution developers that there was a bug, because their machines are all so fast that the performance problem was imperceptible.
Compare this to the Windows XP desktop which sucks just as much memory, but is SNAPPY on a sub-GHz machine.
Another example, on my slow (600MHz) machine, I didn't even realize OO *had* right click menus, because after right clicking in the document they take about 3 seconds to appear. It feels like OO is generating the context menu from some XML template or something and then popping it up AFTER the right click is received. Completely unusable. Compare Windows, which has obviously generated, rendered and cached these menus and just has to blit them in from offscreen when the user right clicks (instantaneous).
Memory isn't the problem, people. We're wasting WAY too many CPU cycles on bullshit.
Our bloat problem Posted Aug 6, 2005 16:44 UTC (Sat) by CSEESystems (guest, #29772) [Link] I think people are missing something when saying the clock-applet is bloated. The clock-applet is not just a clock. It is also a calendar and it has an interface into the evolution-data-server that allows it to display your evolution todo list and your appointments on the calendar. Now all of that in 10 megs. Is that too much? I don't know. For me its alright, but all of my machines have plenty of ram and I don't generally feel the memory crunch from anything but firefox with pages with lots of flash or images loaded.
Maybe someone needs to come up with a clock-applet-lite that only does the clock and not the rest of that stuff. Oh, and on my x86 system, the clock-applet has an RSS of 8.6 MB with the calendar loaded and everything.
Calendar ui size Posted Aug 7, 2005 4:35 UTC (Sun) by eru (subscriber, #2753) [Link] It is also a calendar and it has an interface into the evolution-data-server that allows it to display your evolution todo list and your appointments on the calendar. Now all of that in 10 megs. Is that too much? I don't know.Around 1987 I participated in writing a kind of office system for MS-DOS (commercial proram, but did not go far, very few people ever used it). Memory was tight, as the target was not expected to have more that 768Kb of memory and the network stack took a large chunk of that. Interface was based on textual "forms", not a GUI but close in ease of use. I did the overlay for a simple calendar. Entries were maintained with a simple B-tree database library, with files on a server so several users could consult each other's calendars. If I remember right, my overlay turned out to be the most "bloated" part, at about 20Kb of code (this excludes the DB and forms libraries, which were shared by all overlays of the app) ... and since this was 16-bit "compact" memory model code, everyone's in-RAM data had to fit in 64Kb. So yes, I think 10Mb is too much for an interface to a calendar server. I agree the above 20kb+64kb would be too little for a modern program, but shouldn't around 10x, like 800Kb for code+data, suffice for even a quite sophisticated user interface? Consider that the entire Windows v2 GUI system could operate in a smaller RAM (640Kb) than that!
|
|||||||||||||||||||||||