LWN: Comments on "Our bloat problem"

8 byte characters?

miallen — Sat, 20 Aug 2005 06:24:35 +0000

Many apps use UCS-2 internally, because it's *MUCH* faster to work with for many things than UTF-8 .

I donno about that. First, it is a rare thing that you would say "I want 6 *characters*". The only case that I can actually think of would be if you were printing characters in a terminal which has a fixed number of positions for characters. In this case UCS-2 is easier to use but even then I'm not convinced it's actually faster. It your using Cyrillic, yeah, it will probably be faster but if it's 90% ascii I would have to test that. Consider that UTF-8 occupies almost half the space of UCS-2 and that CPU cache misses account for a LOT of overhead. If you have large collections of strings like from say a big XML file the CPU will do a lot more of waiting for data with UCS-2 as opposed to UTF-8.

In truth the encoding of strings is an ant compared to the elephant of data structures and algorithms. If you design your code well and adapt interfaces so that modules can be reused you can improve the efficiency of your code much more than petty compiler options, changing character encodings, etc.

Our bloat problem

oak — Thu, 18 Aug 2005 20:10:32 +0000

> Aggregating this stuff into a snapshot of the whole system at a point in
> time would let you really point fingers in terms of bloat and figure out
> where to concentrate efforts.

If the patch Robert mentioned is the same which results I have seen
(post-processed), you can already get that information from Linux kernel
by patching it a bit. The results I saw included machine wide statistics
and process specific stats of how many pages reserved for the process were
ro/rw/dirty, how much each linked library accounts for the process memory
usage (relocation table sizes etc., not how much each library allocates
heap for the process[1]). This is quite useful for system overview,
whereas valgrind/Massif/XResTop/top tell enough of individual
applications.

[1] that you can get just with a simple malloc wrapper that gets stack
traces for each alloc and some data post-processing heuristics to decided
which item in the stack trace to assign the guilt. The hard part is
actually the post-processing, deciding who in e.g. the allocation chain of
App->GtkLabel->Pango->Xft2->FontConfig->Freetype should be blamed for the
total of the allocations done at any point in the chain as you don't know
whether the reason for the total allocations is valid or not without
looking at the code...

Best would be an interactive allocation browser similar to Kcachegrind
with which one could view also the source along with accumulated
allocation percentages.

8 byte characters?

hp — Sat, 13 Aug 2005 02:53:30 +0000

Unicode doesn't fit in 16 bits anymore; most apps using 16-bit encodings would be using UTF-16, which has the same variable-length properties as UTF-8. If you pretend each-16-bits-is-one-character then either you're using a broken encoding that can't handle all of Unicode, or you're using UTF-16 in a buggy way. To have one-array-element-is-one-character you have to use a 32-bit encoding.

UTF-8 has the huge advantage that ASCII is a subset of it, which is why everyone uses it for UNIX.

shared libraries [ab]use

dag- — Fri, 12 Aug 2005 23:53:33 +0000

Every Gnome user interested in a clock was secretly running a complete evolution client ? I don't think this is funny. Really. Who's bright idea was this ? And everybody agreed to that ?

It's about time...

obi — Fri, 12 Aug 2005 23:28:26 +0000

Well, unlike OpenOffice Abiword and Gnumeric feel very light. Opening an Excel file in Gnumeric has _never_ failed for me (in my personal experience, 100% compatible - maybe I haven't used difficult documents enough), and on my 800mhz machine startup is nearly instant. Abiword is a bit less compatible with Word than I'd like it to be, but it's definitely usable, produces very nice output in XHTML, is kept simple - nice!

(BTW Inkscape/GIMP start up in a fraction of the time needed for their commercial counterparts - which might have a bit more features, but I seriously doubt a lot of people use them)

Just to mention that it's not all bad.

anti-bloating microsoft docs

astrophoenix — Fri, 12 Aug 2005 15:55:21 +0000

my friend was writing her Ph.D. dissertation in microsoft word. now that she has to get it formatted to pass the university, I'm helping her convert it to LaTeX (we found a LaTeX class for her university to handle the formatting). the first step was to filter the .doc through antiword, producing a nice ascii file. even the tables came out nice. her 400 page .doc file was converted to ascii so fast that I wasn't sure it worked at first!

Our bloat problem

astrophoenix — Fri, 12 Aug 2005 15:32:52 +0000

check out lavaps

libraries

ringerc — Fri, 12 Aug 2005 14:03:08 +0000

Actually, as far as I know a well written lib won't have much non-read-only static data (so it can be shared efficiently), and should incur only a very small memory overhead for any unused portions. If I recall correctly unused parts of the library aren't even read from disk.

There are many things to complain about with shared libraries, but their on-disk size is not one of them unless you're building embedded systems. If you are, you can build a cut down version of most libraries quite easily.

gcc4's -fvisibility=hidden

ringerc — Fri, 12 Aug 2005 13:58:29 +0000

Support for this should be *REALLY* easy to add to a library that works under win32, or an app that already supports plug-ins under win32. You can piggyback the required attribute directives on the win32 export macros.

It does help, and is well worth doing since in many cases it won't require much more than tweaking the export macros in a header, then scratching together some more m4 macros for your configure scripts.

LTSP

ringerc — Fri, 12 Aug 2005 13:55:30 +0000

My experience with LTSP at work is that the KDE and XFCE desktops both use minimal RAM. KDE seems to use a scarily small amount of additional RAM per instance after the first one - good use of static read only data etc I suspect.

OO.o, firefox, and Evolution, however, are lethal. Evolution in particular uses awe-inspiring amounts of RAM when working with large IMAP mailboxes. I've clocked it at > 250MB.

GNOME doesn't seem to be too bad with RAM use either. It looks a lot like the real issue is the apps.

8 byte characters?

ringerc — Fri, 12 Aug 2005 13:45:56 +0000

Many apps use UCS-2 internally, because it's *MUCH* faster to work with for many things than UTF-8 . With utf-8, to take the first 6 characters of a buffer you must decode the UTF-8 data (you don't know if each character is one, two, or four bytes long). With UCS-2, you just return the first 12 bytes of the buffer.

That said - it's only double. For text, that's not a big deal, and really doesn't explain the extreme memory footprints we're seeing.

Our bloat problem

anton — Thu, 11 Aug 2005 12:24:44 +0000

I don't see why program would allocate memory without writing to it

I do this when I need contiguous memory, but don't know how much. Then I allocate lots of space; unused memory is cheap. The result looks like this:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
anton    17100  0.0  0.2   6300  1196 pts/0    S+   14:16   0:00 gforth

The large VSZ is caused mainly by unused allocated memory.

So if you want to know the real memory usage, counting private anonymous mappings is not good enough.

Unshared memory usage

sholden — Thu, 11 Aug 2005 05:16:17 +0000

pmap 2430 | awk '/rw/ {total += $2}; END {print total}'

Unshared memory usage

daniel — Tue, 09 Aug 2005 04:22:27 +0000

pmap 2430 | grep rw | awk '{total += $2}; END {print total}'

Bloat must be hunted down and exterminated wherever it is found :-)

Our bloat problem

hp — Sun, 07 Aug 2005 21:59:26 +0000

Of course you can figure this out painstakingly. What you can't do though is get all that info aggregated for 1) each process and 2) a current snapshot of all processes, in an easy way.

Aggregating this stuff into a snapshot of the whole system at a point in time would let you really point fingers in terms of bloat and figure out where to concentrate efforts.

It's not easy enough now, which is why people just use "top" and its misleading numbers.

Even better of course would be to take multiple snapshots over time allowing observation of what happens during specific operations such as log in, load a web page, click on the panel menu, etc.

A tool like this would probably be pretty handy for keeping an eye on production servers, as well.

Our bloat problem

oak — Sun, 07 Aug 2005 20:56:34 +0000

I don't know how to get very useful numbers out of top myself - what one would want for a bloat metric is "malloc'd RAM unique to this process" or something,

Malloc => heap. If the program has written to the allocated heap page, it's private. I don't see why program would allocate memory without writing to it, so in practice all of heap can be considered private to the process.

You can already see heap usage from /proc and with Valgrind you can actually get a graph where it goes.

perhaps "plus the size of each in-use shared page divided by number of apps currently sharing it,"

During his Guadec 2005 speech, Robert Love mentioned a kernel patch which will produce the information about how much memory is private (dirty = allocated heap that has been written to, shared library relocation tables etc.) to a process. He promised to add a link to it on to the Gnome memory reduction page.

perhaps "plus resources allocated on the X server side on behalf of this app."

XresTop tells this. Some programs can push amazing amounts of memory to the Xserver (the huge number number shown in 'top' for Xserver comes from memory mapping the framebuffer though I think).

shared libraries [ab]use

oak — Sun, 07 Aug 2005 20:45:04 +0000

I think the argument was that each *redundant* (unused) shared library
linked to the binary has a cost.

For example: Just by linking a library (that your program doesn't use),
you've increased your program's memory usage by at least 4KB. At least one
private (non-read-only) page is reserved for symbol resolving of each
linked shared library.

The new linker has a command line option to not to link dynamically
libraries that are not used. I think it was discussed on the Gnome
mailing lists, but globally enabling it had broken some program, so it's
not enabled by default.

Calendar ui size

eru — Sun, 07 Aug 2005 10:35:43 +0000

It is also a calendar and it has an interface into the evolution-data-server that allows it to display your evolution todo list and your appointments on the calendar. Now all of that in 10 megs. Is that too much? I don't know.

Around 1987 I participated in writing a kind of office system for MS-DOS (commercial proram, but did not go far, very few people ever used it). Memory was tight, as the target was not expected to have more that 768Kb of memory and the network stack took a large chunk of that. Interface was based on textual "forms", not a GUI but close in ease of use. I did the overlay for a simple calendar. Entries were maintained with a simple B-tree database library, with files on a server so several users could consult each other's calendars. If I remember right, my overlay turned out to be the most "bloated" part, at about 20Kb of code (this excludes the DB and forms libraries, which were shared by all overlays of the app) ... and since this was 16-bit "compact" memory model code, everyone's in-RAM data had to fit in 64Kb.

So yes, I think 10Mb is too much for an interface to a calendar server. I agree the above 20kb+64kb would be too little for a modern program, but shouldn't around 10x, like 800Kb for code+data, suffice for even a quite sophisticated user interface? Consider that the entire Windows v2 GUI system could operate in a smaller RAM (640Kb) than that!

Unshared memory usage

komarek — Sun, 07 Aug 2005 05:45:57 +0000

I ran the same commandline on gkrellm2, which includes a clock among many other monitors. I get 3436, more than wmaclock but still less than the clock applet. Crazy stuff.

shared libraries [ab]use

hp — Sun, 07 Aug 2005 01:02:03 +0000

It has a read-only display of your evolution calendar and todo list, if you haven't put anything into evolution then you won't see anything.

Our bloat problem

nix — Sat, 06 Aug 2005 22:46:00 +0000

I'm sorry, but much of this post is just plain wrong.

String operations become more expensive. Processing an UTF-8 string is usually n times more expensive than processing an ASCII string, because there is no match anymore between byte numbers an character numbers.

Actually, only some operations (random access, basically) become more expensive. Random access inside strings is rare, and apps that do it a lot (like Emacs and vim) have had to face this problem for years.

We can't make the world speak English, no matter how much we might like to. Half the world's population speaks languages that require Unicode.

o Conversion tables need to be in memory. Most applications load their own conversion tables, which means they are n times in memory. These tables are also loaded on startup, decreasing application startup times.

I think you meant `increasing' there. :) but yes, this is a potential problem. It would be nice if there were some unicode daemon and corresponding library (that talked to the daemon) which could cache such things... but then again, the extra context switches to talk to it might just slow thigns right back down again.

* Java, Mono, Python, Perl, TCL - These programming languages require runtime environments. The runtime environment needs to be loaded, can be slow itself and most importantly can use quite a bit of memory. It especially becomes bad if one multiple runtime environents get loaded on one desktop. Script languages can be good for scripts, but are bad for the desktop. The popularity of Java and Mono is propably a bad thing regarding the bloat on our machine.

Actually, Python, Perl and Tcl have very small runtime environments (especially by comparison with the rampaging monster which is Sun's JRE). The problem with these languages is that their data representations, by explicit design decision, trade off size for speed. With the ever-widening gulf between L1 cache and RAM speeds, maybe some of these tradeoffs need to be revisited.

* C++ - Even the de facto standard C++ is sometimes a problem. Especially if templates are being used C++ compilers output large amounts of code behind a programmers back. This can cause huge libraries and executables.

Now that's just wrong. The use of templates in C++ only leads to huge code sizes if you don't know what you're doing: and you can write crap code in any language.

The size problem with C++ at present is the (un-prelinkable) relocations, two per virtual method table entry... this can end up *huge*, even more so in apps like OOo which can't be effectively prelinked because (for reasons beyond my comprehension) they dlopen() everything and so most of them goes unprelinked.

There was a paper recently on reducing OOo memory consumption which suggested radical changes to the C++ ABI. Sorry, but that isn't going to fly :)

* Shared libraries - Many programmers are under the impression that use of shared libraries is free. WRONG. They need to be loaded, resolved and even if you only use part of them a large amount of code is executed within them before you know it.

Wrong. Most shared libraries contain no, or very few, constructors, and so no code is executed within them until you call a function in them. (Now many libraries do a lot of initialization work when you call that function, but that'd also be true if the library were statically linked...)

The dynamic loader has to do more work the more libraries are loaded, but ld-linux.so has been optimized really rather hard. :)

Oh, and it doesn't have to load and resolve things immediately: read Ulrich Drepper's paper on DSO loading. PLT relocations (i.e. the vast majority, that correspond to callable functions, rather than those which correspond to data) are normally processed lazily, incurring a CPU time and memory hit only when the function is (first) called.

o Libc is just a C runtime library but has unfortunately grown to several megabytes.

That's because it also implements all of POSIX that isn't implemented by the kernel, and passes the rest through to the kernel. Oh, and it also supports every app built against it since glibc2 was released, which means that old interfaces must be retained (even the bugs in them must be retained!)

This nceessarily costs memory, but most of it won't be paged in (and thus won't cost anything) unless you run apps that need it.

You do rather seem to have forgotten that binaries and shared libraries are demand-paged!

Our bloat problem

CSEESystems — Sat, 06 Aug 2005 22:44:39 +0000

I think people are missing something when saying the clock-applet is bloated. The clock-applet is not just a clock. It is also a calendar and it has an interface into the evolution-data-server that allows it to display your evolution todo list and your appointments on the calendar. Now all of that in 10 megs. Is that too much? I don't know. For me its alright, but all of my machines have plenty of ram and I don't generally feel the memory crunch from anything but firefox with pages with lots of flash or images loaded.

Maybe someone needs to come up with a clock-applet-lite that only does the clock and not the rest of that stuff. Oh, and on my x86 system, the clock-applet has an RSS of 8.6 MB with the calendar loaded and everything.

Our bloat problem

nix — Sat, 06 Aug 2005 22:30:40 +0000

KDE can already use the visibility attributes (available in GCC 4 without the need to patch).

I'm not sure if OOo can use them, but if it can't, it should. :)

It's the CPU too...

rlrevell — Sat, 06 Aug 2005 18:22:21 +0000

Whenever Linux desktop bloat comes up everyone wants to talk about memory wastage. But I think excessive CPU suckage is just as bad of a problem.

Recently I found a horrible performance bug in Evolution where it repeatedly goes over every message header with an empty search pattern if the message search bar is empty (rather than short circuiting the entire search process).

I distributed my quick hack solution to a few people on LKML and most people said it made browsing their mail 10x faster.

Unfortunately it took weeks and much flamage for me to even convince the Evolution developers that there was a bug, because their machines are all so fast that the performance problem was imperceptible.

Compare this to the Windows XP desktop which sucks just as much memory, but is SNAPPY on a sub-GHz machine.

Another example, on my slow (600MHz) machine, I didn't even realize OO *had* right click menus, because after right clicking in the document they take about 3 seconds to appear. It feels like OO is generating the context menu from some XML template or something and then popping it up AFTER the right click is received. Completely unusable. Compare Windows, which has obviously generated, rendered and cached these menus and just has to blit them in from offscreen when the user right clicks (instantaneous).

Memory isn't the problem, people. We're wasting WAY too many CPU cycles on bullshit.

shared libraries [ab]use

rlrevell — Sat, 06 Aug 2005 17:59:16 +0000

Heh, really? It doesn't do any of that fancy stuff for me.

Right clicking only lets me "Copy Date", "Copy Time", "Adjust Date & Time" or set the "Preferences".

Left clicking brings up a view of the current month, that only lets me scroll back and forth to select a different day and month. AFAICT there's nothing you can DO with it.

Are you sure you did not mean "in the FUTURE this will provide access to evolution calendaring functionality?"

Our measurement problem

sandmann — Sat, 06 Aug 2005 14:00:50 +0000

The kernel actually has a system call - mincore - that tells you which pages of a mapped file are in memory. I wrote a small program a long time ago that uses it to print more details about memory usage:

http://www.daimi.au.dk/~sandmann/freon.c

You have to be root to run it.

Unfortunately a similar thing can't be done for malloc()d memory because mincore doesn't work for anonymous pages.

Our bloat problem

dmantione — Sat, 06 Aug 2005 11:22:08 +0000

In other words, it is bloat. All that people want fom libpng is read and
write png files and I doubt it is being used for more than that in the
majority of situations.

I know the differences between iff and png. I'd say png is even easier to
read than iff.

shared libraries [ab]use

hp — Fri, 05 Aug 2005 20:25:28 +0000

Avoiding anything "not free" isn't a useful guideline.

The question is what are the tradeoffs. In the shared lib case, for example, avoiding them often means efficiency loss:
- more code running total considering all apps
- each self-implemented version of the code is slower and buggier
than a highly polished common version

Not to mention downsides such as security updates (you want to download libpng, or every single app using libpng?), bugs, lack of features, longer development time.

So if shared libs aren't a significant percentage of the overall problem, then "not free" is irrelevant; if they are a significant percentage of the overall problem then "not free" is probably more an argument for fixing shared libs (or coming up with some less-broken module system than ELF shared libs) rather than an argument for static linking and hand-rolled broken code.

Simplistic answers like "just don't use libs" are not useful. You don't want to warp the UI and implementation of every app in extensive ways just to work around a module system that needs optimizing.

Our bloat problem

roelofs — Fri, 05 Aug 2005 19:56:17 +0000

Also don't forget the small ones, libpng is over 200kb on my system, just to interpret some metadata grouped into fourcc chunks. This exclusive decompression,

That's a rather simplistic view. libpng supports 19 pixel formats, depth-scaling, a moderately complex 2D interlacing scheme, alpha compositing, CRC verification, and various other transformations and ancillary bits. It also includes row-filtering, which is a fundamental component of the compression codec. I won't defend everything that's gone into libpng, but it's highly misleading to refer to all of it as bloat. If libpng (or a higher-level library built on top of it) didn't include it, all of your PNG-supporting applications would have to.

that is in libz (better, 80kb, but I remeber Pkzip being 30kb on my MS-Dos system).

Your memory is slightly faulty there. PKZIP 2.04g was 42166 bytes, and if all you care about is compressing your files, I can do far better with my 3712-byte trunc utility--and at 100% compression, too! But if you'd actually like to decompress them again someday, you'd better add PKUNZIP 2.04g, which rang in at 29378 bytes. IOW, the PKWARE codec was 71544 bytes (plus a number of other standalone utilities), while my copy of libz.so.1.2.3 is 71004 bytes--and 1.2.2 was 66908 bytes. Keep in mind also that significant parts of PKZIP and PKUNZIP were written in assembler, which, though capable of producing much smaller binaries, is generally not considered by most of us to be the most productive or maintainable of programming languages. (And, btw, libpng includes additional MMX assembler code for decoding row filters and for expanding interlace passes.)

I'm sure the Commodore Amiga was able to read iff files (on which png is based) in less code.

PNG was not based on IFF. The gross structure may have been suggested by someone familiar with IFF, but IFF itself was considered and rejected as a basis for PNG.

Greg

shared libraries [ab]use

dann — Fri, 05 Aug 2005 16:38:11 +0000

Well, using lots of shared libraries affects the startup time.
Here's an example:

echo 'int main (void) {return 0;}' > t.c
gcc t.c
LD_DEBUG=statistics ./a.out 14237:
14237: runtime linker statistics:
14237: total startup time in dynamic loader: 550907 clock cycles
14237: time needed for relocation: 250684 clock cycles (45.5%)
14237: number of relocations: 99
14237: number of relocations from cache: 5
14237: number of relative relocations: 0
14237: time needed to load objects: 158487 clock cycles (28.7%)
14237:
14237: runtime linker statistics:
14237: final number of relocations: 100
14237: final number of relocations from cache: 5

gcc t.c -lgnomevfs-2
env LD_DEBUG=statistics ./a.out
14253:
14253: runtime linker statistics:
14253: total startup time in dynamic loader: 14162855 clock cycles
14253: time needed for relocation: 10427627 clock cycles (73.6%)
14253: number of relocations: 1757
14253: number of relocations from cache: 2501
14253: number of relative relocations: 0
14253: time needed to load objects: 3357732 clock cycles (23.7%)
14253:
14253: runtime linker statistics:
14253: final number of relocations: 1793
14253: final number of relocations from cache: 2501
(this is on a 2.8GHz P4)

Obviously the above is only a small part of the startup time.
Creating and maintaing the page tables, running _init for each .so,
etc etc are other parts.

Maybe a kernel person can tell us what are the kernel overheads from having lots of shared libraries...

shared libraries [ab]use

dmantione — Fri, 05 Aug 2005 13:11:53 +0000

No, that's the wrong assumption. Shared libraries are less expensive than
statically linked ones, but not free.

Our bloat problem

dmantione — Fri, 05 Aug 2005 09:37:16 +0000

2M for a script. Try to measure again for a real app, say Mandrake's
urpmi and you'll see the Perl overhead starts to cost.

I'm coding a lot in TCL by the way, but Pascal is my main language, as
I'm developer for Free Pascal. Incidentally Free Pascal is a very good
tool for fighting bloat.

shared libraries [ab]use

hp — Fri, 05 Aug 2005 01:51:08 +0000

Those specific ones must be gnome-vfs then, for https and authentication.

It's simple enough to cut down library deps for certain apps, but I doubt it makes much real performance difference. (unused code isn't mapped anyway)

Maybe it does, but that's why real data is needed.

shared libraries [ab]use

dann — Thu, 04 Aug 2005 22:16:31 +0000

OK, ignore the clock then, let's look at any other applet, just as an example:
ldd /usr/libexec/null_applet | wc -l
63
and it does link to the krb, ssl, gss*, gnome-keyring libraries (this seems
to be true for all the applets that I have tried, and in general all the Gnome applications seem to link to a huge number of libraries).

Emacs?

iabervon — Thu, 04 Aug 2005 21:00:16 +0000

I'm surprised that your emacs is so large. My experience with it is that it could actually be an acronym for "eight megs and constantly swapping", because it's always been about eight megs big (which I can barely remember being enough to constantly swap).

If you're getting that size by using a bunch of emacs processes, you should probably look into emacsclient, which acts like an editor but edits the file in an existing emacs process. This also makes it practically instantaneous to start.

It's about time...

jrigg — Thu, 04 Aug 2005 20:47:36 +0000

>It's significant that most people don't
>_use_ the "lightweight" apps, though,
>unless they really genuinely don't
>have the compute power.

I tried KDE recently after using WindowMaker and rxvt for the last few years. It was irritatingly slow even on a dual Opteron, so I reverted to my old setup. Maybe I'm impatient, but I prefer the speed of lightweight apps running on fast hardware.

Unshared memory usage

pphaneuf — Thu, 04 Aug 2005 19:49:58 +0000

I use wmcalclock, and the same command-line you get tells me 288. ps' vsz is 2868.

Maybe I'm just too old school or something.

shared libraries [ab]use

hp — Thu, 04 Aug 2005 19:33:55 +0000

The talk about the clock is sort of bogus; the clock isn't just a clock, it's a small evolution frontend with all the calendar stuff in there. Those deps are coming from libedataserver (the evo backend)

The size of the calendar on the screen is small but it still has to have the same data available as a full calendar app and be able to talk to calendar servers and all that crap.

Our bloat problem

henning — Thu, 04 Aug 2005 19:32:42 +0000

The problem with qt (one big lib for small tasks) is identified, and resolved in qt-4 . Now you must only load the needed parts for the gui, or the network specific lib for example.

Our bloat problem

berntsen — Thu, 04 Aug 2005 19:29:12 +0000

If people malloc like you multiply, I see where the bloat is comming from, you have a factor of 10 wrong ;-)

/\/