Weekly Edition
Return to the Development pageRecent Features
| |||||||||||||
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine)
MozillaZine points to a weblog entry by Firefox hacker Ben Goodger about memory usage in Firefox 1.5. "What I think many people are talking about however with Firefox 1.5 is not really a memory leak at all. It is in fact a feature.
To improve performance when navigating (studies show that 39% of all page navigations are renavigations to pages visited < 10 pages ago, usually using the back button), Firefox 1.5 implements a Back-Forward cache that retains the rendered document for the last few session history entries. This can be a lot of data. It's a trade-off. What you get out of it is faster performance as you navigate the web."
(Log in to post comments)
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 19:59 UTC (Wed) by dmantione (guest, #4640) [Link] How about improving the speed of the rendering engine instead? Codeoptimization seems to be regarded as black magic nowadays that proper coders stay away from :(
1.8 code *is* faster Posted Feb 15, 2006 20:37 UTC (Wed) by gvy (guest, #11981) [Link] Seems to me it got faster too.
Actually I like new Gecko's behaviour (with Seamonkey), it's only that things like such cache are better configurable. Don't see anything in about:config filtered by "cache".
Oh well.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 20:56 UTC (Wed) by jwb (guest, #15467) [Link] The speed of the engine is already pretty spiffy. The problem with other browsers that are "faster" is that they cut out 95% of Mozilla's functionality. You can render the page a lot more quickly, if you are willing to do it incorrectly.
As for the comments on "memory leaks", this is not a leak. A leak is unbounded. The back/forward page cache is bounded.
There were tons of comments on this topic on Slashdot from armchair hackers. A lot of newly-minted undergraduate computer science students held forth on the merits of special-purpose allocators, heap compaction, and so forth. Well, I hate to point out the obvious, but the Mozilla architects are actually pretty smart people. Mozilla does use special purpose allocators, arena allocators, shared strings, and all kinds of other tricks for saving time and memory. It is a very good system, it works well, and it takes a reasonable amount of memory for the job it does.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 21:30 UTC (Wed) by k8to (subscriber, #15413) [Link] All right, I'll bite.
Is consuming 40 megabytes of real memory on initial launch before a single page has been rendered a reasonable amount of memory? How about 200MB of resident memory with one window and two tabs after some days of use?
Both these values seem highly unoptimized from here.
When I last dug into the mozilla suite (a long time ago), XUL guis tended to use several times the amount of memory that WxWidgets did. I chalked this up to the DOM reflection design which required multiple objects in RAM for each widget or screen aspect. I don't know if XUL has been cleaned up, or if the bulk of firefox's memory allocations are actually the page content, dwarfing this issue. However, it seems to me that once the page rendering has completed, the actual job of displaying and scrolling the page is not fundamentaly different now than it it was with prior generations of browsers. Thus, I do not understand why Firefox consumes approximately four times the memory if it is truly efficient as it can be.
Feel free to educate me or point me to where I can educate myself more rapidly than with vi, gdb, and the firefox source.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 0:15 UTC (Thu) by bk (guest, #25617) [Link] Right now on my machine, firefox-bin is consuming ~70MB of writeable/private RAM (not including shared libs). That's pretty reasonable, I think, for such a complex piece of software. I wish KDevelop would get down to something like that (and it is just a glorified emacs, after all...).
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 10:15 UTC (Thu) by lacostej (guest, #2760) [Link] Saying that it works for you doesn't remove the fact that it doesn't work for many people.
I have firefox loaded in front of me with 20 tabs opened.
I close 11 tabs:
I minimize and resize the window:
Now here's my cache (about:cache):
Memory cache device
Number of entries: 1375
I am pretty sure that if I close all my tabs but one, go back to first page in history, load a new simple page (clearing the next history), firefox will still take over 300M.
So now maybe it's due to some extensions? I have the same issue on systems with very few (1 or 2) and mostly official (chatzilla, DOM) extensions.
Saying that the problem is a feature is not what I call good engineering.
Give me the tools to identify where the leaks come from. No about:mem page yet?
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 12:58 UTC (Thu) by bk (guest, #25617) [Link] That 300MB includes shared libs, many of which are not unique to the firefox process and thus probably shouldn't be counted. That's why I quoted the writeable/private stat (obtained with pmap), which is more representative.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 13:46 UTC (Thu) by lacostej (guest, #2760) [Link] 1- so why do the shared memory tops under 50M when firefox is started?
2- with top & pmap
5563 jerome 15 0 491m 374m 21m S 2.7 24.8 55:53.75 firefox-bin
jerome@dolcevita> pmap -d 5563
Not better... (but now I have around 25 tabs opened.
And after closing 12 tabs
bfba9000 4 rw--- 00000000bfba9000 000:00000 [ anon ]
Way better right?
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 21:19 UTC (Thu) by oak (guest, #2786) [Link] This might be explained by Glibc allocator which is fast & good, buttends not to free memory very easily. If you get even a small allocation at the top of the heap (e.g. new tab), freeing 400MB memory beneath that allocation doesn't return it back to the system (or make it non-resident), it's just returned back to the C-library. The small allocation could be something valid, or a leak, it doesn't matter for the heap (heap/sbrk() works like a stack). Glibc does >128KB sized allocations with mmap() i.e. not from heap, so those don't suffer from this problem. They get returned back to the system immediately when the application free()s them. BSD allocator uses mmap() & madvise() for all allocations, but that means using more memory when the memory usage is not so dynamic. Neither Linux glibc nor BSD allocator is better, they are different.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 19:54 UTC (Thu) by k8to (subscriber, #15413) [Link] I'm not sure how you evaluate 70MB of private writable memory as reasonable. I compared the memory usage to earlier browser generations, which are simpler in code paths to be sure, but should not require more memory for page display. For this task, I have used browsers that fit large and simultaneous page display in memory ranges from 4 to 10 megabytes. Firefox tends to use from 40 to 200 megabytes for the same task.
Do not forget in your evaluation of Firefox's memory footprint how much memory is being allocated in the X server for off-screen bitmaps.
I'm willing to trade some ram for improved functionality, or more stable operation due to more modern development techniques. Firefox however is a standout among all the software I regularly use as using an order of magnitude more memory. I wish it was possible to get it to use less.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 22:32 UTC (Wed) by dmantione (guest, #4640) [Link] Standard compliance sounds like a lame excuse for not being fast. You canuse that against IE, but not against Opera. Opera 9 does a lot better than Firefox on the Acid2 test, yet it is also very fast and therefore doesn't need such tricks.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 10:07 UTC (Thu) by lacostej (guest, #2760) [Link] Complying with the Acid 2 Test doesn't doesn't mean anything with regard to overall standard support. It's like saying that Linux is fast because it performs well on one benchmark.
Standard compliance is achieved through compliance over many tests.
Note that I think Opera is a very good browser.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 13:02 UTC (Thu) by bk (guest, #25617) [Link] I haven't noticed Opera 8.x being noticably faster, either in terms of rendering speed or startup time. It's a good browser to be sure, but I suspect the speed claims are either outdated or irrelevant fanboy-ism.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 20:12 UTC (Thu) by beoba (guest, #16942) [Link] How's Konqueror/Safari/KHTML at this? What's it like in memory usage?
Smart people and their ways Posted Feb 16, 2006 0:19 UTC (Thu) by man_ls (subscriber, #15091) [Link] Well, I hate to point out the obvious, but the Mozilla architects are actually pretty smart people.That is not very convincing. I'm sure Microsoft architects are super-smart people working with a huge budget; but somehow most of their products end up being horrible. Should we trust their security just because it has been the focus of the company for the last few years? And Microsoft are just the easy target. Even in my subjective experience, I consider that I am a good programmer, and have worked with people much better than myself; and yet I have participated in many shameful projects. Management was often, well, not up to par; or budget was short; sometimes goals were fuzzy; and so on. If you have a point to make, by all means make it; but just asking us to trust a product because smart engineers work in it will not do it.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 20:34 UTC (Wed) by coriordan (guest, #7544) [Link] Why not cache pages that are below a certain size? or instead of caching the last 10 pages, have pages disappear from the cache by an algorithm which deletes bigger pages sooner?
For an over-simplified example, when:
(page_size_in_k * place_in_history_list) > 2000
bin it. So a 200k page would get deleted after 10 other pages have been visited, and a 400k page would get deleted after 5 other pages have been visited. As I say, that's over simplified, but I can't believe that a practical implementation is too difficult.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 20:48 UTC (Wed) by jwb (guest, #15467) [Link] This is the opposite of the desired behavior. You need the back-button cache _more_ for a larger page, not less.
yup Posted Feb 15, 2006 21:18 UTC (Wed) by vblum (guest, #1151) [Link] That's indeed the problem from a practical viewpoint, and where Firefox 1.5 so much outshines 1.0.x ... Trivial comment? Maybe ... all I am saying is, for me, specifically, that was the singular biggest practical improvement from the 1.5 upgrade ... I do not have to wait forever for large pages to reconstruct themselves.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 21:36 UTC (Wed) by coriordan (guest, #7544) [Link] Ah. For me, I'm most annoyed by the browser lag when contacting sites. Maybe the majority agree with you.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 9:18 UTC (Thu) by hein.zelle (guest, #33324) [Link] Lag when connecting to sites is something I've recently started to notice in firefox as well, but I can't say for sure if it's since 1.5 or if 1.07 already had it. It became very obvious while trying out the graphical version of links a while ago - even though I probably shouldn't compare the two as the rendering of firefox is orders of magnitude better, links is pretty much instant with showing most pages, while firefox always seems to take a minimum amount of time, perhaps 0.5 seconds or so. If I open a new tab and hit the front page of lwn, it consistently takes over a second to show (on a relatively fast machine with 2gb ram and a solid internet connection).Is there anything known about what causes that delay, and if it's due to networking/html/server-client stuff or due to rendering being done in the background?
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 9:47 UTC (Thu) by grouch (guest, #27289) [Link] The Firefox delay you mention is one of the reasons I use Mozilla 1.7.8. That delay can get to be excruciating on dial-up.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 9:55 UTC (Thu) by rmyorston (subscriber, #6626) [Link] I don't know if this is what you're after, but there is the nglayout.initialpaint.delay setting which by default causes Firefox to wait for 250ms before rendering a page while it's waiting for data.
It's mentioned here: http://www.mozilla.org/support/firefox/tips
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 4:50 UTC (Thu) by proski (subscriber, #104) [Link] If we consider caching the contents, it's not as effective for large pages as for small ones, since the protocol overhead (DNS, TCP, server response time) is much more significant for smaller pages per unit of cached data (i.e. per page byte count).However, caching the rendering data would likely be more effective for large pages per unit of cached data, because larger pages tend to have more complex layout, so every piece of rendering data would potentially need more time to be calculated.
Disk? Posted Feb 15, 2006 21:38 UTC (Wed) by AJWM (guest, #15888) [Link] If the cache is getting that big, how about saving it to disk? That's still a heck of a lot faster than downloading it again, especially if you save the rendered version.
Implement it so that the pages "furthest" (back/forward) from the current page are disk cached, with only the adjacent couple memory cached. Preload as appropriate whenever back or forward is clicked.
Configurable thresholds, of course.
Disk? Posted Feb 15, 2006 22:24 UTC (Wed) by proski (subscriber, #104) [Link] Unless you want the prerendered pages to be useful after Firefox restarts, the disk saving functionality is already provided by the OS in the form of swap. If Firefox consumes too much memory, the least used part of it gets swapped out.
Disk? Posted Feb 15, 2006 22:44 UTC (Wed) by k8to (subscriber, #15413) [Link] That memory can be swapped to disk doesn't really mean that you should never intentionally move data to disk in an application.
The application can (at least theoretically) know things about usage patterns that the virtual memory subsystem cannot know. Performance tuning of this nature _is_ difficult and fiddly, and it is difficult to know ahead of time whether this particular approach would help or hurt.
Disk? Posted Feb 16, 2006 3:16 UTC (Thu) by mepr (guest, #4819) [Link] Ditto (IANA Software Engineer)However, the point is maybe being missed.. the change to firefox 1.5 is not to have it start caching _content_. It was already doing that. The change is that it now caches the rendering job. It already didn't have to redownload things, now it also doesn't have to re-render.
Mark
Disk? Posted Feb 16, 2006 15:51 UTC (Thu) by proski (subscriber, #104) [Link] What makes you think so? I don't think I missed anything.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 22:38 UTC (Wed) by job (guest, #670) [Link] This is plain silly. Everbody who's actually used Firefox knows it's pretty brittle. The one I'm using right now has 200MB RSS, which means I can wait until that doubles until I should restart it before I risk a crash. This is on the official build without any extensions.
Caches and buffers, sure, but there's no real reason to ever grow beyond some 100MB unless you open very big files. And crashes should never be tolerated. As long as the Mozilla developers continues to deny there are serious problems with the code, these things won't get fixed.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 15, 2006 23:53 UTC (Wed) by Mithrandir (subscriber, #3031) [Link] I haven't experienced a crash of Firefox for years. This is excluding crashes caused by plugins like java and acrobat which are pretty frequent. Totem seems to be the latest offender.
I've also noticed that Firefox peforms much better on Linux than on Windows. In fact the peformance of Windows' swapping routines has become more and more offensive to me as the Linux kernel gets better and better. It always seems to me that the application I want to use has been swapped out if I haven't touched it for half an hour or so. On linux things only get laggy if I do something crazy like open 100 images in the GIMP.
And the regularity of explorer crashing on XP is just priceless. Most people don't notice it because it just looks like the screen is redrawing. But it interrupts what you're doing and loses your state.
Sorry, this seems to have turned into a tirade against Windows... :(
Error proneness Posted Feb 16, 2006 0:29 UTC (Thu) by man_ls (subscriber, #15091) [Link] You have been lucky. Just reading my automated error reports you would have got a fair collection of URL's to crash Firefox.There's anyway no reason to gloat. I remember a study from some time ago that found that, when Firefox's and IE's engines were put to work through random input, IE behaved much better. Surprisingly. Just "being smug" (as our grumpy editor put it at the time) leads us nowhere. Let's fix what can be fixed, and try to break what can be broken.
Error proneness Posted Feb 16, 2006 3:21 UTC (Thu) by mepr (guest, #4819) [Link] hear hear.As if to make the point, I crashed firefox multiple times in a row today by opening a web page that spits 2.5Mb of binary data to the screen. It took about 3 minutes for the "this application is not responding, would you like to exit the program" dialog to work on windows. The point being that the test mentioned above concerned throwing malformed html at it (this being the degenerate case of that).
Error proneness Posted Feb 17, 2006 8:23 UTC (Fri) by Wol (guest, #4433) [Link] The trouble is, MS's html generation tools are generous with what they give. IE *needs* to be robust in what it expects, because it never knows if what it's going to be given is correct.
Free Software is far stricter about sticking to standards, so it tends to be less robust when fed duff input.
Cheers,
Error proneness Posted Feb 17, 2006 9:08 UTC (Fri) by man_ls (subscriber, #15091) [Link] Well, it definitely should not be less robust. Both do in fact deal with the same chaotic environment, i.e. the World Wide Web. It is alright to stop rendering properly or output errors; but a crash is something else, it is a Denial of Service (DoS): at least a nuisance, at most a security concern.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 2:46 UTC (Thu) by Ross (subscriber, #4065) [Link] I get crashes often when submitting forms or saving files (in fact I cringe when doing the latter with many tabs open). Of course this could be due to the fact Debian ships with 1.0.1 and I'm running on an Athlon64... :)
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 13:44 UTC (Thu) by sammythesnake (guest, #17693) [Link] 2 things that might help you.
1. If you migrate to testing, you'll get 1.0.7 which is much better (and AMD64 is supported, if I've understood correctly)
2. The "Session Saver" extension is so good I'm really hoping its functionality will be included in firefox at some point (though it does occasionally miss a beat) I encourage you to look at it: https://addons.mozilla.org/extensions/moreinfo.php?id=436
Cheers & God bless
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 18, 2006 0:20 UTC (Sat) by TwoTimeGrime (guest, #11688) [Link] Session saver functionality is planned for FF 2.
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 21:19 UTC (Thu) by patipati (guest, #28059) [Link] The only reason people use the "Back" button so often is that they don't know how to use tabs. If I want to follow a link from a page that I might want to use some more, I open the link in a tab. I hardly ever go "Back", and I don't really trust it because so many pages use sessions/cookies/form data/etc.
In my typical usage at work I have dozens and dozens of tabs open. One window is full of Bugzilla tabs, another is full of ViewCVS tabs, etc. So if I am proliferating tabs while avoiding "Back", and each tab must maintain large amounts of useless state, then this "optimization" is a complete lose-lose for me. I have recently noticed problems with Firefox going to 95% MEM so I guess now I know why.
If anybody knows how to shut this thing off, please tell me!
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 16, 2006 22:04 UTC (Thu) by patipati (guest, #28059) [Link] Ok, reading the blog entry I see this is a global cache and not per-tab. And you can turn it off. So I need to blame something else for my oft-rediculous memory consumption (or just by more RAM :).
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 17, 2006 8:27 UTC (Fri) by Wol (guest, #4433) [Link] "Just buy RAM". That may work for you, but not for others.
For example, I got my ram when it was cheap (£13 for 256Mb). EVERY slot is full, with the LARGEST dimm it'll take. If I upgrade it'll be new everything just to get a bit more ram... I've got 3x256Mb, which is getting to be small by modern standards!
Whatever happened to the old days, when 32Mb was massive... :-)
Cheers,
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 17, 2006 16:39 UTC (Fri) by sammythesnake (guest, #17693) [Link] 32 *MB* massive?
I remember when most home computers were around a thousandth that! (though my first computer had (*gasp*) 512k of ram!)
Cheers & God bless
Ben Goodger Explains Higher Memory Usage in Firefox 1.5 (MozillaZine) Posted Feb 20, 2006 10:53 UTC (Mon) by ddaa (guest, #5338) [Link] You mean, like that? http://geekz.co.uk/lovesraymond/archive/in-my-day
|
|||||||||||||