Fsyncers and curveballs (the Firefox 3 fsync() problem)

> Does it suck as much as file corruption wiping out your bookmarks after your computer (not
Firefox) crashes? 

1) write to bookmarks.new , 
2) every 5 min  
   a) fsync
   b) mv -f bookmarks.new bookmarks.html
   c) open bookmarks.new (new)
   d) write all data to bookmarks.new
3) goto 1

easy, fast(if file below 3MB) and safe , it seems FF2 does it this way?

I've been using Firefox 3 from b5 version and I recently updated it to version rc1(Debian
distribution) and, with an extremely technical expression I have to say that it sucks really
bad IMO! The handling of memory is better now, but there are a lot of regressions from version
2. Except for the problem discussed here(which, by the way, I noticed too), there are other
problems like for example a big list of websites which are handled really badly, with
appearances and usability completely destroyed. I did a rapid search of the starting date of
Firefox 3 development, and I found that a first alpha release was ready on December 2006!
After so much time, I really don't know how cn they have still so many problems. All this
without considering the fact that webkit already completely passes acid test 3, while usually
firefox(gecko) scores around 60-65%.

The RPM program also does extremely frequent fsync()/fdatasync() calls for the same "stupid"
reason. Just as Firefox, RPM should work on a non-sync copy of it's database and only sync it
once (at the end of a transaction) before renaming the copy over the original db.


Btw, for people who want to know where their system has such latencies.. the latencytop
program (www.latencytop.org) shows this and other bottlenecks right away. Linux is doing
rather poorly in some of these things, even when you ignore the "ext3 has a bad fsync, use
btrfs instead" problem ;-)

Am I the only one who has a problem in ff3 where pressing right mouse button on a link to
bring up context menu sometimes does some random action with this link instead (opens it in a
new window, adds to bookmarks, and so on)? I just wonder, this looks like a really obvious
problem, but it was in b5 and now continues to be in rc1.

There's clearly plenty of stuff broken in Firefox 3, and the "fsync seemed cheap because my OS
doesn't actually implement it /at all/" thing has beaten them up worse than they're admitting.

Here's a trivial example from the beta 5 browser in front of me. I hit "Zoom in" and it blocks
on a disk write. Hit "Zoom out" and it blocks again. In, out, in out, in, in, out out, each
time the disk blocks. Somehow, this tiny change in visual state got such a priority inside
Firefox that it needs not just to be written to permanent storage for future reference, but
actively forced to disk as an urgent priority.

The "cool feature" that's being used over and over to justify the need for yet another
database inside the Mozilla engine is that when I type a partial URL, instead of completing
the URL Firefox now needlessly searches my bookmarks, the titles of every page I've ever read,
and any other data it can get its hands on, so that it can offer ridiculous suggestions, e.g.

Previously typing L offered me various LWN URLs I'd recently visited, a Livejournal page, and
a few other things beginning with L. But now, thanks to Firefox 3's new database, I'm offered
"Garfield vs Garfield" and "JWZ Mixtape on Blogspot" because I've visited those more recently
and they have an L in them somewhere. Yes, it is that stupid.

About a dozen new hidden about:config options were introduced in the creation of this feature,
not a single one of which restores the old, well loved functionality. I've yet to decide
whether I should learn to put up with this (whereupon core Firefox devs will declare it's a
"success" even though many of their users hate it) or add to the growing numbers who've
installed a Firefox extensions solely to put it back how it was....

For most of the data the Firefox developers are offering a false dilemma. They suggest that
the alternatives are using fsync() or giving up all data integrity. In fact this applies only
because they wanted to use database structures. If you're willing to write a log structure
instead of building a database, you lose nothing. If you write both, you lose nothing, avoid
fsync() and can restore when, inevitably, the database corrupts itself without any help from
the OS or the power company.

I think the article makes it clear that Firefox is heading in the wrong direction, very much
like its predecessor Netscape. In my opinion the performance problem described is symptomatic
of a too complex product. The data storage abstraction used probably isn't the one needed.
History is linear data and I would expect it to be best stored in a log (i.e. appended), and
indexed piecemal or even on load.

But I'm not a Firefox developer so I might be missing the bigger picture here. I am however a
user, and most new features are misfeatures to me. The new history system gets all cluttered
and is useless to me. I just want to turn it off. Much like the user interface changes in 2.0
which was pretty much all for the worse. All I want is better rendering, perhaps better
performance, but above all for it to stop allocating all my memory after a while.

It seems there has been more problems around sqlite performance in Firefox. The article links
to Christopher Blizzard who describes a problem with Firefox as shipped by Red Hat: Firefox
upgraded sqlite, noticed a regression, and backed out the change. Meanwhile, Red Hat package
maintainers did not know about the regression and they shipped a faulty package.

I do find the article in question very biased towards the Firefox developer's point of view.
They would rather have their upstream release shipped in distributions. From my point of view
as a user that's not desirable for many reasons. The reason why Red Hat missed the regression
is the problem that needs to be solved, not the other way around. Why did it happend? Was it
not documented? Why did the build script even accept a library version with a known
regression?

All this points to a big closed world mentality inside Mozilla where the product is everything
and the bigger ecosystem lost. I hope that is not the case as Firefox is a very important
product and I would be sad to see it take the Netscape road.

FWIW, this problem is not limited to Firefox.  Evolution, as an example, does a disk write and
fsync on every mouse event received when you drag certain things in its interface.  Speaking
for myself, I haven't noticed these problems since I installed a huge NVRAM-fronted RAID in my
desktop computer ;)

Some people are using my libeatmydata LD_PRELOAD library (it disables fsync) to work around
the problem... however, it also removes crash-safety, so use with care.


http://www.flamingspork.com/projects/libeatmydata/

If in the unlikey event that your machine crashes, while do some mission critial web browsing,
while creating a new bookmark to an important site that you'll never remember in 5 minutes
time.

Does your world really end?

When I first found out many years ago that fsync() under Linux doesn't sync just the file I
was rather surprised and figured it was a minor oversight that would be remedied soon. Instead
we're more than 5 years later and the problem still exists.

It was like that recent article in LWN about barriers for disk I/O. I thought "that's a rather
blunt instrument, there's probably some nicer mechanism to specify exactly which blocks depend
on which others". I was wrong, the barriers are the only way, which causes the current issues.

As a person who worked on another open-source database, I'd really like this to be finally
fixed.

I want to know if JWZ thinks that SQLite in FF3 is an improvement over his original history DB
in Mozilla...

k3n.

I can't blame firefox.

It seems to me that fsync is the problem.  No user level program should have access to any
kind of system call that can cause this kind of performance problems.  Or, at least if it is a
performance problem, it should affect the errant program and not the whole system.

So I have to put the blame on the Kernel, or maybe it it is the Unix standard.

Hopefully, this will be fixed where it needs to be fixed, which is in the Kernel.