This is what makes RPM slow as well [LWN.net]

This is what makes RPM slow as well

Posted May 26, 2008 16:07 UTC (Mon) by arjan (subscriber, #36785)
Parent article: Fsyncers and curveballs (the Firefox 3 fsync() problem)

The RPM program also does extremely frequent fsync()/fdatasync() calls for the same "stupid"
reason. Just as Firefox, RPM should work on a non-sync copy of it's database and only sync it
once (at the end of a transaction) before renaming the copy over the original db.


Btw, for people who want to know where their system has such latencies.. the latencytop
program (www.latencytop.org) shows this and other bottlenecks right away. Linux is doing
rather poorly in some of these things, even when you ignore the "ext3 has a bad fsync, use
btrfs instead" problem ;-)

(Log in to post comments)

This is what makes RPM slow as well

Posted May 26, 2008 17:19 UTC (Mon) by lmb (subscriber, #39048) [Link]

I've had a crazy idea here a few weeks ago. If you look at OCFS2, you can see that modern
filesystems can have more than one journal and coordinate across them. (Even if they might be
merged into a single journal space on disk, logically it could be several.)

You could then have processes join transaction groups - as long as the transactions happen in
unrelated areas of the filesystem, there'd be no need for coordination, and performance would
improve.

Some refinement is needed, but single journal really is sooo 1999 ;-)

This is what makes RPM slow as well

Posted May 26, 2008 17:20 UTC (Mon) by arjan (subscriber, #36785) [Link]

At least btrfs doesn't even have a journal...

... having a journal is soooo  1999 ;)

This is what makes RPM slow as well

Posted Jun 1, 2008 22:19 UTC (Sun) by jlokier (guest, #52227) [Link]

Trees and cascades of multiple journals go naturally together.  See btrfs, logfs, ubifs, zfs,
reiser4...  After a while it stops looking like a journal.  But all the tricks to make a high
performance and reliable journal with async barriers can apply to a tree as well.

sync_file_range

Posted May 27, 2008 18:12 UTC (Tue) by stephen_pollei (guest, #23348) [Link]

Isn't long sync_file_range(int fd, off64_t offset, off64_t nbytes, unsigned int flags); the
better more modern alternative to fsync fdatasync? Would smart use of that help both rpm and
sqlite?

sync_file_range

Posted Jun 1, 2008 22:15 UTC (Sun) by jlokier (guest, #52227) [Link]

Perhaps but it's not implemented all that well at the moment.  The documentation is rather
confusing and leaves out a few important things.  The flags are rather specific artifacts of
the way Linux mm works, not a natural "kernel-neutral" API for syncing a file range.