Garrett: ext4, application expectations and power management [LWN.net]

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 2:37 UTC (Mon) by Tara_Li (guest, #26706)
Parent article: Garrett: ext4, application expectations and power management

I'm at a loss on one thing...

When I start up Gnome, or KDE, I'm not making any changes to my configuration - so why are all of the config files getting re-written? For that matter, why are they using a truckload of tiny files, instead of one big file of key=value lines? That way, you only update the metadata once, you pull it all into memory, edit it however you need to, and once you're done editting it, dump it back to the HD in one single write operation...

This trash of tiny little files buried down in a tree under a .directory, often containing binary data making it a PITA to manually edit in case of complete Windows-ism. Looks just like the stupid Windows Registry that is at the heart of 90% of the problems in Windows, as far as I can tell.

to post comments

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 4:38 UTC (Mon) by flewellyn (subscriber, #5047) [Link] (8 responses)

>When I start up Gnome, or KDE, I'm not making any changes to my
configuration - so why are all of the config files getting re-written?
For that matter, why are they using a truckload of tiny files, instead
of one big file of key=value lines? That way, you only update the
metadata once, you pull it all into memory, edit it however you need to,
and once you're done editting it, dump it back to the HD in one single
write operation...

You would think, wouldn't you? Lots-of-tiny-files is not a good idea for performance reasons, although I can see the advantage in not having to do any parsing, just traverse the directory tree to find the file with the setting you want.

But truncating and recreating them with (possibly) the same data? That's just insane. There is NO good reason for doing that. Even if the files are used the way I described above, and you need to make a change, there is no good reason to use O_TRUNC. Open a new file, write, fsync, close, rename. That's the safe way to do it. And don't recreate a file that isn't changing, for heaven's sakes! What are those desktop environment developers thinking?

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 5:24 UTC (Mon) by bojan (subscriber, #14302) [Link] (6 responses)

> there is no good reason to use O_TRUNC

O_TRUNC is used on the new file:

3.b) fd = open("~/.kde/foo/bar/baz.new", O_WRONLY|O_TRUNC|O_CREAT)

That's because the file by the same name (i.e. baz.new) may exist and contain random garbage, which is then removed by O_TRUNC.

> And don't recreate a file that isn't changing, for heaven's sakes!

Yeah, that's one of the key things. Don't touch what didn't change.

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 6:12 UTC (Mon) by flewellyn (subscriber, #5047) [Link] (5 responses)

Sorry, I should have been more specific. There's no good reason to use O_TRUNC on a file that the program itself created. Especially a config file.

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 8:46 UTC (Mon) by gdt (subscriber, #6284) [Link] (4 responses)

So how do you propose to retain ownership, permission bits, creation time and other attributes? Or is your's one of those annoying applications which alter the permissions that I set manually?

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 14:40 UTC (Mon) by flewellyn (subscriber, #5047) [Link] (3 responses)

Reading the ownership, permissions, and other attributes from the existing file is not an onerous task. As for creation time, if it's a config file in the "one setting per file" mode, that really doesn't need to be preserved.

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 15:40 UTC (Mon) by cortana (subscriber, #24596) [Link] (2 responses)

It is very onorous!

You have to replicate *all* of the following!

* user and group owner
* mode
* ACLs
* XFS ACLs (it uses its own non-posix API I think)
* attributes (some of which only root may set)
* extended attributes (some of which only root may set, and some of which only root may read!)
* user extended attributes
* XML user extended attributes (again, it uses its own non-posix API)
* reiserfs extended attributes (again, I believe it uses its own non-posix API. And probably reiser3 and reiser4 use different APIs...)

And this is just for apps for Linux. If you want your program to run on Windows, Mac OS X, FreeBSD, etc. etc., you have an entirely different set of tasks to perform...

Garrett: ext4, application expectations and power management

Posted Mar 17, 2009 12:13 UTC (Tue) by nye (guest, #51576) [Link] (1 responses)

Surely you can make a copy of the file and truncate that - or am I being naive?

Garrett: ext4, application expectations and power management

Posted Mar 17, 2009 13:23 UTC (Tue) by cortana (subscriber, #24596) [Link]

Is there a system call to duplicate a file and all those properties?

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 15:21 UTC (Mon) by nix (subscriber, #2304) [Link]

Does KDE recreate everything when you start it? I see no sign of this.
Indeed the ksycoca system takes exactly the approach you suggest,
*reading* all the KDE configuration (heaps of tiny files, some
in /usr/share, some in ~/.kde) ahd dumping it into a binary file
in /var/tmp for use by the KDE apps.

Regular rewriting of the configuration files certainly doesn't happen
after that, because KDE maintains inotify watches on all the configuration
files and reruns kbuildsycoca whenever they change. This takes several
seconds so you do *not* want to trigger it unnecessarily, by doing
pointless writes.

Looking at my KDE session now, only one process has any files under ~/.kde
open, and that's akregator and those are its archive files which are most
definitely *not* small files, nor rewritten constantly (they come to about
100Mb altogether).

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 5:13 UTC (Mon) by drag (guest, #31333) [Link] (1 responses)

> Looks just like the stupid Windows Registry that is at the heart of 90% of the problems in Windows, as far as I can tell.

The windows registry is a real database that stores it's data in few hive files.

The developers for the Linux desktop are using the file system as the backing store for their settings. Which is different. This way is much superior in a stability and usability way, but tends to tax the file system.

Of course Linux is Unix, right? Very file oriented... And thus it should be able to handle lots of small files in a easy manner, right?

Because I REALLY REALLY dislike the tendency for developers to be lazy and simply dump their configurations into a SQLite file. Flat file access is actually usually much faster and robust. The single file or sql approach is actually very Windows-like. The wrong bit out of order and your toast.

What Tso recommended for desktop developers to do, which is use a database or single file for storing everything... That is actually what Windows does. (Maybe the registry concept makes better sense now? Considering it was designed for Fat32 days..)

------------------------

You say one big file for storing hundreds of application's configs would be superior, but I don't think so.

Not any more then mbox format is superior to maildir.

Remember your not dealing with a single application, or even a dozen applications. The Gnome or KDE folks are trying to design a generic API that can be accessed and used by thousands of programs without them stomping on each other and without the application developer really having to focus on low-level features.

With Gconf I can 'reset' a application simply by logging out then deleting it's configuration out of my home directory. And it's much easier to deal with then having each application trying to dream up it's own .*rc file format (mostly because I only have to learn the syntax once rather then a thousand different syntaxes and almost all the key/value pairs are self-documenting via the gconf-editor interface.. they give you possible values and a English description of what they do quite often.)

Now I don't know how KDE deals with stuff like that. KDE always has been much harder for me to deal with then Gnome... but to each their own.

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 15:27 UTC (Mon) by nix (subscriber, #2304) [Link]

KDE has a daemon that keeps an inotify watch on all the configuration
directories, and rebuilds its binary-form system configuration cache from
the configuration state whenever that state changes.

Garrett: ext4, application expectations and power management

Posted Mar 16, 2009 15:33 UTC (Mon) by cortana (subscriber, #24596) [Link]

Gconf will actually quite happily write to a single data file. But for some reason, the default is for a million little directories, each with a data file containing just the values for that directory in the Gconf hierachy; and there is no user-accessible way to switch from one format to the other.

I just checked gconf's README.Debian file and it says,

> By default, the home directory structure is created as a tree layout
> since it improves write performance. If you want to use a merged tree
> on the home directory, you should run the following command:
> gconf-merge-tree ~/.gconf/

It occurs to me that the write performance would not be an issue if it didn't write out the entire file each time an entry was changed, but instead collected up the logical writes and did a single rewrite of the %gconf-tree.xml file after a certain period of idle time (or 5 seconds maximum). I should probably file a bug about that...