Obtaining real-world aged file systems

Posted Jul 7, 2006 2:46 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

the problem with aged file systems is that you may have a file system (say ext2) that has been in use for years, with lots of stuff added at different times.

now you have a new filesystem (newfs) that you want to test.

if you just copy everything from the old filesystem to the new one you end up with a optimaly layed out newfs filesystem since all the files were created at one time, useually with each file being created in one operation (no fragmentation). The resulting performance is vastly different then if the same contents had been put there the same way they were put on the origional ext2fs filesystem.

so the next step is to not record the filesystem, but record the operations on the filesystem (each write, delete, create, etc) and replay those against the new filesystems.

but since modern filsystems delay allocations for several seconds when something is done to them the replays end up with many of the operations canceling out in memory (never hitting disk) and so the result still doesn't match.

the work-around for this is to do lots of sync's to force the filesystem to actually perform the writes to disk instead of short-circuiting them.

Posted Jul 7, 2006 15:27 UTC (Fri) by Max.Hyre (subscriber, #1054) [Link]

[Darn it, someone else comes up with the same idea while I'm writing my comment. I'll post this anyway, in hopes it's got some useful additions.]

Another issue is the fact that file system performance is usually only tested on fairly young, unfragmented file systems. The file systems development community should work with researchers on better ways of creating aged file systems quickly.

Seems to me the best way to create aged filesystems quickly is to have them already prepared, waiting to be used. Ask people to send in backups of live old systems, and create a library of them---you need a few, go pick them up from the filesystem store.

Obviously this has two drawbacks: it's a bit much to have a decent number of examples when each one is multiple terabytes, and people would probably object to passing their data out for world-wide use.

We could address both by having the donors use a program which saves only the metadata including its position on disk. Remove all the file contents, rename the files to something original like `1', `2', ..., and the result should be both a good bit smaller and free of private information. The restore program (not a standard restore, which arranges things neatly as it goes) would put the metadata in the same location on the new disk, thus giving the same fragmentation, ordering of filenames in directories, &c.

Of course some organizations [say, the NSA] might consider even file sizes and directory structures, to say nothing of timestamps, too sensitive to release. Think traffic analysis. Actually, timestamps could probably be omitted, too.

In fact, the dehydrated FS need only contain that metadata needed for the testing, and everything else omitted, to be faked at reconstitution time.

Need a file system? Pull one off the shelf, reconstitute by doing a special restore which writes all data as nulls (or doesn't even write it---just take whatever happens to be lying around in the data blocks), generate new values for incidental metadata, such as timestamps, and voila! You're ready to go.

The skeletal systems could even be classified by characteristics: need to test against a ton of small files? Use one of these. Want ugly fragmentation? Use one of those.

Of course, there would need to be a large set to draw from, to avoid optimizing FSes for a fixed, though realistic, set of instances. New old ones would be continually solicited to avoid ossification. On the other hand, they could be used like PRNGs: when you want to test different implementations against the same data, you've got it.

Or has this already been put into place, and I just haven't noticed?

Posted Jul 7, 2006 20:26 UTC (Fri) by pimlott (guest, #1535) [Link]

Isn't this what e2image does?