| |||||||||||||
Obtaining real-world aged file systemsObtaining real-world aged file systemsPosted Jul 6, 2006 19:38 UTC (Thu) by wilck (subscriber, #29844)Parent article: The 2006 Linux Filesystems Workshop (Part III)
This is actually an idea of a colleague of mine.
Wrt the problem of obtaining real-world aged file systems, we wondered if it would be worthwile to create a "file system content scrambler" - a tool that would duplicate a file system exactly 1:1, but overwrite all real content - data, file and directory names - with artificial or random data.
The cloned file system would have the same properties as the original real-world file system without revealing possibly sensitive information to the person analyzing or doing benchmarks.
Such a tool could be used to gather file systems that would exhibit real-world fragmentation etc.
Perhaps not an idea for the truly paranoid...
(Log in to post comments)
Obtaining real-world aged file systems Posted Jul 7, 2006 2:46 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link] the problem with aged file systems is that you may have a file system (say ext2) that has been in use for years, with lots of stuff added at different times.
now you have a new filesystem (newfs) that you want to test.
if you just copy everything from the old filesystem to the new one you end up with a optimaly layed out newfs filesystem since all the files were created at one time, useually with each file being created in one operation (no fragmentation). The resulting performance is vastly different then if the same contents had been put there the same way they were put on the origional ext2fs filesystem.
so the next step is to not record the filesystem, but record the operations on the filesystem (each write, delete, create, etc) and replay those against the new filesystems.
but since modern filsystems delay allocations for several seconds when something is done to them the replays end up with many of the operations canceling out in memory (never hitting disk) and so the result still doesn't match.
the work-around for this is to do lots of sync's to force the filesystem to actually perform the writes to disk instead of short-circuiting them.
Obtaining real-world aged file systems Posted Jul 7, 2006 2:53 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link] One thought that hit me just after posting the last comment,
now that there is the new timer base in the kernel, and we are nearing useability with the 'tickless' patches, how about setting up a special kernel with a custom timesource that's driven by the process writing the aged filesystem?
it knows the timestamp for all the origional filesystem actions, and it can find out all outstanding timers, after each filesystem action have it set a timer action for the time of the next filesystem action and then use the tickless capabilities to advance time up to that point (steping through each of the other timers on the way). this way the modified kernel and it's filesystem code would think that the replay took place in the same real-time that the origional actions that are being replayed took place in, andy delayed actions will take place appropriately (benifiting things if the delay would have helped in the origional, but happening between actions if they wouldn't have)
as long as the kernel isn't busy doing other stuff at the same time this replay should be very fast, and this avoids the haphazard benifits of trying to insert lots of sync calls.
David Lang
Filesystems aged like sharp cheese Posted Jul 7, 2006 15:27 UTC (Fri) by Max.Hyre (subscriber, #1054) [Link] [Darn it, someone else comes up with the same idea while I'm writing my comment. I'll post this anyway, in hopes it's got some useful additions.]Another issue is the fact that file system performance is usually only tested on fairly young, unfragmented file systems. The file systems development community should work with researchers on better ways of creating aged file systems quickly.Seems to me the best way to create aged filesystems quickly is to have them already prepared, waiting to be used. Ask people to send in backups of live old systems, and create a library of them---you need a few, go pick them up from the filesystem store. Obviously this has two drawbacks: it's a bit much to have a decent number of examples when each one is multiple terabytes, and people would probably object to passing their data out for world-wide use. We could address both by having the donors use a program which saves only the metadata including its position on disk. Remove all the file contents, rename the files to something original like `1', `2', ..., and the result should be both a good bit smaller and free of private information. The restore program (not a standard restore, which arranges things neatly as it goes) would put the metadata in the same location on the new disk, thus giving the same fragmentation, ordering of filenames in directories, &c. Of course some organizations [say, the NSA] might consider even file sizes and directory structures, to say nothing of timestamps, too sensitive to release. Think traffic analysis. Actually, timestamps could probably be omitted, too. In fact, the dehydrated FS need only contain that metadata needed for the testing, and everything else omitted, to be faked at reconstitution time. Need a file system? Pull one off the shelf, reconstitute by doing a special restore which writes all data as nulls (or doesn't even write it---just take whatever happens to be lying around in the data blocks), generate new values for incidental metadata, such as timestamps, and voila! You're ready to go. The skeletal systems could even be classified by characteristics: need to test against a ton of small files? Use one of these. Want ugly fragmentation? Use one of those. Of course, there would need to be a large set to draw from, to avoid optimizing FSes for a fixed, though realistic, set of instances. New old ones would be continually solicited to avoid ossification. On the other hand, they could be used like PRNGs: when you want to test different implementations against the same data, you've got it. Or has this already been put into place, and I just haven't noticed?
/And/ another thing... Posted Jul 7, 2006 15:40 UTC (Fri) by Max.Hyre (subscriber, #1054) [Link] It just occurred to me that renaming files to numbers could mess things up if the directory structure depends on the filename lengths. So the rename would have to be something the same length as the original (leading zeroes?). Which means trouble when the original name is shorter than the new name....The fix is left as an exercise for the reader. :-)
/And/ another thing... Posted Jul 15, 2006 11:41 UTC (Sat) by nix (subscriber, #2304) [Link] The rename might well need to come up with something which *hashes* to the same value as the original. Good luck making *that* work for more than one file a year. :)
Filesystems aged like sharp cheese Posted Jul 20, 2006 23:49 UTC (Thu) by efexis (guest, #26355) [Link] The problem is when you want to test a new feature, such as a new algorithm for deciding where to place new blocks on the disk... the filesystem has to be created using this code. Grabbing an old filesystem that wasn't created using this code is completely useless.
You need to replay all the actions, all the file creates, writes, moves, deletes etc, in an order they would actually happen, to see the result.
Obtaining real-world aged file systems Posted Jul 7, 2006 20:26 UTC (Fri) by pimlott (guest, #1535) [Link] Isn't this what e2image does?
|
|||||||||||||