Recent Features
| |||||||||||||
The 2006 Linux Filesystems Workshop (Part III)The 2006 Linux Filesystems Workshop (Part III)Posted Jul 7, 2006 15:26 UTC (Fri) by PaulMcKenney (subscriber, #9624)In reply to: The 2006 Linux Filesystems Workshop (Part III) by arjan Parent article: The 2006 Linux Filesystems Workshop (Part III)
(1) OK, so in a sense, directories split across chunks in exactly the same way that files do, but for different reasons. Files split across chunks because they are large, while directories split across chunks because of the location of the primary inodes (or whatever a non-continuation inode is called) of the files within a given file. No replication. So one area that will require careful attention would be the performance of reading directories that had been split across chunks.
(2,3) Good to hear that rename and unlink are still atomic! I bet I am not the only one who feels this way. ;-)
(4) Also good to hear!
(5) Having specialized chunks could be a very good thing, though the administrative tooling will have to be -very- good at automatically handling the differences between chunks. Otherwise sysadmins will choke on it.
(6) OK to be incompatible, but my guess is that it will be very important to be able to easily migrate from existing filesystems to chunkfs. One good thing about current huge and cheap disks is that just migrating the data from one area of disk to another is much more palatable than it would have been a few decades ago.
(7) Good point -- I suppose that in the extreme case, a delayed allocation scheme might be able to figure out that the file is large enough to have whole chunks dedicated to its data.
(Log in to post comments)
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 7, 2006 21:24 UTC (Fri) by arjan (subscriber, #36785) [Link] for your (1) point... the good news is that hardlinks are relatively rare....
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 9, 2006 20:12 UTC (Sun) by PaulMcKenney (subscriber, #9624) [Link] Good point on hard links being relatively rare (ditto mv and rename, I would guess). But the split directories they engender would persist. With N chunks, you have only 1/N chance of spontaneous repair, resulting in increasing directory splitting over time, even with low rates of ln, mv, and rename. So my guess is that there would need to be the equivalent of a defragmenter for long-lived filesystems. (A rechunker???)
I suppose that one way to do this would be to hold some chunks aside, and to periodically re-chunk into these chunks, sort of like generational garbage collection.
But perhaps see how it does in tests and benchmarks?
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 11, 2006 23:13 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link] this is an area where changing the basic chunk size could have a huge effect.
the discussion was to split the disk into 1G chunks, but that can result in a LOT of chunks (3000+ on my home server :-). changing the chunk size can drasticly reduce the number of chunks needed, and therfor the number of potential places for the directories to get copied.
In addition, it would be possible to make a reasonable sized chunk that holds the beginning of every file (say the first few K, potentially with a blocksize less then 4K) and have all directories exist on that chunk, then only files that are larger would exist in the other chunks (this would also do wonders for things like updatedb that want to scan all files)
this master chunk would be absolutly critical, so it would need to be backed up or mirrored (but it's easy enough to make the first chunk be on a raid, even if it's just mirroing to another spot on the same drive)
this sounds like something that will spawn endless variations and tinkering once the basic capabilities are in place.
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 13, 2006 18:15 UTC (Thu) by PaulMcKenney (subscriber, #9624) [Link] Is there an influence on concurrency? Can operations local to a chunk proceed entirely independently of other chunks? Per-chunk log for local operation, with global log for cross-chunk operations? Now -that- should be trivial to implement, right??? Just a small matter of software... ;-)
But, yes, 3,000 chunks does seem a bit of a pain to manage -- at least unless there is some way of automatically taking care of them. But do the chunks really need to be the same size? Could the filesystem allocate and free them, sort of like super-extents that contain metadata?
If we explore all the options, will it get done before 2020? Even if we don't explore all the options, will it get done before 2020? ;-)
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 23, 2006 19:12 UTC (Sun) by rapsys (guest, #39313) [Link] I was wondering if you may add in your schema the following features :- each 5chunk beeing marked as md5 control sum (some of raid5 over chunk) - blacklist chunk that did 1-2 errors (as they have a weak magnetic surface for example)
I think that two idea are interesting as I readed some article about
The point is that if we are able to have 3-5 head on a hard disk, why not
The 2006 Linux Filesystems Workshop (Part III) Posted Jul 23, 2006 19:40 UTC (Sun) by rapsys (guest, #39313) [Link] Hum, I forget something.
The interest of the previous idea is if your data has been physicaly
The more interesting things is that the hard disk will remap the
The only problem I see if an overpowered writes cross the chunk
So we will need to have (group of )block checksum for the control sums'
I made two assumption :
|
|||||||||||||