KHB: A Filesystems reading list
KHB: A Filesystems reading list
Posted Aug 22, 2006 1:14 UTC (Tue) by nlucas (guest, #33793)Parent article: KHB: A Filesystems reading list
Another excelent article by Valerie Henson for those like me who don't really want to be filesystem experts but always want to know a little more.
I'll be waiting for the next! :-)
One thing I miss is some info about any generic filesystems for the now common flash disks, or even CD+-RW/DVD+-RW. Last time I checked jffs/yafs can't be used on generic block devices, only on memory devices (or with some wrapper driver). Couldn't we be using some variation of jffs2 for CD+-RW/DVD+-RW? Maybe with the help of FUSE, as the kernel doesn't know how to write to the CD/DVD?
It seems ideal for incremental backups, at least...
Posted Aug 22, 2006 17:45 UTC (Tue)
by nix (subscriber, #2304)
[Link] (9 responses)
Posted Aug 22, 2006 23:25 UTC (Tue)
by nlucas (guest, #33793)
[Link] (8 responses)
Yes I know, but last I checked it's read-only, i.e. you can't use it on a partition as a normal filesystem, like the old msdos filesystem.
You have no easy way (AFAIK) to just add a file to a DVD-RW, like copy/paste or drag a file to a DVD-RW folder.
I would be happy enough when I can do something like this from the command line:
My home PC still have an USB 1.1 hub and so I'm used to wait until it flushes all writes to the USB pen when unmounting. I only ask for something similar for a DVD-RW (the speed should not be that much different than for USB 1.1).
And you have the advantage that you can actually prevent the user from ejecting the CD/DVD before all writes are flushed, so less risk of user generated filesystem corruption (at least for users that don't use that cute little hole to open the "cup-holder" ;-).
Posted Aug 23, 2006 1:24 UTC (Wed)
by nlucas (guest, #33793)
[Link] (1 responses)
It seemed weird, but followed your path without thinking more about it ;-)
Posted Aug 23, 2006 16:34 UTC (Wed)
by nix (subscriber, #2304)
[Link]
Posted Aug 23, 2006 16:37 UTC (Wed)
by nix (subscriber, #2304)
[Link] (2 responses)
It's ISO-9660 that has to be built in one lump.
Posted Aug 24, 2006 21:51 UTC (Thu)
by nlucas (guest, #33793)
[Link] (1 responses)
It seems the problem is only on the software side (for modern RW drives, I mean), and maybe cdrecord "flame wars" have delayed the adoption of a standard linux solution for the typical user cases (like maybe a FUSE DVD UDF filesystem that works the same as a USB pen, i.e. can add/delete files on the fly).
Well, I still have a lot of doubts about the process, but the truth is this thread is becoming off topic so I'll refrain my thoughts for latter times.
Posted Aug 27, 2006 15:53 UTC (Sun)
by nix (subscriber, #2304)
[Link]
(It does indeed have significant overhead, as you say.)
Posted Aug 24, 2006 2:12 UTC (Thu)
by wookey (guest, #5501)
[Link] (2 responses)
growisofs -M /dev/dvd files
will add files to a disk (CD or DVD).
Posted Aug 24, 2006 7:36 UTC (Thu)
by drag (guest, #31333)
[Link]
My favorite backup technic for my own home directory is to use growisofs with splitpipe.
Splitpipe is a nice utility that splits files just like 'split' does, but it performs a action on each section it makes.
Check it out:
It'll burn a DVD until it gets finished, pause, prompt you for you to press enter so you can have time to pop in a new DVD and off you go.
Then they have a 'joinpipe' command for rebuilding the data.
Supports md5sum'ng the data and makes sure you don't accidently try to join the data back together in the wrong order and such.
Of course if you have a file that is less then 4 megs you could just go:
cat filenmae |growisofs -Z /dev/dvd -
And I think that should work out fine. Then you should be able to 'cat /dev/dvd > whatnot' to get it back.
Haven't tried it personally like that, but I expect it to work fine.
Posted Aug 24, 2006 21:57 UTC (Thu)
by nlucas (guest, #33793)
[Link]
Thanks for the heads-up. I'm an happier man now :-)
Posted Aug 22, 2006 18:27 UTC (Tue)
by plougher (guest, #21620)
[Link] (3 responses)
>
Squashfs is read-only, but it does support appending to the filesystem. This was added specifically to support incremental backups.
Posted Aug 22, 2006 23:49 UTC (Tue)
by nlucas (guest, #33793)
[Link] (2 responses)
If it's 1) then it doesn't matter what filesystem is used. If it's 2), it's nice for incremental backups, but it will not allow you to take advantage of the fact that you can delete files on a -+RW disk (but good for -+R disks).
Hope it's 3)...
Posted Aug 23, 2006 1:25 UTC (Wed)
by nlucas (guest, #33793)
[Link]
Posted Aug 24, 2006 18:27 UTC (Thu)
by plougher (guest, #21620)
[Link]
You could do 2) and store the Squashfs filesystem image inside a UFS/ISO9660 filesystem. If the UFS/ISO9660 filesystem was writable, then the Squashfs file could be updated and extended in place.
>Hope it's 3)...
I always write the Squashfs filesystem directly to the disk (not within a UFS/ISO9660 filesystem). Mounting is then done by mounting directly from the disk (i.e. mount -t squashfs /dev/hda /mnt). Appending to this can be done, but it requires the ability to update the superblock (block 0), and to re-write the directory structure. I've never tried to do this, and so don't know if it is possible.
Pure appending to the Squashfs filesystem (so no blocks in the original filesystem are touched) isn't possible, because of the need to update the superblock and directory structure (stored at the end of the filesystem). Extending Squashfs to do this isn't impossible, it requires Squashfs to append the new superblock (and changed metadata) to the end of the filesystem, and then scanning for any new superblocks on mounting - i.e. what log structured filesystems do.
Posted Aug 24, 2006 2:26 UTC (Thu)
by wookey (guest, #5501)
[Link] (1 responses)
Yes. This is the case. It would be nice to have the robustness of log-strucutreed filesystems on flash devices accessed as block devices, but it's actually very difficult to achieve. The problem is that although USB sticks and SD/MMC cards and CF cards use flash, they hide it behind a controller which does who-knows-what behind the scenes in order to make it look enough like a disk that cameras can pretend it is a DOS FAT disk. The controllers remap blocks to retire bad blocks and (on the better ones) to avoid premature wear in the cluster tables at the start of the FAT format. But precisely because they do these things there is little benfit to using a filesystems like YAFFS/JFFS which knows about flash directly and does the right thing itself. You might as well just use something designed for disks.
So you can use YAFFS/JFFS on smartmedia and xD cards because they expose raw NAND flash, but not on all the others.
Various sorts of shims have been considered but the general consensus seems to be that it's not worth bothering with.
Posted Aug 24, 2006 22:08 UTC (Thu)
by nlucas (guest, #33793)
[Link]
What I mean is that we could conceive one of those micro-atx VIA boards with an IDE flash disk (I believe they use a normal Compact Flash memory) running a linux system, but FAT seems awfull for such a system (if it works at all). I tried to google about it but couldn't find a definitive answer.
CD-RW et al are what UFS is designed for. JFFS has completely different KHB: A Filesystems reading list
constraints (i.e. zero-seek-time filesystems rather than
very-high-seek-time filesystems). I haven't tried it but I'd wager the
performance of JFFS on CD-RW would be utterly awful.
KHB: A Filesystems reading list
CD-RW et al are what UFS is designed for.
$ ufs-cp somefile.txt /media/cdrw/somedir/
and let it do the rest in the background (and not wait an eternity, off course). Much like the old days with DOS formated floppy disks.
s/UFS/UDF/KHB: A Filesystems reading list
Um. Yes. Brainfart, sorry.KHB: A Filesystems reading list
What? Packet-written CD-RWs use UDF filesystems, and act just like a normal filesystem (except that the device is really rather slow and the filesystem overhead is ridiculous). In modern CD-RWs and all DVD-RWs this is even in hardware (while I haven't yet seen a CD rewriter that has packet-writing firmware that works well, this may not be true of DVD-RW drives).KHB: A Filesystems reading list
I now understand the problem has nothing to do with filesystems, except the UDF overhead problem (and ISO-9660 compatibilty), as it's the only usable CD/DVD RW filesystem (because it can be read everywhere).KHB: A Filesystems reading list
We already have exactly that feature: the packet-writing driver. If you're KHB: A Filesystems reading list
packet-writing to CD-RW (or DVD-RW) you don't *need* cdrecord, or FUSE.
growisofs meets your command line criteria, I think.KHB: A Filesystems reading list
growisofs even supports pipes...KHB: A Filesystems reading list
tar c /home | splitpipe -s dvd -o 'growisofs -Z /dev/dvd=/dev/stdin'
I'm almost certain I had read the growisofs manual page some moons ago, but somehow missed that command.KHB: A Filesystems reading list
>Couldn't we be using some variation of jffs2 for CD+-RW/DVD+-RW? Maybe >with the help of FUSE, as the kernel doesn't know how to write to the >CD/DVD?KHB: A Filesystems reading list
>
>
Try squashfs - this is a compressed filesystem which is being used a lot for liveCDs. It also achieves better compression than JFFS2.
>It seems ideal for incremental backups, at least...
KHB: A Filesystems reading list
Try squashfs - this is a compressed filesystem which is being used a lot for liveCDs. It also achieves better compression than JFFS2.
I really don't care about compression, but I suppose it doesn't hurt.
Squashfs is read-only, but it does support appending to the filesystem. This was added specifically to support incremental backups.
But how is this used on a DVD-RW?
1) You have a squashfs disk image and append to it, writing the final image to the CD/DVD when a certain size is reached?
2) You use a CD/DVD-RW (with UFS or whatever) and update a squashfs image file on the disk?
3) Other...
s/UFS/UDF/KHB: A Filesystems reading list
>If it's 1) then it doesn't matter what filesystem is used. If it's 2), >it's nice for incremental backups, but it will not allow you to take >advantage of the fact that you can delete files on a -+RW disk (but good >for -+R disks). KHB: A Filesystems reading list
Last time I checked jffs/yaffs can't be used on generic block devices, only on memory devices (or with some wrapper driver).
KHB: A Filesystems reading list
If we don't know what the controller does, is there any filesystem that is better than others on this kind of disks?KHB: A Filesystems reading list