Weekly Edition
Return to the Kernel page
| |||||||||||||
Squashfs submitted for the mainlineThe Squashfs compressed filesystem is used in everything from Live CDs to embedded devices. Many or most distributions ship it in such situations, but squashfs has been maintained outside of the mainline kernel for years. That appears to be changing as it was recently submitted for inclusion in the mainline by Phillip Lougher. The reaction has been generally favorable, with Andrew Morton requesting that Lougher move it forward: "Please prepare a tree for linux-next inclusion and unless serious problems are pointed out I'd suggest shooting for a 2.6.29 merge." So it seems like a good time to take a look at some of the features and capabilities of Squashfs. The basic idea behind Squashfs is to generate a compressed image of a filesystem or directory hierarchy that can be mounted as a read-only filesystem. This can be done to archive a set of directories or to store them on a smaller capacity device than would normally be required. The latter is used by both Live CDs and embedded devices to squeeze more into less. It has been nearly four years since Squashfs was last submitted to linux-kernel. Since that time, it has been almost completely rewritten based on comments from that attempt. In addition, it has gone through two filesystem layout revisions in part to allow for 64-bit sizes for files and filesystems. Another major change is to make the filesystem little-endian, so that it can be read on any architecture, regardless of endian-ness. The mksquashfs utility is used to create the image, which can then be mounted either via loopback (from a file) or from a regular block device. One of the features added since the original attempt to mainline Squashfs—to address complaints made at that time—is the ability to export a Squashfs filesystem via NFS. Squashfs uses gzip compression on filesystem data and metadata, achieving sizes roughly one-third that of an ext3 filesystem with the same data. The performance is quite good as well, even when compared with the simpler cramfs—a compressed read-only filesystem already available with the kernel. According to Lougher, these performance numbers were gathered a number of years ago, with older versions of the code; newer numbers should be even better. Previously, some kernel developers were resistant to adding another compressed filesystem to the kernel, so Lougher outlines a number of reasons that Squashfs is superior to cramfs. Certainly support for larger files and filesystems is compelling, but the fact that cramfs is orphaned and unmaintained will likely also play a role. In addition, Squashfs supports many more "normal" Linux filesystem features like real inode numbers, hard links, and exportability. Morton had a laundry list of overall suggestions for making Squashfs better in the email referenced above, but documentation is certainly one of the areas that is somewhat lacking. In particular, Squashfs maintains its own cache, which puzzles Morton:
Why not just decompress these blocks into pagecache
and let the VFS handle the caching??
The real bug here is that this rather obvious question wasn't answered anywhere in the patch submission (afaict). How to fix that? Methinks we need a squashfs.txt which covers these things. One of the reasons that Squashfs doesn't use the page cache is that it allows for multiple block sizes, from 4K up to 1M, with a default of 128K. Better compression ratios can be achieved with a larger block size, but that doesn't work well with the page cache as Jörn Engel notes: "One of the problems seems to be that your blocksize can exceed page size and there really isn't any infrastructure to deal with such cases yet."
Lougher has moved the code into a git repository, presumably in preparation to get it into linux-next. He notes that the CE Linux Forum has been instrumental in providing funding over the last four months to allow him to work on getting Squashfs into the mainline. With the additional testing that will come from being included in linux-next, it seems quite possible we could see Squashfs in 2.6.29. (Log in to post comments)
Squashfs submitted for the mainline Posted Oct 30, 2008 1:18 UTC (Thu) by sbergman27 (guest, #10767) [Link]
Squashfs can also make use of lzma compression which achieves an amazing level of compression (depending upon the data, of course). I believe that capability exists in 3rd party patches and is not part of the current, official SquashFS. OpenWRT uses this for their router firmware images, and what they are able to pack into a few MB is nearly unbelievable.
Squashfs submitted for the mainline Posted Oct 30, 2008 9:48 UTC (Thu) by Trou.fr (subscriber, #26289) [Link]
Talking about LZMA, are there any explanations for why it is not going to be included in the kernel (it seems) ?
Squashfs submitted for the mainline Posted Oct 30, 2008 10:02 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
I've seen some discussions about the possibility of using LZMA for the boot kernel, and one issue that I've seen is that LZMA can use incredible amounts of ram under corner cases
Squashfs submitted for the mainline Posted Oct 30, 2008 11:43 UTC (Thu) by Tuxie (guest, #47191) [Link]
But AFAIK that depends on how the data was compressed, which parameters was used. You CAN do LZMA compression with extreme parameters which will only slightly improve compression but will take 100x the time and/or RAM usage for both encoding and decoding. Just use sane parameters when compressing the kernel and everything will be fine...
Squashfs submitted for the mainline Posted Oct 30, 2008 11:59 UTC (Thu) by epa (subscriber, #39769) [Link]
But can you guarantee a maximum memory usage for decompressing? With gzip you can allocate a 32kbyte window and a little bit of housekeeping and you're guaranteed to decompress any valid stream. No dynamic memory allocation is needed. Can LZMA say the same?
Squashfs submitted for the mainline Posted Oct 30, 2008 13:22 UTC (Thu) by ajb (guest, #9694) [Link]
If you're doing the compressing, you ought to be able to guarantee a memory limit for decompression. Just try decompressing while you still have the original around, and tweak the compression parameters if you run out of memory. That's how MPEG encoders work - consumer products that do mpeg stream decompression have a defined amount of memory (2Mbits, if I remember correctly).
Admittedly that might be considered an expensive solution, but it proves that a solution exists.
Squashfs submitted for the mainline Posted Oct 30, 2008 13:42 UTC (Thu) by Tuxie (guest, #47191) [Link]
According to Wikipedia, "LZMA is effectively deflate (zlib, gzip, zip) with a larger dictionary size, 32MB instead of 32kB".
It also say: "the amount of RAM required during decompression is principally determined by the size of the sliding window used during compression. Small code size and relatively low memory overhead, particularly with smaller dictionary lengths, make the LZMA decompression algorithm well-suited to embedded applications."
I'm not sure if that's a good enough answer. :)
Squashfs submitted for the mainline Posted Oct 30, 2008 9:51 UTC (Thu) by Tuxie (guest, #47191) [Link]
I was going to post about LZMA support also. AFAIK the only reason it's not part of the official SquashFS is to make the LKML beards less resistant to including it in mainline. Too bad, because it makes quite a big difference in read-performance on slow media such as CDs, NBD and USB sticks, and space on storage-starved embedded devices.
That's not the question of pointless resistrance Posted Oct 30, 2008 10:46 UTC (Thu) by khim (subscriber, #9252) [Link] Squashfs with LZMA support includes all this code in filesystem itself and that's "just wrong(tm)". It must be available for other pieces of kernel as well. I suppose squashfs developers can do this - but that's move patch from "yet-another-filesystem-patch" (easy to get approval and integrate) to "yet-another-patch-with-changes-to-core-API" (much harder to process). So the split is right. I sure hope developers will not stop at just adding squashfs but will proceed with LZMA support as well, but... later.
Squashfs submitted for the mainline Posted Oct 30, 2008 20:03 UTC (Thu) by ikm (subscriber, #493) [Link]
Wow, at last this is happening! I've always wondered why didn't they do that. I've used squashfs numerous times in numerous projects, and never understood how such a popular filesystem can stay out of tree for such a long time. At last, this is changing.
Btw, I'd vote for LZMA in squashfs as well. Used it, liked it.
Squashfs submitted for the mainline Posted Oct 31, 2008 16:01 UTC (Fri) by giraffedata (subscriber, #1954) [Link] Another major change is to make the filesystem little-endian, so that it can be read on any architecture, regardless of endian-ness. I'll bite. What's the connection between being little-endian and being readable on all endianness machines?
Squashfs submitted for the mainline Posted Oct 31, 2008 16:26 UTC (Fri) by droundy (subscriber, #4559) [Link]
I believe this is in contrast to always using the endianness of the machine you're on. You could alternatively store the endianness in the FS itself, but if you just assume the endianness of the machine you're running on then you won't be able to read a given squashfs filesystem on both little- and big-endian machines.
Squashfs submitted for the mainline Posted Oct 31, 2008 18:24 UTC (Fri) by jake (editor, #205) [Link]
> I believe this is in contrast to always using the endianness of the
> machine you're on.
This is correct I believe. mksquashfs would make it for the endian-ness of the machine it was run on (it also had options to make it for the other endian-ness for cross-fs-creation). A kernel could only read a squashfs made for its endian-ness. So, the fact that it is now little-endian is somewhat immaterial, it is that the endian-ness is fixed that matters.
jake
|
|||||||||||||