Embedded Linux: Using Compressed File Systems

Posted Feb 8, 2007 17:52 UTC (Thu) by moxfyre (guest, #13847)
In reply to: Embedded Linux: Using Compressed File Systems by endecotp
Parent article: Embedded Linux: Using Compressed File Systems

I know of only two filesystems that are both writeable and compressed: jffs2 and e2compr. jffs2 works only on directly-connected flash (i.e. not the USB and IDE flash drives that I use), and IIRC e2compr is not included in the kernel because it doesn't have any fsck support, which is rather worrying to say the least. The remaining option is to use a union filesystem to combine a read-only compressed filesystem with a writeable uncompressed filesystem. This is fine at first but becomes useless once you've "apt-get upgrade"ed - eventually it uses more space than a normal uncompressed filesystem would have done. So there's definitely scope for improvements. There are other things you could do, such as initially storing the data uncompressed and having a low-priority background process do the compressing once files seemed to be stable. Maybe this could happen as part of the journal writeback system?

I agree completely. The lack of transparent r/w compression is a frustrating situation, especially for those who want to make Live USB Flash distributions. For a Live CD, you can just use squashfs with unionfs... since the media isn't easily rewritable. But with a space-constrained installation on a USB Flash disk, I'd like to have transparent read/write support AND compression. Something like what "dblspace" or "Stacker" did under MS-DOS.

The best-known project to do this under linux, e2compr, is basically moribund. I think there has not been much demand for such a filesystem for two reasons:

Hard drives have gotten bigger so quickly that there's little need to compress most of the files on a desktop or laptop computer.
For those files that *do* take up a lot of space (audio, video, source code archives) there are domain-specific file-level compression algorithms that give much better performance than a general-purpose solution.

I imagine that the widespread use of flash-based storage will soon bring back the demand for good, transparently compressed storage. Here are a couple promising projects to add such a filesystem to Linux: compFUSEd and fusecompress.

to post comments

Embedded Linux: Using Compressed File Systems

Posted Feb 27, 2007 11:39 UTC (Tue) by matthiaswinkler (guest, #43648) [Link]

I think moxfyre is absolutely right. There are already so much
different approches, but e2compr is the only one that fullfills
the needs at my acutal project.

So, we will need to port e2compr to the toplevel 2.6-kernel.

1.) I think one of the first steps would be to make add e2compr
to cvs for easier development.

2.) The last working patch was for kernel 2.6.10. So, at first,
I would be thankful for any hints on what changed in the kernel
since that time. Changes would be in vfs and ext2-code, I guess.

3.) Maybe it would be also possible to make one unique patch, to
adopt e2compr to ext2 and ext3. Maybe we should overthink the kernel
architekture. Maybe the data-compression should happen between vfs and
the filesystem implementation. Like this:

VFS
------------
COMPRESSION + ENCRYPTION
------------
FS (ext2, ext3 ...)

Any ideas?

I would encourage everybody, to participate or at least give
me some hints, what has to be done and how!!!!

Matthias