make_ext4fs to generate filesystem [LWN.net]

make_ext4fs to generate filesystem

Posted Oct 8, 2020 22:39 UTC (Thu) by jhhaller (guest, #56103)
Parent article: The ABI status of filesystem formats

I wrote an alternate version of this tool at Alcatel-Lucent (twice because the first one got lost). Historically, Unix mkfs could make a filesystem from a list of files and contents, and write that filesystem to a file or device. Most any team generating filesystem images needs such a tool if they want to generate that ISO without needing root access. And, who really wants to give root access to one's CI system? We typically used this as a TFTP boot image for embedded Unix systems, and the filesystem was in RAM. That was another story, as the TFTP large block option was required, and no one had implemented a TFTP server with large block support at that time, and the filesystems were typically more than the 64K 512 byte block TFTP transfer size limit.

It was not hard to slightly modify mke2fs to take a manifest of files (and file of contents for regular files), and use the functions available to mke2fs to write the special files, permissions, directories, and file content to the ISO. I based the manifest loosely on the syntax of the contents of the proto argument of Unix 6th edition mkfs command - http://man.cat-v.org/unix-6th/8/mkfs - and added options for other file types including both symbolic and hard links. But, by using mke2fs, it at least used the functions built into the filesystem to populate the parts of the filesystem, making corruption significantly less likely. It would make two passes through the internal data model of the manifest, the first time writing inodes, and the second time writing the file contents. I don't remember if all-zero blocks were suppressed to support holes, I suspect not, as few files we installed had large all-zero blocks.

It was just more work than I wanted to do to get approval for releasing the changes externally, and get it accepted into ext4 source code, so it stayed internal. Since we never redistributed the tool externally, there was no need to share it, even though the change was under GPL. Sorry, Google and Ted Ts'o.

to post comments

make_ext4fs to generate filesystem

Posted Oct 9, 2020 2:58 UTC (Fri) by djwong (subscriber, #23506) [Link] (2 responses)

Yeah... mke2fs -d takes care of slurping a directory tree into the new filesystem now.

Also, that's a very interesting man page link-- now I've learned where the mkfs.xfs "protofile" format comes from! The manpage for mkfs.xfs even features a very similar example file.

Er, thanks!

make_ext4fs to generate filesystem

Posted Oct 9, 2020 15:05 UTC (Fri) by jhhaller (guest, #56103) [Link] (1 responses)

The -d option would still require root access to allow creating device file entries which may not match the host's devices.
Setting the correct file ownership is another aspect requiring root access. The protofile addresses both of these issues,
as it overwrites the filesystem owner, permissions, and allows creating device files. I doubt mkfs was originally
concerned with CI systems, but creating filesystem images on a non-native host was likely an early concern.

make_ext4fs to generate filesystem

Posted Oct 12, 2020 16:36 UTC (Mon) by mebrown (subscriber, #7960) [Link]

The yocto build system has a utility called 'pseudo'.

It uses an LD_PRELOAD library such that anything doing a mknod() libc call is saved to a database instead of invoking the underlying mknod() call, and instead just creates a regular placeholder file.

Then when you run mke2fs, similarly the LD_PRELOAD then hooks stat() such that mke2fs reading the placeholder file actually reads the details from the database instead of the filesystem.

Seems a little hacky at first, but seems to work well enough in practice.

make_ext4fs to generate filesystem

Posted Oct 9, 2020 17:40 UTC (Fri) by tytso (✭ supporter ✭, #9993) [Link] (1 responses)

Interesting. I didn't know about this feature in the 6th edition Unix's mkfs. It's not something which the BSD Fast File System supported, and my involvement with Unix started with BSD 4.3.

Using a proto file so that the system can more easily create device files does make sense, and of course these days we need to have something which supports SELinux attributes as well. That's one of the reasons why contrib/e2fsdroid exists. (The other is that e2fsdroid uses the Android libsparse file to create an image which omits the all zero blocks and which is compatible with the existing Android image creation tools, and I didn't want to drag that into mke2fs as a shared library dependency, since that would make distro installer release engineers.... cranky.)

Patches to support some kind of proto file which supports SELinux attributes, and which uses the qemu image file creation functions in libext2fs for mke2fs would certainly be gratefully accepted. If we had that, plus the per-block dedup functionality to create shared blocks file system added, then yes, we could upstream all of that functionality into mke2fs, and allow other embedded systems developers access to e2fsdroid functionality without having to build AOSP....

make_ext4fs to generate filesystem

Posted Oct 9, 2020 17:42 UTC (Fri) by tytso (✭ supporter ✭, #9993) [Link]

Sorry: s/libsparse file/libsparse library/