|
|
Log in / Subscribe / Register

Changing filesystem resize patterns

Changing filesystem resize patterns

Posted May 12, 2022 14:44 UTC (Thu) by Hello71 (guest, #103412)
Parent article: Changing filesystem resize patterns

> While the problems have been identified, solutions have not been; he was hoping that attendees had some interesting ideas. One thing that would be useful, Ts'o said, is to have a standard format for large filesystem images that could be inflated onto block devices into the full size of the filesystem. The ext4 developers have been experimenting with using the qcow format; there is a utility in e2fsprogs called e2image that can create these images. They only contain the blocks that are actually used by the filesystem, so they are substantially smaller than the filesystem they will create. The XFS developers have also been looking at xfsdump, since it has some similar capabilities, but for XFS filesystems.
>
> When he and Wong were talking, they agreed that some single standard format would be useful. One possibility is qcow, but it is not well-specified and the QEMU developers, who created the qcow format, discourage its use as an interchange format, he said. Perhaps there are others that could be considered, but getting agreement between the various filesystems is important. That would help move away from the idea that dd is the state-of-the-art tool for transferring filesystems.

why not use tar or cpio? as i see it, the benefits of raw filesystem images are: 1. they can be directly mounted, and 2. they can be installed on a storage device using extremely simple and common programs and algorithms (useful for e.g. minimal recovery environments). if you use qcow or some other custom format, neither of these benefits applies anymore, and you may as well use tar (which is more widely supported than your new format).


to post comments

Changing filesystem resize patterns

Posted May 12, 2022 22:03 UTC (Thu) by dgc (subscriber, #6611) [Link] (1 responses)

> why not use tar or cpio?

Because it's much slower than a sequential copy from a linear file to a linear block device. The OS and storage devices are highly optimised for efficient/fast sequential data operations, tar/cpio have to do heaps of metadata modifications to create all the small containers (dirents, files, etc) that end up getting spread all over the place (can result in small random metadata writeback IO) before it can do all the data copying. And even the data copy is sub-optimal in that it runs as lots of small discontiguous data writes spread all over the device.

The whole point of using qcow2 as an image transport mechanism is that it ships as a linear file, but the internal format is a set of a offset-based data regions that can be copied directly to the storage. And with image building, the initial qcow2 image will actually be written as ascending offset order regions. IOWs, instead of a single sequential data copy to the block device, it turns it into a sequential image file read and a set of smaller sequential {seek, write} (or pwrite()) operations that are run in ascending address order. That's almost as efficient and fast as a straight linear copy, but it skips both transporting and copying all the empty space in the filesystem image.

IOWs, using qcow2 as a image transport mechanism means we gain sparse image distribution support and we lose almost no performance at the image building stage or the "copy image to storage device" deployment phase. We also gain the ability to efficiently ship near empty 100TB filesystem images containing the container image so they only take up as much space as the existing linear 2GB image technique. Then if we deploy the sparse image to sparse storage devices....

As I said, we already have almost all the pieces we need to solve this whole class of issues once and for all with simple mods to image building, transport, deployment and kernel filesystem accounting.

-Dave.

Changing filesystem resize patterns

Posted May 18, 2022 17:52 UTC (Wed) by MrWim (subscriber, #47432) [Link]

A file system image inside a tar file created with --sparse would have the same properties IIUC.


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds