Punching holes in files

By Jonathan Corbet
November 17, 2010

The XFS and OCFS2 filesystems currently have the ability to "punch a hole" in a file - a portion of the file can be marked as unwanted and the associated storage released. Josef Bacik, noting that this capability may be added to other filesystems in the near future, came to the conclusion that the kernel should offer a standard interface for hole punching. The result is an extension to the fallocate() system call adding that ability.

In particular, this patch adds a new flag (FALLOC_FL_PUNCH_HOLE) which is recognized by the system call. If the underlying filesystem is able to perform the operation, the indicated range of data will be removed from the file; otherwise ENOTSUPP will be returned. The current implementation will not change the size of the file; if the final blocks of the file are "punched" out, the file will retain the same length. There has been some discussion of whether changing the size of the file should be supported, but the consensus seems to be that, for now, changing the file size would create more problems than it would solve.

Index entries for this article
Kernel	fallocate()

Punching holes in files

Posted Nov 18, 2010 8:00 UTC (Thu) by stefanha (subscriber, #55072) [Link] (2 responses)

Isn't there already an FITRIM ioctl that at least ext4 implements with vfs hooks for other file systems to follow?

Punching holes in files

Posted Nov 18, 2010 13:33 UTC (Thu) by Tomasu (guest, #39889) [Link] (1 responses)

Hole Punching has very little to do with TRIM. What Hole punching does is remove unneeded blocks from an existing file. TRIM is lower level than that and acts to tell a storage device that the blocks are free.

I could actually see the file system or vfs sending down a TRIM command after a hole is punched in a file.

Punching holes in files

Posted Nov 18, 2010 13:53 UTC (Thu) by stefanha (subscriber, #55072) [Link]

I was asking about the FITRIM ioctl in particular, which I thought might serve a similar purpose. After some more digging, the best explanation seems to be:
http://lwn.net/Articles/397538/

FITRIM will only discard freed blocks in a filesystem. It does not punch new holes in a particular file.

Punching holes in files

Posted Nov 18, 2010 15:09 UTC (Thu) by Trelane (subscriber, #56877) [Link] (5 responses)

I don't see the use case for this. Clearly I'm missing something.

Punching holes in files

Posted Nov 18, 2010 15:51 UTC (Thu) by josefwhiter (guest, #39238) [Link] (4 responses)

So say you have a virt guest that sends down a discard. If the guest is on a sparse file nothing happens, you continue to have the space allocated to the sparse file, so eventually even though your guest may only be using say 4g of an 8g sparse file, it could have allocated all of that 8gig, so you are still having to deal with an 8 gig file on the host. With hole punching you can de-allocate that space, so you only ever use as much space on the host as you are using in the guest.

Punching holes in files

Posted Nov 18, 2010 16:08 UTC (Thu) by Trelane (subscriber, #56877) [Link]

Aaah, ok. I think I can see it (I'm guessing that "sending down a discard" means deleting something on a virtual disk?) Thanks!

Punching holes in files

Posted Nov 18, 2010 16:54 UTC (Thu) by chmouel (subscriber, #6335) [Link] (2 responses)

I am not sure I catch the "sending a discard" by the guest, what does it do ?

Punching holes in files

Posted Nov 18, 2010 18:40 UTC (Thu) by nix (subscriber, #2304) [Link]

"This bit of this file is mapping a part of a virtual filesystem known to be empty. Its contents do not matter."

Currently, the backing file still takes up space on the host's filesystem when that is done. It need not. (Possible downside: files with huge numbers of holes scattered through them had a historical propensity to get enormously fragmented, but for VM filesystem images this is probably completely unimportant, as the 'fragmentation' is going to be precisely in line with actual *files* in the contained FS.)

Punching holes in files

Posted Nov 19, 2010 16:55 UTC (Fri) by ewan (guest, #5533) [Link]

I suspect that this will bring us full circle to the TRIM command, and that this will actually work with the guest OS issuing a TRIM to the virtual hardware, and that being implemented on the host as 'punch hold in file', which may then also cause the host to issue a TRIM to its real underlying storage.

Useful feature

Posted Nov 19, 2010 1:51 UTC (Fri) by brouhaha (subscriber, #1698) [Link]

It will be great to have this feature in Linux, in a file-system-agnostic way.

Back when my colleagues and I developed the original ReplayTV (which was not Linux-based, alas), we used the same concept to implement the variable-sized live-TV rewind buffer. Some of the engineers wanted to use a fixed-size file as a circular buffer, but I pointed out that if we used hole-punching (and 64-bit file offsets), the live-TV buffer could potentially use all otherwise unused disk space, and that as we needed to reclaim space for new video, we could just punch a hole at the beginning of the file. Effectively, the conceptual beginning of the file moves forward as the oldest video in the buffer is discarded.

This worked out really well, though I'm not sure whether they continued to use this approach when they switched to VxWorks for the 4000-series and later units.

Traditional interface is zero-filling

Posted Nov 28, 2010 19:06 UTC (Sun) by kolk (guest, #36699) [Link] (7 responses)

Actually, there is traditional interface for both hole punching in files and releasing DASD blocks.

Standard long ago has specified that holes in sparse files are read as zeros.
Similarly, block should be freed or punched out automatically when it's overwritten with zeros, may be conditionally if it's not optimal thing to do.

There is no need for any special interfaces on any level.
And any such interface should just zero-fill underlaying storage if not supported as is, and never return its own error code.

Traditional interface is zero-filling

Posted Nov 29, 2010 1:04 UTC (Mon) by nybble41 (subscriber, #55106) [Link] (3 responses)

Sometimes you can't use sparse files. For example, swap files need to have all blocks allocated, even if those blocks happen to contain only zeroes. Also, a program may wish to fully allocate a file to ensure that later writes will be able to succeed without running out of disk space, which could happen if sparse blocks were overwritten with non-zero data.

So a separate interface is required after all.

Traditional interface is zero-filling

Posted Nov 29, 2010 12:17 UTC (Mon) by mfedyk (guest, #55303) [Link] (2 responses)

maybe you can write ones instead if you don't want the blocks deallocated

Traditional interface is zero-filling

Posted Dec 29, 2010 15:33 UTC (Wed) by yoe (guest, #25743) [Link] (1 responses)

That would be a horrible, horrible hack.

Traditional interface is zero-filling

Posted Jan 20, 2011 2:22 UTC (Thu) by jzbiciak (guest, #5246) [Link]

And here's hoping you never swap a page full of zeros by accident...

Traditional interface is zero-filling

Posted Jan 20, 2011 3:23 UTC (Thu) by brouhaha (subscriber, #1698) [Link] (2 responses)

If I want to punch a 5GB hole in the middle of a file, I don't want to have to do 5GB of writes to make it happen. And yes, I actually have a use for punching really big holes.

Traditional interface is zero-filling

Posted Jan 27, 2011 18:16 UTC (Thu) by mcortese (guest, #52099) [Link] (1 responses)

If I want to punch a 5GB hole in the middle of a file, I don't want to have to do 5GB of writes to make it happen.

Wouldn't you need to do it anyway when you allocate that space again?

Traditional interface is zero-filling

Posted Jan 27, 2011 21:34 UTC (Thu) by brouhaha (subscriber, #1698) [Link]

I wouldn't be punching a hole if I intended to fill it any time soon.

In any case, supposing that I did want to punch a big hole, then fill it, I wouldn't want to have to write the whole region twice.