very useful

Posted Jan 22, 2015 15:53 UTC (Thu) by jtaylor (guest, #91739)
Parent article: Inserting a hole into a file

I have always been wondering why these types of operations are only appearing now, to me they seem very useful.
Video editing is a an obvious example, but there are also uses for plenty other applications. E.g. in scientific environments the need to put something into the middle of a large files comes up often.
Many commonly used data formats allow for padding between logical sections so that block alignment requirement is not a big problem.

What I'd still love now is a way to dump one file into a newly created hole in another file without needing any copies. Though I assume this is probably tricky as it would depend on capabilities of the filesystem, e.g. whether it can do copy-on-write or or not.

very useful

Posted Feb 4, 2015 13:14 UTC (Wed) by nye (subscriber, #51576) [Link] (2 responses)

>I have always been wondering why these types of operations are only appearing now, to me they seem very useful.
>Video editing is a an obvious example, but there are also uses for plenty other applications. E.g. in scientific environments the need to put something into the middle of a large files comes up often.

I feel like this implementation is unlikely to gain much traction in any but the most specialised of environments, because the abstraction leaks too badly.

The requirement for an application to understand things like the block size of the underlying filesystem seems like a textbook example of unwarranted chumminess with the implementation, and I'm honestly surprised the feature hasn't seen more pushback, especially with the experience of things like fanotify where a feature was added with the appearance of being conceptually general-purpose but actually specific shortcomings that limit it to a handful of use cases.

very useful

Posted Feb 5, 2015 18:31 UTC (Thu) by flussence (guest, #85566) [Link] (1 responses)

I can already see one immediate use for this: FLAC currently zero-fills a large (about 8KiB) chunk at the start of each file "just in case" you want to add more metadata without reallocating the entire file (which may be hundreds of megabytes). It's large enough to be wasteful for small text edits and too small to be useful for album art; with the filesystem feature it wouldn't need to waste the space at all.

very useful

Posted Feb 9, 2015 11:09 UTC (Mon) by nye (subscriber, #51576) [Link]

>with the filesystem feature it wouldn't need to waste the space at all

But FLAC can't know at creation time that the metadata will only ever be edited on a filesystem/OS combo that supports this feature, so it's unlikely to make the assumption that the pre-allocation is not needed.

This is a part of what it means to be too specialised: entirely non-portable, relying on assumptions that can be broken by moving a file even within one system. That doesn't necessarily have to be a bad thing, and it's not that I think this is a *bad* feature, just that in practice it will have to be used transparently and opportunistically as an internal operation, and probably by applications that either don't produce an output file, or already have a final 'rendering' pass or similar, where they can take the opportunity to defragment the result while they're at it.

If it didn't rely on the ability to safely leave small holes in the final files, it would be a lot easier for programs to use the API opportunistically where supported, but since it's not transparent I believe those opportunities will be few and far between.