Two new ways to read a file quickly

Posted Mar 6, 2020 22:58 UTC (Fri) by NYKevin (subscriber, #129325)
In reply to: Two new ways to read a file quickly by nkiesel
Parent article: Two new ways to read a file quickly

You can emulate that in userspace: just allocate a buffer one byte longer and pass bufSize+1 in the first place. You even get to read the first "extra" byte for free!

Two new ways to read a file quickly

Posted Mar 7, 2020 10:58 UTC (Sat) by intgr (subscriber, #39733) [Link] (2 responses)

Although I understand how people might consider non-power-of-two sized buffers ugly. The extra 1 byte in an allocation of 8193 bytes would have an alignment overhead of 15 bytes.

In some cases malloc() will also call mmap() to get whole-page allocations, and that would cause a whole 4k page to be wasted for 1 byte.

Two new ways to read a file quickly

Posted Mar 7, 2020 18:06 UTC (Sat) by zlynx (guest, #2285) [Link] (1 responses)

Power of two sized buffers turn out to be a memory wasting trap though.

There was a blog article I read a while ago, linked off Hacker News I think, talking about some research into memory usage this programmer had done. It turned out that many allocations for buffers of sizes such as 4,096 bytes ended up having 16 bytes or so added to it. This was for memory allocation tracking, or if passed into some API the library would put it into a structure with other variables.

If I remember correctly the author determined that allocating 4,000 bytes was a nice round number that tended to round up to a 4,096 page size much more reliably.

Two new ways to read a file quickly

Posted Mar 19, 2020 12:43 UTC (Thu) by mgedmin (subscriber, #34497) [Link]

Was it https://blog.mozilla.org/nnethercote/2011/08/05/clownshoe... That was a great article from a Firefox developer working on memory optimization.

Two new ways to read a file quickly

Posted Mar 8, 2020 16:29 UTC (Sun) by nkiesel (guest, #11748) [Link]

Nope, because if the result is bufSize+1 the caller again does not know if the file fit perfectly into the buffer or if there are more bytes to read.

Two new ways to read a file quickly

Posted Mar 10, 2020 16:27 UTC (Tue) by ThomasBellman (guest, #67902) [Link]

Even better, take a cue from snprintf() and return the number of bytes that would have been read if the buffer was infinitely large (i.e, return the file size). And in case the kernel can't quickly determine the size of the file, return either MAXINT or bufSize+1.