Adaptive file readahead

Adaptive file readahead

There are two reasons for readahead. The article mentions one: you can speed up an application thread by doing the I/O in a parallel thread, while the application thread is munching on what it previously read.

For this purpose, you want to set the readahead window size exactly to the application burst size -- the amount of reading the thread does in each read/process cycle.

This effect is irrelevant if the application doesn't have anything else to do or never waits for reads (a multi-threaded application).

But just as important is the fact that a block device has greater capacity when given more stuff at a time to read. It can schedule disk and head motion better and combine contiguous reads to eliminate some overhead.

This effect drops off as the sizes go up. And it's irrelevant if you aren't driving the disk drive at capacity.

An ideal adaptive readahead system would use feedback to see when increasing the window size stops increasing throughput. That handles both the readahead effects. You also stop when the memory manager says you're using too much memory (which is the inactive list thing mentioned in the article). With something that automatic, there would be no need for that arbitrary 1M readahead limit.