Preallocating requests

Posted Nov 10, 2017 16:02 UTC (Fri) by abatters (✭ supporter ✭, #6932)
Parent article: Block layer introduction part 2: the request layer

> Another important difference between the multi-queue layer and the single-queue layer is that, for multi-queue, the request structures are all preallocated.

I thought I would share my experiences with this. I was one of the early adopters of scsi-mq. I submitted a lot of small fixes around the days of 3.17 and 3.18 when scsi-mq first made its appearance. The preallocation of the request structures resulted in some unexpected behavioral changes for higher-level code - since the requests are preallocated to a limited fixed number, if you try to allocate more requests than the preallocated number, your attempt to allocate the request will either fail, or if requested, block waiting for a previous request to complete. With the old single-queue layer, you could allocate and submit many more requests without blocking.

Here is an example commit of where it matters:

7772855a996e ("sg: fix EWOULDBLOCK errors with scsi-mq")

The SCSI generic driver provides an interface that enables you to submit multiple SCSI commands in non-blocking mode and then wait for any of them to complete. But with scsi-mq enabled (after the commit above), the system call to submit a command might block to wait for a different command to complete so that it can allocate a request, so it is no longer always non-blocking. The old single-queue layer didn't have that problem. Not a complete fail, but not ideal either.