Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 6, 2013 7:08 UTC (Thu) by dlang (guest, #313)
In reply to: Hypothetical progams might desire cross-CPU coalescing of IO by faramir
Parent article: The multiqueue block layer

but how many people are running such parallel processing of single files?

And of those who are doing so, how much do they care if their files get processed one at a time using every CPU for that one file, or many files at a time with each CPU processing a different file (and therefor not needing the combined I/O logic)

Yes, there are going to be some, but is it really worth crippling the more common cases to help this rare case?

Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 6, 2013 7:28 UTC (Thu) by dlang (guest, #313) [Link]

a comment I posted elsewhere is also relevant to this discussion. I'll post a link rather than reposting the longer comment

in https://lwn.net/Articles/553086/ I talk about how I/O coalescing should only be done when the I/O is busy (among other things)

Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 6, 2013 12:06 UTC (Thu) by ejr (subscriber, #51652) [Link] (3 responses)

See uses of HDF5 and NetCDF file formats. There are many software systems that store in a single, large file, emulating a file system more aligned with the higher-level semantics. Also, think of databases. Counting a case as rare v. common requires an application area.

But... Parallel HDF5, etc. interfaces handle some coalescing in the "middleware" layer. They've found that relying on the OS leads to, um, sufficient performance variation across different systems and configurations. Yet another standard parallel I/O layer in user-space could help more than trying to jam the solution into the OS.

But relying on user-space won't help the case multiqueue is attacking.

Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 7, 2013 2:27 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

when you have things like databases where many applications are accessing one file, how common is it for the different applications to be making adjacent writes at the same time?

It may happen, but it's not going to be the common case.

Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 7, 2013 10:06 UTC (Fri) by ejr (subscriber, #51652) [Link] (1 responses)

In my world (HPC), having multiple coordinated processes accessing slices of a range *is* the common case. We have to fight for anything else to have even reasonable performance support. See lustre.

But this case often is better handled outside the OS. There could be different interface where clients post their read buffers and the OS fills them in some hardware-optimal order, but that's been considered for >20 years and has no solution yet.

Hypothetical progams might desire cross-CPU coalescing of IO

Posted Jun 7, 2013 18:18 UTC (Fri) by dlang (guest, #313) [Link]

you are processing slices of a range, but are you really writing the adjacent slices at almost exactly the same time from multiple processes? that's what it would take to give the OS the chance to combine the output from the different processes into one write to disk.

Also, in HPC aren't you dealing with many different systems accessing your data over the network rather than multiple processes on one machine?

What we are talking about here is the chance that things running on different CPUs in a single system are generating disk writes that the OS could combine into a single write before sending it to the drives

for reads this isn't as big a deal, readahead should go a long way towards making the issue moot.