IO CPU affinity testing series

Hi,

Here's a new round of patches to play with io cpu affinity. It can,
as always, also be found in the block git repo. The branch name is
'io-cpu-affinity'.

The major change since last post is the abandonment of the kthread
approach. It was definitely slower then may 'add IPI to signal remote
block softirq' hack. So I decided to base this on the scalable
smp_call_function_single() that Nick posted. I tweaked it a bit to
make it more suitable for my use and also faster.

As for functionality, the only change is that I added a bio hint
that the submitter can use to ask for completion on the same CPU
that submitted the IO. Pass in BIO_CPU_AFFINE for that to occur.

Otherwise the modes are the same as last time:

- You can set a specific cpumask for queuing IO, and the block layer
  will move submitters to one of those CPUs.
- You can set a specific cpumask for completion of IO, in which case
  the block layer will move the completion to one of those CPUs.
- You can set rq_affinity mode, in which case IOs will always be
  completed on the CPU that submitted them.

Look in /sys/block/<dev>/queue/ for the three sysfs variables that
modify this behaviour.

I'd be interested in getting some testing done on this, to see if
it really helps the larger end of the scale. Dave, I know you
have a lot of experience in this area and would appreciate your
input and/or testing. I'm not sure if any of the above modes will
allow you to do what you need for eg XFS - if you want all meta data
IO completed on one (or a set of) CPU(s), then I can add a mode
that will allow you to play with that. Or if something else, give me
some input and we can take it from there!

Patches are against latest -git.

-- 
Jens Axboe