Hmm, I didn't realize that old versions of i386 didn't implement cmpxchg. Not sure there's many of these processors out there that will be using newer kernels that will need these features, but none-the-less, it should be fixed.
I actually like your solution. The assignment is still needed, but only for the first instance of the NMI, but the label could be moved.
It's great that you brought this up on LWN, but it would be much better if you posted a patch :-)