If you follow Andrew Morton's reasoning, "am I in atomic" is something a caller should never
ask. It should be told explicitly, as in the case of kmalloc().
The case where "in_atomic" is getting used "legitimately" is in a fault handler (which should
not be a fastpath pretty much by definition). It sounds like they're abusing preempt_count to
coax a particular behavior out of the fault handler, rather than just stating the intended
behavior directly. That doesn't necessarily sound like clean design to me, but rather an
overly clever hack.
I'm sure someone more familiar with this mechanism can explain why it is or is not a good
mechanism.