Nastiness explained (sort of)

Actually, on UP we see everything in order, regardless of preemption.
What happens is that in setlk we update the per-inode list of file_lock 
(making it non-empty), then check that descriptor still points to the
same file.  In close we remove file from descriptor table, then check
per-inode list.  The trouble is, we have no warranty that these will be
seen in order on another CPU.  IOW, check of per-inode list might see
the value prior to update done by setlk and setlk might see descriptor
table state prior to update done by close.  With both sides deciding
that there's no need to evict file_lock - close() doesn't see it, setlk
thinks that when close() comes it'll evict the sucker.

And file_lock holds a reference to struct file, but doesn't contribute
to refcount, with all joy that comes from _that_.  It's not supposed
to outlive removal of struct file from descriptor table, let alone
struct file itself...

lock/unlock around accesses to descriptor table forces SMP ordering
and solves the problem; modification of file_lock list is guaranteed
to be seen before unlock done after the check in setlk and check in
close is guranteed to come after lock done before removal from descriptor
table.  So that spinlock acts as a barrier, preventing the ordering
problems.

See Documentation/memory-barriers.txt for background in that area.
And yes, that's part of the price to be paid for lockless access
schemes (such as RCU for descriptor tables and lazy taking of BKL
for POSIX locks eviction in filp_close()) - one needs to think of
things that normally would be provided automatically by exclusion
primitives.

Real SMP-only races are not too frequent thing around VFS, but yes,
sometimes they do happen...