We are lucky enough to be able to work around the bug by bonding the interfaces together, tagging them and then passing both VLAN's down both interfaces so it no longer matters which interface packets are delivered on (plus we get some redundancy into the bargain). Others might not be so fortunate (and I've found at least one bugzilla entry [1] that indicates that a large US HPC lab has stumbled across the same issue that "upsets" their Lustre filesystem).
On the other hand our support contact at Red Hat seems to have trouble getting any info out of the RH kernel folks. We've determined ourselves that the fix is a newer version of the Mellanox driver and fed that back to them and I suspect that's what they are planning for RHEL 5.7. I hope.