Doubtful. The key setup would big the biggest bottleneck.
Siphash was designed for this exact scenario. Dan Bernstein isn't known for his lack of performance insights. The paper has gobs of analysis of performance characteristics.
Using a CRC as a hash is just stupid. And I say that as someone who learned that the hard way. People only used it because of the lack of a decent alternative--newer hashes like murmur, etc, didn't have enough proven guarantees to inspire confidence. Siphash was designed to be _the_ go to hash for data structures, with both decent performance and a rigorous security analysis.