I'm not sure if linux's tcp stack is inefficient or not compared to other tcp stacks, but the networking stack is certainly is complex and multi-layered. Consider all the basic tcp protocol code (reliability, packet frag and reassembly, etc), then layer on top netfilter, underneath it routing logic, the ip stack, and etc, and it's easy to construct packets that go through possibly significant code paths.
Certainly all that complexity can't be great for performance.
It's the argument I make for fibre channel v. iscsi. It's true that iscsi hardware (being just standard networking stuff) is a lot cheaper and does the job 90-95% of the time. But in the edge case, especially w.r.t latency, fibre still wins, largely because it's simple in comparison.