Your editor still remembers installing his first Ethernet adapter. Through
the expenditure of massive engineering resources, DEC was able to squeeze
this device onto a linked pair of UNIBUS boards - the better part of a
square meter of board space in total - so that a VAX system could be put
onto a modern network. Supporting 10Mb/sec was a bit of a challenge in
those days. In the intervening years, leading-edge network adaptors have
sped up to 10Gb/sec - a full three orders of magnitude. Supporting them is
a challenge, though for different reasons. At the 2009 Japan
Linux Symposium, Herbert Xu discussed those challenges and how Linux has
evolved to meet them.
Part of the problem is that 10G Ethernet is still Ethernet underneath.
There is value in that; it minimizes the changes required in other parts of
the system. But it's an old technology which brings some heavy baggage
with it, with the heaviest bag of all being the 1500-byte maximum transfer
unit (MTU) limit. With packet size capped at 1500 bytes, a 10G network
link running at full speed will be transferring over 800,000 packets per
second. Again, that's an increase of three orders of magnitude from the
10Mb days, but CPUs have not kept pace. So the amount of CPU time
available to process a single Ethernet packet is less than it was in the
early days. Needless to say, that is putting some pressure on the
networking subsystem; the amount of CPU time required to process each
packet must be squeezed wherever possible.
(Some may quibble that, while individual CPU speeds have not kept pace, the
number of cores has grown to make up the difference. That is true, but the
focus of Herbert's talk was single-CPU performance for a couple of reasons:
any performance work must benefit uniprocessor systems, and distributing a
single adapter's work across multiple CPUs has its own challenges.)
Given the importance of per-packet overhead, one might well ask whether it
makes sense to raise the MTU. That can be done; the "jumbo frames"
mechanism can handle packets up to 9KB in size. The problem, according to
Herbert, is that "the Internet happened." Most connections of interest go
across the Internet, and those are all bound by the lowest MTU in the
entire path. Sometimes that MTU is even less than 1500 bytes.
Protocol-based mechanisms for finding out what that MTU is exist, but they
don't work well on the Internet; in particular, a lot of firewall setups
break it. So, while jumbo frames might work well for local networks, the
sad fact is that we're stuck with 1500 bytes on the wider Internet.
If we can't use a larger MTU, we can go for the next-best thing: pretend
that we're using a larger MTU. For a few years now Linux has supported
network adapters which perform "TCP segmentation offload," or TSO. With a
TSO-capable adapter, the kernel can prepare much larger packets (64KB, say)
for outgoing data; the adapter will then re-segment the data into smaller
packets as the data hits the wire. That cuts the kernel's per-packet
overhead by a factor of 40. TSO is well supported in Linux; for systems
which are engaged mainly in the sending of data, it's sufficient to make
10GB work at full speed.
The kernel actually has a generic segmentation offload mechanism (called
GSO) which is not limited to TCP. It turns out that performance improves
even if the feature is emulated in the driver. But GSO only works for data
transmission, not reception. That limitation is entirely fine for broad
classes of users; sites providing content to the net, for example, send far
more data than they receive. But other sites have different workloads,
and, for them, packet reception overhead is just as important as
Solutions on the receive side have been a little slower in coming, and not
just because the first users were more interested in transmission
performance. Optimizing the receive side is harder because packet
reception is, in general, harder. When it is transmitting data, the kernel
is in complete control and able to throttle sending processes if
necessary. But incoming packets are entirely asynchronous events, under
somebody else's control, and the kernel just has to cope with what it gets.
Still, a solution has emerged in the form of "large receive offload" (LRO),
which takes a very similar approach: incoming packets are merged at
reception time so that the operating system sees far fewer of them. This
merging can be done either in the driver or in the hardware; even LRO
emulation in the driver has performance benefits. LRO is widely supported
by 10G drivers under Linux.
But LRO is a bit of a flawed solution, according to Herbert; the real
problem is that it "merges everything in sight." This transformation is
lossy; if there are important differences between the headers in incoming
packets, those differences will be lost. And that breaks things. If a
system is serving as a router, it really should not be changing the headers
on packets as they pass through. LRO can totally break satellite-based
connections, where some very strange header tricks are done by providers to
make the whole thing work. And bridging breaks, which is a serious
problem: most virtualization setups use a virtual network bridge between
the host and its clients. One might simply avoid using LRO in such
situations, but these also tend to be the workloads that one really wants
to optimize. Virtualized networking, in particular, is already slower; any
possible optimization in this area is much needed.
The solution is generic receive offload (GRO).
In GRO, the criteria for which packets can be merged is greatly
restricted; the MAC headers must be identical and only a few TCP or IP
headers can differ. In fact, the set of headers which can differ is
severely restricted: checksums are necessarily different, and the IP ID
field is allowed to increment. Even the TCP timestamps must be identical,
which is less of a restriction than it may seem; the timestamp is a
relatively low-resolution field, so it's not uncommon for lots of packets
to have the same timestamp. As a result of these restrictions, merged
packets can be resegmented losslessly; as an added benefit, the GSO code
can be used to perform resegmentation.
One other nice thing about GRO is that, unlike LRO, it is not limited to
The GRO code was merged for 2.6.29, and it is supported by a number of 10G
drivers. The conversion of drivers to GRO is quite simple. The biggest
problem, perhaps, is with new drivers which are written to use the LRO API
instead. To head this off, the LRO API may eventually be removed, once the
networking developers are convinced that GRO is fully functional with no
remaining performance regressions.
In response to questions, Herbert said that there has not been a lot of
effort toward using LRO in 1G drivers. In general, current CPUs can keep
up with a 1G data stream without too much trouble. There might be a
benefit, though, in embedded systems which typically have slower
processors. How does the kernel decide how long to wait for incoming
packets before merging them? It turns out that there is no real need for
any special waiting code: the NAPI API already has the driver polling for
new packets occasionally and processing them in batches. GRO can simply be
performed at NAPI poll time.
The next step may be toward "generic flow-based merging"; it may also be
possible to start merging unrelated packets headed to the same destination
to make larger routing units. UDP merging is on the list of things to do.
There may even be a benefit in merging TCP ACK packets. Those packets are
small, but there are a lot of them - typically one for every two data
packets going the other direction. This technology may go in surprising
directions, but one thing is clear: the networking developers are not short
of ideas for enabling Linux to keep up with ever-faster hardware.
to post comments)