Article on TOE

Hi Jonathan,

We found your article on "Linux and TCP Offload Engines" very
interesting. The article discussed the submitted Chelsio TOE patch and
compiled a list of the objections raised by the stack maintainers. We
hope to be given the opportunity to provide some information regarding
the patch, and to clarify some of the points made.
 
As you have noted, the patch itself is really minimal. All in all, a
dozen or so lines of actual code will be needed for 2.6.14 to provide
generic, vendor-independent support for TOE. In any case, we have
resources committed to handling any future maintenance work. Therefore,
this should prove of very little impact on the maintenance of the stack.
 
The maintainers' apprehension regarding TOE in the Linux stack is well
known and shows up in the list of objections. Before we answer these
objections listed in last week's article, it is important to stress the
following points:
 
1) In addition to full offload, a TOE provides all the functions of a
regular NIC, including checksum offload and LSO for non-offloaded
traffic. A TOE can be operated as a NIC without any changes.
 
2) Today, you can buy a 10 Gbps TOE at virtually no price premium
compared to a 10 Gbps NIC. You're basically getting the additional
features for free.
 
3) Adding TOE support in the stack does not bypass the software stack.
It only gives the possibility to enable additional functionality if need
be. TOE is a performance enhancement which should be available to users
who need it.
 
Now, to the objections:
 
* The maintenance issue has been mentioned above, and looking at the
patch itself should address any concerns in that area. Questions,
comments or suggestion regarding it are more than welcome and
appreciated. If there is anything that can be done to further improve
this aspect let us know.
 
* Netfilter support is really not shorted out, and connection acceptance
can still be subjected to regular checking. Also, keep in mind that a
TOE is there to speed up some connections which require it, the rest of
the traffic is still fully processed in the software stack.
 
* Traffic rate control at 10 Gbps speeds is really not practical in
software today. Without arguing if and when that would be possible,
today the Chelsio TOE provide rate control in hardware, so no
functionality is lost in that regard. Clearly, this will depend on
different vendors' implementations, but this is all about choice.
 
* The security and patching issue is dependent on the vendor approaches
and their handling of flaws. However, given that a TOE can be disabled
at any time, one can fully rely on the software stack, while awaiting a
fix. There is no impact compared to regular NICs, besides the
performance loss.
 
* TOE performance has been questioned in the past, and perhaps rightly
so. However, it appears that this has changed recently. The Chelsio TOE
holds the Internet 2 Land Speed Record (7.5Gbps over 33,000Km), where
it maxed out the PCI-X bus and the distance required, with 1,500 byte
frames. This is just one indication, other independent tests by the Los
Alamos
Lab and OSU showed for example that TOE provides about twice the
throughput at half the CPU utilization of a regular NIC for data transfers,
and 60% to 1000% improvement in Web server capacity (see
http://www.chelsio.com/technology/HotInterconnect_2005.pdf). These
improvements were obtained without fully utilizing the TOE capability,
such as zero copy.
 
* It is clear that no one would want to design a 100Mbps TOE today, but
it is also a question whether anyone still has an original 100Mbps
adapter from 1993 in their current system. Technology advances will
obsolete everything we're building now, and in that regard the TOE is no
different from a regular NIC. Assuming you still have the 100Mbps TOE
you bought 10 years ago, you could just disable the offload and use it
as a NIC.
 
* It is important to stress that the TOE patent issue is being taken out
of context when it comes to full offload. The patents in question are
for the partial offload approach which has been taken by Microsoft. Full
offload is not, and cannot be patented as legal studies have determined.
 
* Stateless offload is an option which may work out for some
applications and users. However, the performance gap is still
considerable. Adding CPUs or waiting for CPUs to get faster are
suggestions which ignore the cost part of the equation. It is best to
leave such considerations to the users, who have to optimize their cost
performance measure.
 
* TOE opponents rely on the observation that CPU speeds tend to catch up
with network speeds, obviating the need for TOE. However, the very fact
that TOE is brought up recurrently and ever more pressingly indicates
that this gap is periodic, and it is getting more serious every time.
Today, the performance gap is being filled with exotic inter-connects,
such as InfiniBand, while TCP/IP over Ethernet lags in performance.
Dismissing this market as niche and insignificant would be ignoring the
market realities. As shown in recent studies, such as
http://www.chelsio.com/technology/Cluster_2005_Techical_R...,
a TOE makes TCP/IP over Ethernet again a competitive
technology.
 
It is important to mention that there are many unacknowledged benefits
to performing TCP processing in hardware, including microsecond
granularity retransmission and rate control, and receive data
re-assembly offload. These capability turn out to be very useful when
operating the latest low latency 10 Gbps Ethernet switches-on-a-chip,
which tend to have limited buffering resources and may consequently drop
packets. In addition, a TOE can handle essential TCP features, such as
timestamps, which are usually turned OFF due to their high processing
requirements at 10 Gbps. In addition, a TOE will most likely be required
to enable other technologies such as iSCSI, which is expected to gain
widespread use as a storage networking protocol.
 
TOE's performance has been independently demonstrated by end users, and
the technology can be integrated into Linux with relatively little
effort compared to other options being considered. There are no real
technical reasons for denying TCP offload its place as a useful option,
which users who require high performance should have today. It is our
hope that other reasons can be addressed to the satisfaction of
everyone, and the benefit of the users of TCP/IP over Ethernet