LWN: Comments on "JLS2009: Generic receive offload"

JLS2009: Generic receive offload

samplelin — Wed, 21 Apr 2021 12:58:01 +0000

Q: If GRO will create the packet latency in RX path? How to sure that the latency is not impacted? Depended on user?

JLS2009: Generic receive offload

ollibolli — Fri, 06 Nov 2009 19:33:12 +0000

One 1500 byte packet: 64 bits preamble + 14*8 bits Ethernet header + 1500*8 bits payload + 32 bits FCS + 96 bits interframe gap = 12304 bits
10Gbit/s = 10000000000bits/s
10000000000bits/s / 12304bits/pkt = 812743 pkt/s
Data bits might not translate one-to-one into bits on the media so the actual packet rate might be slightly smaller.

JLS2009: Generic receive offload

RamiRosen — Thu, 05 Nov 2009 08:13:53 +0000

>With packet size capped at 1500 bytes, a 10G network link running at full >speed will be transferring over 800,000 packets per second.

Is it accurate ?

a simple calculation :
1000000000/1500=
666,666

Rami Rosen

UDP merging vs VoIP

intgr — Sun, 01 Nov 2009 16:58:21 +0000

From the article:
> If a system is serving as a router, it really should not be changing the
> headers on packets as they pass through.

Also, VOIP uses UDP simply because TCP would add additional delay when it attempts to retransmit packets that already missed their deadline.

But VOIP is still affected by packet loss. A UDP packet that missed its deadline is never way any worse than a UDP packet that got dropped.

UDP merging vs VoIP

NAR — Sun, 01 Nov 2009 11:00:43 +0000

I think VoIP doesn't mind if a couple of packets is lost (that's why it uses UDP in the first place), the jitter is more problematic. Buffering doesn't really help, because in a phone conversation one might want to interrupt what the other says and seconds of buffering kind of breaks this.

I'm not quite sure where this merging could be done. Only at the endpoints or in routers between?

UDP merging vs VoIP

intgr — Sun, 01 Nov 2009 01:02:45 +0000

> to wait for UDP packets to be merged

The article partly answers your question:
> How does the kernel decide how long to wait for incoming packets before
> merging them? It turns out that there is no real need for any special
> waiting code: the NAPI API already has the driver polling for new packets
> occasionally and processing them in batches. GRO can simply be performed
> at NAPI poll time.

In a low-throughput setting, the kernel uses the normal interrupt-driven networking mode. Individual packets are processed as quickly as they come in over the wire.

Only when the CPU is too pegged to keep up with the interrupt load, does NAPI revert to the polling mode. At this point, without NAPI, the CPU would already be thrashing -- spending time to receive packets that it has no time to process. GRO merely increases the throughput that can be handled in polling mode.

How many packets are grabbed in each polling cycle can be changed with /proc/sys/net/core/netdev_budget (the default 300 is quite modest IMHO)

JLS2009: Generic receive offload

intgr — Sun, 01 Nov 2009 00:22:25 +0000

From the article:
> sending of data, it's sufficient to make 10GB work at full speed

Correction, I think that should be "10Gb". Even current memory controllers have a hard time coping with 10 Gbytes/s. :)

UDP merging vs VoIP

NAR — Sat, 31 Oct 2009 18:31:15 +0000

Is this a good idea to wait for UDP packets to be merged if these packets contain voice data? It could create noticable jitter...

How creaky is Ethernet the spec?

nix — Fri, 30 Oct 2009 20:59:10 +0000

There are lots of 10G cards that also do 1G, but I don't think you'll find
any device drivers for 1G-only cards that work with the quite different
hardware that does 10G :))

Fundamentally though most of the hardware (as in the physical components
on the 10G board) that does 10G is also used to do 1G; and most of the
hardware that does 1G is also used to do 100M. The protocols could be
considered distinct, but they're so similar at heart that giving them the
same name seems sensible to me.

JLS2009: Generic receive offload

sgros — Fri, 30 Oct 2009 16:55:46 +0000

In that case we are speaking of some content provider that has >=10Gb/s link(s) (heavy loaded) that uses linux as:

a) router, or
b) application server (or something similar)

in case of a), no hardware offloading is allowed. In case of b) I somehow doubt that you'll put extreme load on your server(s), i.e. you'll do load balancing.

Of course, there could be some other use scenario I'm not aware of, but the point is that I'm still not convinced. BTW, it doesn't mean I'm against achieving something like this that Herbert is trying to do. If nothing else, it would be definitely cool to say linux can do it, others can't. :)

How creaky is Ethernet the spec?

giraffedata — Fri, 30 Oct 2009 15:47:56 +0000

There appears to be interoperability that justifies calling the new protocol Ethernet. You can replace one link of an existing Ethernet network with a 10G one and it all still works -- and not by simply negotiating down to the old protocol.

My understanding is that is what makes the 1500 byte MTU a constraint on 10GE.

Also: Can you replace a 1G ethernet card in a computer with a 10G and now talk at full speed (to a 10G peer) without replacing the device driver or anything else in the computer?

How creaky is Ethernet the spec?

smoogen — Fri, 30 Oct 2009 02:46:14 +0000

I was told that if you sent a flood of ICMP traffic to certain 1GB switches they would 'fail' into a hub-like mode.

The collision detection is still needed because they are still possible on a pure switch.. usually they happen with certain traffic patterns or if the network gets saturated. However they occur a LOT less than they did on hubs.

JLS2009: Generic receive offload

Shewmaker — Thu, 29 Oct 2009 22:53:42 +0000

Jesper's slides from LinuxCon2009.

JLS2009: Generic receive offload

bangert — Thu, 29 Oct 2009 21:22:47 +0000

At OpenSourceDays last weekend here in Copenhagen Jacob Brouer presented his work on
getting 10 Gbit routing going on Linux. His point of departure was a full-duplex 2 port solution.

The result was that routing 40Gbit/s with a packetsize of 1500 is doable. Reducing the average
packet size to the internet average the performance would drop off obviously, but things are
getting quite close.

As far as i understood, his company is looking at this ~6000 dollar setup as a alternative to 60000
dollar equipment.

http://opensourcedays.org/CommunityDay2009/print/133
Slides and video forthcoming.

JLS2009: Generic receive offload

iabervon — Thu, 29 Oct 2009 20:20:15 +0000

The issue is that you'd actually like to receive 1Mb/s each from 10000 different points across the public internet all past the same router. You want to speak at 10Gb/s, and you're limited to the MTUs on each of the remote networks.

How creaky is Ethernet the spec?

bronson — Thu, 29 Oct 2009 20:03:25 +0000

From my perspective, it was the star topology that brought Ethernet's big win.

I remember trying to wrestle thick cable around corners without causing network issues and fighting endlessly with vampire taps (and SCSI terminators, but that's unrelated). Token ring was definitely better than this mess.

Then thin cable brought Ethernet competitive with token ring. Now the biggest problem with both systems was employees rolling their chairs over the cable under their desks and bringing the entire network down. And the occasional NIC that loses its mind and goes into a retransmit flood. And sometimes terminators. But, overall, it was OK.

Then 10baseT and cheap switches came along and allowed Ethernet to use an inexpensive star topology. This is what absoulutely murdered token ring. Twisted-pair star setups were so much faster to set up and more reliable to keep working that nobody looked back.

Funny how admins at the time would look at the neck-thick bundle of wires slamming into 80-port switches in the server closet and think, "Good lord, I'd never want to try to keep all that mess working." It didn't take them much time adminning a twisted pair ethernet network for them to grow to love it though.

Fun times.

How creaky is Ethernet the spec?

mebrown — Thu, 29 Oct 2009 19:41:17 +0000

"All 1Gb and above Ethernet is full duplex, point to point"

Actually not true. When 1Gb Ethernet first came out, you could buy actual hubs which had actual collisions and the wonderful-ness (or lack thereof) that entailed (meaning half-duplex-only). They were very quickly obsolete when almost all vendors switched to using switches exclusively and I dont think you could actually buy a hub anymore.

JLS2009: Generic receive offload

sgros — Thu, 29 Oct 2009 17:25:24 +0000

Herbert says that jumbograms are problem on the Internet, but this is not a rock solid argument. How many access points allow for 10Gb/s, and more importantly, how many of them communicate at full speed? Namely, if you have 10Gb/s link between two points that goes over the Internet, then I think that you also have control over what is between those two points.

How creaky is Ethernet the spec?

Shewmaker — Thu, 29 Oct 2009 16:50:48 +0000

Collisions are not an issue ... data is not lost immediately, but contention at a transmit port of a switch is a problem. When that happens, one packet gets transmitted while the other is placed on a queue. That queue may be in memory shared by all ports, a subset of ports (a blade in the switch, for instance), and some switches allow you to set aside dedicated memory for a port's queue.

I've got a simple illustration in this presentation.

Packet loss occurs when the queue overflows, and TCP uses that as a signal to slow down. It has worked remarkably well over the years, but switches have been making it work partly be increasing the amount of memory for the queues. Large queues cause large variations in delay and mean that TCP will have increasingly more packets in flight before it knows that it needs to slow down.

Infiniband has a reliable (connected) mode that allows packets with an MTU of 64k, which is quite nice for high performance storage networks and parallel file systems that are IP based.

There are new Ethernet standards coming out that have equivalents to Infiniband's features, but I don't know how much longer until products are out. Data Center Ethernet and Converged Enhanced Ethernet

Incidentally, my research is focused on providing performance guarantees even on current inexpensive Ethernet hardware (LAN, not WAN). I expect it will also be beneficial on the reliable (i.e. lossless) network technologies as it allows arbitrary guarantees (unfairness) that are more flexible than priority based QoS schemes.

How creaky is Ethernet the spec?

SEJeff — Thu, 29 Oct 2009 14:01:09 +0000

Doesn't make sense to call it CSMACD if there won't ever be collisions at the
very least.

How creaky is Ethernet the spec?

epa — Thu, 29 Oct 2009 11:08:28 +0000

All 1Gb and above Ethernet is full duplex, point to point, no hubs (only switches) and therefore collisions are not an issue.

So it doesn't really make sense to call it Ethernet any more; mostly just a marketing name.

How creaky is Ethernet the spec?

felixfix — Thu, 29 Oct 2009 04:57:04 +0000

Makes sense ... hadn't thought of it that way. If there are no more collisions, is there any reason to keep collision detection in the spec? I seem to recall the packet preamble had to be long enough to cover the collision detection. Would it add anything to get rid of that?

It's funny how the old cable strung from one node to the next has stuck in my mind even tho I haven't seen that kind of install in ages.

How creaky is Ethernet the spec?

agrover — Thu, 29 Oct 2009 04:11:37 +0000

All 1Gb and above Ethernet is full duplex, point to point, no hubs (only switches) and therefore collisions are not an issue. 10Gbps networks can hit wire speed, taking header overhead and interframe gap into account. Like the article says, the big problem really is that the Ethernet packet has not scaled with the enormous throughput increase.

How creaky is Ethernet the spec?

felixfix — Thu, 29 Oct 2009 03:58:57 +0000

Back when Token Ring and Ethernet were battling it out, I was amazed at the sloppiness of the Ethernet protocol with collision detection at its heart. It seemed so clunky, and some reports said Ethernet collisions with even a few nodes slowed it down to Token RIng's 4Mbps. I never quite understood why Token Ring lost out ... but I didn't envision the big networks that came down the pike where Token Ring would have been lost. Switches made a huge difference compared to hubs, too.

Have there been any attempts to replace Ethernet that have actually gone anywhere? I suppose if there had been, I wouldn't be asking. But the idea of looking for collisions on so many dinky packets seems even clunkier to me now. What kind of real thruput do 10Gbps networks actually achieve today?