Reconsidering network channels [LWN.net]

When Van Jacobson presented his network channels idea at linux.conf.au last January, he set a bit of a fire in the Linux networking community. By making some significant changes to the processing path for incoming packets, and by pushing most of the work as close as possible to the destination application, Van was able to achieve significant performance improvements - eliminating as much as 80% of the processing overhead on multiprocessor systems. With numbers like that, it seemed like the question of whether Linux would incorporate channels need not even be asked.

Since then, however, reality has begun to make itself felt - something which reality is wont to do, sooner or later. Which is why David Miller's latest pronouncement on network channels reads like this:

Don't get too excited about VJ netchannels, more and more roadblocks to their practicality are being found every day.... All the original costs of route, netfilter, TCP socket lookup all reappear as we make VJ netchannels fit all the rules of real practical systems, eliminating their gains entirely.

The issue at hand had to do with the integration of channels and netfilter. The hope had been that packets could be identified and sorted into their respective channels before the netfilter (firewall) processing was done. Then said processing could be performed close to the application, on the same processor. It turns out, however, that netfilter can change the real destination of the packet. So packets must be filtered before entering a channel, and much of the performance benefit of using a channel is lost.

Alexey Kuznetsov has posted a detailed criticism of channels, asserting that most of the claimed benefits are illusory. Says Alexey:

It is an amazing toy. But I see nothing, which could promote its status to practical. Exokernels used to do this thing for ages, and all the performance gains are compensated by overcomplicated classification engine, which has to remain in kernel and essentially to do the same work which routing/firewalling/socket hash tables do.

Finally, it seems that many of the benefits of channels can be had by carefully taking advantage of the capabilities of modern hardware. In particular, an increasing number of devices can perform simple packet classification and (via targeted interrupts) direct packets to the CPU where the destination application is running. That technique will get rid of the cache misses caused by performing interrupt processing on one processor and protocol processing on another.

In the end, it appears that yet another seemingly bright scheme may not make the transition into real-world deployments. Some of its core concepts, such as using cache-friendly data structures and trying (even harder) to improve cache locality, will likely influence the future direction of the network stack, however. So, while there may not be a revolutionary new mechanism in the network stack's future, some of the promised performance improvements should eventually be realized anyway. And, as David says, "At least, there is less code to write."

(Log in to post comments)

Reconsidering network channels

Posted Jul 27, 2006 13:48 UTC (Thu) by cventers (guest, #31465) [Link]

If someone's prepared to offer up a well-tested alternative to netfilter
that supports all netfilter does (and maybe more), and that will work in
a new netchannel framework, then perhaps one barrier will be removed.

But netfilter is a _long_ way from obscure; particularly, think of all of
the SOHO routers out there and you end up counting a _lot_ of netfilter
users.

But offer a "fast path?"

Posted Jul 27, 2006 17:32 UTC (Thu) by AnswerGuy (guest, #1256) [Link]

I think the point to which the original poster here may have been referring is that allowing netfilter design considerations to dictate the availability
of some optional "fast path" code might not be the best strategy.

Many systems are dedicated to very narrow purposes (and put into an
infrastructure which guarantees that only certain classes of packets will
reach them. (Think of systems arrayed behind a load-balancer).

In those cases it might be appropriate to have a netchannels option to offer the fastest processing of that traffic. Essentially the "classifier" has been scaled out to a different system entirely (the load balancer).

Reconsidering network channels

Posted Jul 28, 2006 18:39 UTC (Fri) by PlaguedByPenguins (subscriber, #3577) [Link]

some people already want a total absence of anything like netfilter.
by this I don't just mean being able to turn it off in .config, but also that netfilter requirements do not impact upon the structure of the driver and hence the speed of the NIC.

as people in netdev mentioned (more in the RDMA threads than netchannels), nobody in their right mind would ever dream of using netfilter on their sub 10microsecond cluster interconnects.
so users are already split into two camps - those who want to use fast hardware, and those with gigE and slower who might want to do routing and netfiltering. as low latency hardware like Infiniband and Myrinet goes more mainstream this split will become more evident.

if netchannels can only be architected so that it's usefully fast when netfilter isn't required then that's more than fine for the whole class of users who already turn off netfilter. it can go into the kernel so that netchannels only appears when netfilter is off, and people who care about performance (like me) would probably use it.
anybody who turns off netfilter in the .config already knows what they are doing and what they are losing when they do it.

as netfilter already can be turned off it follows that you should be able to write drivers and infrastructure that only works when netfilter is off. I don't see a problem with this... ???

cheers,
robin