LWN: Comments on "Van Jacobson's network channels"

VERY interesting - but security implications to others?!?

efexis — 2006-02-11T09:33:37+00:00

If my memory serves me correctly, there are one or two servers out there on the internet that run linux. I know it's not very common, but some of them actually rent space/bandwidth/etc so that people can host their own websites, and they allow these users to run code (cgi scripts) or even have shell access via telnet/ssh.

You obviously don't see this practice taking off, but even so, I think if you told someone who offers shared hosting, not to bother protecting against ways unprivilidged users can cause havock, because somebody "can simple break into the datacenter with a boot CD or laptop"... you'd probably get an incredibly sarcastic response.

But no, you stick to talking about how to improve the performance of a 10Gig heavy loaded network interface --on a workstation-- where it'll really count.

Van Jacobson's network channels

burki99 — 2006-02-09T09:05:59+00:00

After reading this article I immediately though: Wow, maybe I should finally subscribe after years of reading LWN a week delayed. This is the kind of article that differentiates this publication from the hundreds of other publications that just grabb a quote from LKML (Linus says no to GPL3) and add no insight at all.

VERY interesting - but security implications to others?!?

jzbiciak — 2006-02-08T13:14:41+00:00

I wonder if you can still get most of the benefits of network channels if you limit their accessibility to special user IDs, and then require non-privileged applications to use cooperating threads--one privileged, one not--to send packets.

That way, the TCP/IP implementation can be stored away in a fixed implementation that root checks in on (and the kernel may even checksum at lauch time), but the processing still lives in userspace. It looks a little like the priv-sep that sshd uses.

Granted, with two cooperating threads, you get back to some of the context switching issues, but still it feels a little more flexible than keeping it in kernel space.

Interrupt Latencies?

csamuel — 2006-02-08T09:02:50+00:00

Erm, actually VJ was removing code from the interrupt context - he gave
the example of the e1000 driver going from ~700 lines of code executed in
an interrupt context down to ~400 lines.

Van Jacobson's network channels & 10 Gb/s ethernet

csamuel — 2006-02-08T08:57:44+00:00

Yes, this is about TCP, not just pushing datagrams out..

Van Jacobson's network channels

arafel — 2006-02-07T18:15:32+00:00

Well, y'know, I'm sure Ulrich's not got enough to do these days... ')

Exokernels

lutchann — 2006-02-05T01:47:51+00:00

On the other hand, most connection-oriented application protocols these days have a timeout mechanism above the TCP. If you suspend your IMAP or XMPP client for more than a minute or two you're likely to have a broken connection when you foreground the application again.

Exokernels

kbob — 2006-02-04T19:44:33+00:00

It would be unfortunate if TCP became incompatible with job control. I often suspend network jobs for various reasons, secure in the knowledge that the kernel will keep the TCP connection alive indefinitely.

Van Jacobson's network channels - how about network failovers?

caitlinbestler — 2006-02-03T19:00:53+00:00

Why wouldn't you be able to create a net channel
to a non-physical netdevice? You still have
removed the L4 processing from the kernel and
gotten the same reduction in context switching.

VERY interesting - but security implications to others?!?

caitlinbestler — 2006-02-03T18:57:50+00:00

The same filter rules that route inbound packets
can be used to validate outbound packets. You
simply do not accept packets from a channel if
the response packet would not be routed to
the matching channel.

So the privileged end of the channel can validate
that every packet on it is for a TCP connection
that is actually assigned to that channel.

Sneaky indeed!

xoddam — 2006-02-03T05:48:33+00:00

Ok, freely mapped memory doesn't cut it then. I wonder what the
performance impact of changing packet buffers' page permissions would be,
relative to copying (and relative to keeping the TCP implementation in
kernel space)?

protocol validity checks

zblaxell — 2006-02-03T04:15:47+00:00

kernel can check it out in place...while the user, maybe on another CPU, switches a few bits just after the kernel check but before the network card picks up the data.

Interrupt Latencies?

xoddam — 2006-02-03T00:34:14+00:00

> This way work is moved from kernel threads into userspace and
> into the interrupt handler.

Packet handling isn't currently done in kernel threads, it's done in
tasklets (OK, tasklets do run in a softirqd thread with the RT patch).
That's a great way to hog a CPU, and the channel implementation fixes the
problem by passing work to a thread, just as you suggest! In the slides
you'll see that the example channel 'producer' code wakes the listening
thread, so the 'consumer' is necessarily a task (but not necessarily a
userspace one).

An O(log n) search algorithm in the isr would indeed have high latency
with a very large number of channels to choose from -- but there is no
reason why the isr would have to select the final target amongst millions
of user sockets; channels are just as good for intermediate queueing
within the kernel as they are for delivery to userspace.

Van Jacobson's network channels

xoddam — 2006-02-02T23:56:18+00:00

> Wouldn't it lead to code duplication?

Yes. So does inlining :-)

Van Jacobson's network channels

iabervon — 2006-02-02T21:46:51+00:00

I don't see any reason that all of the channels would have to go to userspace. If a packet is to a kernel NFS client, it would end up in the kernel code, but without all the copies between the network and the VFS.

Of course, the kernel would have to keep a TCP implementation, but that's not surprising, since static binaries that use sockets should continue to work.

Van Jacobson's network channels

caitlinbestler — 2006-02-02T20:53:05+00:00

Connections can be channelized *after* they have
passed netfilter inspection.

Van Jacobson's network channels

jonabbey — 2006-02-02T20:12:50+00:00

Please, someone tell me they're not re-inventing STREAMS.

Interrupt Latencies?

simlo — 2006-02-02T13:04:32+00:00

This way work is moved from kernel threads into userspace and into the interrupt handler. The amount of work needed to be done in interrupt context is probably also dependent on the number of channels, i.e. number of open network sockets. Wouldn't we open the machine up to an effective DDOS attack: Spam the network with small packets. If the machine has a lot of network sockets open the each interrupt takes a long time to excecute and at some point there isn't any cpu left.

This issue is not so much problem if you run the handling in a thread (ksoftirqd) which will get lower priority as it starts to eat a lot of CPU. That way packets are dropped, but the rest of the system can run.

Van Jacobson's network channels

samj — 2006-02-02T12:29:41+00:00

If it doesn't cost anything, why not? You'd just plug netfilter in before the app and map packet buffers into its address space first. This can all be done in a separate security context too. Looks to me like it would mean a lot less in the way of protocol specific handling and would allow you to chain such tasks easily (eg netfilter->ipsec or netfilter->reverse proxy->web server etc.).

Exokernels

csamuel — 2006-02-02T11:55:05+00:00

VJ said that the only reason that TCP/IP was done in the kernel in Multics in the first place was because that was the only place you could be guaranteed not to get paged out for two minutes at a stretch.

Van Jacobson's network channels

NAR — 2006-02-02T09:57:15+00:00

Wouldn't it lead to code duplication? For example, a box doing NAT would need a (limited?) TCP/IP implementation in kernel space, while the host running e.g. an FTP client would need the full TCP/IP implementation in user space.

Bye,NAR

Van Jacobson's network channels - how about network failovers?

jiri.hlusi — 2006-02-02T07:11:57+00:00

The throughput gain numbers presented are certainly interesting. But does it affect e.g. network failovers (bonding) as well? There is loads of useful stuff [potentially] done in the kernel space, if one only sees the need for using it. Moving lots of the add-on functionality to the user space libraries might put own price tag to the complexity of those libraries as such.

-- Jiri

Van Jacobson's network channels

xoddam — 2006-02-02T05:45:08+00:00

The phased implementation described only moves packet processing to
userspace at the very last stage. At the 'ends' of the network this is
appropriate for efficiency. But even before that stage, channels are a
better way to pass packets around within the kernel. The task-oriented
interface (using wakeups instead of soft interrupts) would probably mean
netfilter no longer runs in tasklet context. We might instead see
several netfilter kernelspace daemons (like kswapd and friends), one for
each CPU.

HURD fast?

xoddam — 2006-02-02T05:36:51+00:00

Not with Linux evolving at this rate :-)

Van Jacobson's network channels & 10 Gb/s ethernet

bos — 2006-02-02T05:35:19+00:00

Plenty of Linux networking gear can drive 10Gbps hardware at line rate, and I'm not even talking about fancy TOE hardware in all cases.

Not having been at the talk, I don't know what circumstances he was talking about (perhaps specifically TCP at 10Gbps?), so I'm not picking a nit with his assertion, just pointing out that something along those lines can be done without scads of memory bandwidth.

That's not the only meaning of that statement

Ross — 2006-02-02T05:28:22+00:00

If you're only point is that security shouldn't depend on the network not being compromised I agree. However malicious users with unfettered physical access are not at all equivalent to malicious processes running under unpriviledged ids and that anything which makes them equivalent is decreasing security. Does it matter for well designed programs? No. But unfortunately tons of commonly used software is not well designed. If you can't trust IPs, port numbers, etc. many things break down. If you can't trust a program a user downloaded you should worry, but your network is not automatically compromised unless there is something which can be exploited on the system.

reinvention?

bcd — 2006-02-02T04:53:50+00:00

Not quite true: STREAMS had built-in facilities to help avoid data copies. It is a heavily layered model, and the queueing is not extremely efficient, but byte-for-byte copies are minimal in a properly configured STREAMS system.

These channels are a totally different concept, though, and address a different problem entirely.

VERY interesting - but security implications to others?!?

jamesh — 2006-02-02T04:11:33+00:00

There isn't much problem with this model on the receive end, assuming that the kernel correctly classifies packets and sends them to the right process.

As for sending, all that is necessary is a packet verifier, that makes sure the packet is appropriate for the given socket (which should be a lot simpler than a full TCP send implementation). If the packet shouldn't be getting sent from the socket, the kernel doesn't transmit it.

Exokernels

ttfkam — 2006-02-02T04:11:15+00:00

Reading about moving the network stack out to userspace made me think of the design of MIT's Exokernel OS.

Just as Ingo Molnar's work in migrating the scheduler to an O(1) algorithm was a taste of more extensive implementation of O(1) algorithms in the kernel, perhaps elements of the exokernel will find their day in various parts of Linux.

That's not the only meaning of that statement

elanthis — 2006-02-02T04:05:48+00:00

And my point remains... what is that unprivileged process going to do that you couldn't do by plugging in a laptop or some other device onto the network?

If you are implicitly trusting every packet sent by some 'trusted' host (which, if it were truly trusted, would never be running any malicious code anyhow), or trusting anything running on port 1024 down, you're not running a very secure network at all.

There is no security at the IP level at all. If you want trust and security, you have to put it all in higher layers.

Van Jacobson's network channels -- Microkernel?

atai — 2006-02-02T04:05:14+00:00

So the Hurd still has hope for speed?

Van Jacobson's network channels -- Microkernel?

jwb — 2006-02-02T02:13:12+00:00

In this case, the advantage is also that protocol processing is being done 1) in parallel, on an SMP machine; and 2) in the same cache space as the interested user process.

Van Jacobson's network channels -- Microkernel?

JoeBuck — 2006-02-01T20:48:38+00:00

It's not really so surprising. The cost being avoided in both cases is context switching. The default way of doing things is that the lower-level processing is in the kernel and the higher-level processing is in userspace. Either moving almost everything into the kernel, or moving almost everything out, reduces the overhead.

Van Jacobson's network channels -- Microkernel?

NAR — 2006-02-01T20:12:39+00:00

Isn't it ironic that khttpd/tux sped up web server perfomance by moving protocol processing into the kernel, but now Van Jacobson can speed up web server perfomance by moving protocol processing out of the kernel :-)

Bye,NAR

VERY interesting - but security implications to others?!?

NAR — 2006-02-01T20:08:44+00:00

Absolutely nothing stops a user from booting their workstation with a LiveCD that they have root access to.

Except the fact that this user can be an unauthorized one who've just cracked into the system from an other continent using the latest bug in a PHP BBS and his processes are running as 'nobody' user. He'd have hard time putting a live CD into the computer, but still we really don't want him to send arbitrary packets into the network.

Bye,NAR

Van Jacobson's network channels

NAR — 2006-02-01T20:00:21+00:00

I'm not sure I fully understand this, but it seems that these channels are used when there is a socket to the user space, i.e. an application running on the host sends/receives data to/from the network. But what about the case when there's no application? As far as I know, in routers the IP packets usually don't get to user space, but if protocol processing is moved to user space (netfilter), it might degrade performance, mightn't it?

Bye,NAR

reinvention?

vmole — 2006-02-01T16:39:59+00:00

Streams were about introducing lots of layers, each doing some sort of processing on the packets (or whatever), and each layer involving copying the data. VJ's channels are about getting the data into user space as quickly as possible, minimizing the actual processing and layering.

Users don't always WANT to attack

dwheeler — 2006-02-01T16:35:22+00:00

Sure, if a user is malicious, they can boot the system into some OS where they're fully privileged and attack. Or just unplug the network, plug in their own laptop where they have all privileges, and attack.

That's not the point. Not all users are malicious. Sometimes they run programs that APPEAR to do one thing, but do another. Sometimes systems run servers (like web servers) that an attacker can somehow subvert. In THOSE cases, I'd like the system to still limit what the attacker can do, INCLUDING limits on how the attacker can attack other systems.

Sure, all systems should be invulnerable to all attackers. But they aren't. Anyone who's managed a big network knows how hard it is to keep EVERYTHING secure, ESPECIALLY since there are some vendors who do not release patches for KNOWN vulnerabilities (names withheld, but Google can help you find them rather quickly). So you really need defense-in-depth: you need to try to make it so that attackers have to break down MULTIPLE barriers to get the goods.

Limiting the network-level actions of unprivileged accounts is not the be-all of security. But it's one of the few mechanisms we CURRENTLY have deployed widely that slow the spread of attacks across a network. Diseases that spread rapidly are often unstoppable, because you just don't have enough time to react. Slowing the spread of a disease is key to countering it. Similarly, in the network world, slowing down attack vectors is also key to countering it.

I'd like to see that packets from untrusted user apps are still FORCED to obey certain limits on what they can send. You don't need a system-wide lock to do that kind of checking; after a call to the kernel, the memory could be mapped out and checked WITHOUT harming the cache lines of other systems. For most systems it'd just involve checking a few bytes... nothing expensive, and certainly taking less time than sending something down any network port.

reinvention?

macc — 2006-02-01T15:51:16+00:00

I had the same feeling of similarity.

could you elaborate on the differences?

Van Jacobson's network channels -- Microkernel?

smoogen — 2006-02-01T15:41:37+00:00

From a 10,000 m view, this looks either like the days of DOS where every application had its own TCP stack :), or a better take on microkernels. The kernel sets up the basic stuff for the machin, and the channels for the smaller daemons that then handle things like iptables, the stack etc. Each of these would be a herd of daemons interconnecting between each other.

[I am sorry for the hurd joke.. it was a very long night and I found it funny.]