If Mach was the failure point, why?

Your mentioning that gave me an impulse to read over the channels stuff again, and it is pretty neat stuff, even if it seems to have hit a few snags (any idea what happened to it? I presume that at least some of the ideas got taken up). I must admit that what I had in mind was just the idle thoughts of someone not particularly knowledgeable about networking, and I certainly hadn't given thought to the details of CPU core cache and things.

Just for the fun of it I will try to give those thoughts a shape though.

* an in-kernel network device driver (probably with a /dev entry) which can accept data to transmit and return data, as naked ethernet frames, or raw TCP data, or whatever the card can do in hardware.
* a userspace server to multiplex the driver which can accept socket connections or whatever from clients and can accept and return data as ethernet frames, ip packages or whatever, but can tell the clients what it can handle most efficiently. It will do whatever checks are needed on outbound data to be sure that the client is allowed to send what it is sending, and will do whatever checks are needed on the inbound data to know where to send it. That is likely to put an end to keeping things on a single core unless the checks can be offloaded to the hardware.
* hopefully some optimisation of the kernels socket primitives so that as much as possible of the data copying accross sockets can be collapsed into single operations, preferable DMA between card and the final client. Should be doable for e.g. outbound TCP data with clever cards, perhaps even for inbound data if the card can buffer it for a short time and give the server just enough information to know where to forward it to.

I'm sure that there are millions of holes in that logic, just for a start what happens when in-kernel clients need to send network data (if someone pushing a network stack into userspace really cares about in-kernel clients). Feel free to poke a few more if you like...