LWN: Comments on "The EFF launches a router project"

The EFF launches a router project

nysan — Mon, 15 Feb 2016 13:13:45 +0000

"My understanding of DPDK and ODP is that they are mostly interested in running proprietary network stacks in user space, in order to implement features there that don't exist in the kernel, but at the same time completely bypassing the kernel and the features that exist in it. The fact that Intel and some others can do DMA into caches is independent of that and you can do that in both kernel and user space based network stacks."

Have a look at openfastpath on ODP.
Linux is still used as controlplane with --enable-sp, ARP, RIPv2 et.c. is sent to the kernel stack.
Standard cmds "arp", "ip" et.c. are used to configure parts of the stack thats common between both.
Control plane stack state is mirrored via netlink for fast access by packet churning cores, i.e. ARP table, FIB et.c.

Possible alternative: Raspberry Pi + Modem

bronson — Sat, 02 Aug 2014 23:55:22 +0000

Production environments tend to be highly dynamic these days.

Possible alternative: Raspberry Pi + Modem

dlang — Fri, 01 Aug 2014 22:10:54 +0000

That is true, but the more you centralize, the more dangerous that control system becomes. With the virtual datacenter you have the same system configuring the switches as is configuring the servers.

As I say, it's nice for highly dynamic environments, but I question the value of it for production networks.

There's also the question of if the dynamically configured switches are going to create unexpected bottlenecks. not a problem on a small network, but a serious concern on a large one.

Possible alternative: Raspberry Pi + Modem

raven667 — Fri, 01 Aug 2014 20:47:06 +0000

That's in some way the same security problem of any network management plane, there is less fundamental difference between a JunOS or IOS supervisor and a PC running Linux than one might think, in fact most of the modern network gear runs on Linux or FreeBSD, sometimes on re-badged commodity servers, with just an old-timey CLI put on top. Whether you have an SDN controller or management server talking over a network devices or you have a chassis connected to a FEX it is surprisingly similar when you get down to the details. So protecting your management plane, which has traditionally been done with local packet filters/ACLs on the device could be extended into the new designs.

Possible alternative: Raspberry Pi + Modem

dlang — Fri, 01 Aug 2014 18:45:31 +0000

yep, all your security boils down to control over the management server, if you get hold of that you can configure anything to talk to anything else.

convenient if you have a network that changes a lot (like a lab), for a production network I'm not so sure.

Possible alternative: Raspberry Pi + Modem

kjp — Fri, 01 Aug 2014 14:54:30 +0000

Fascinating post. It seems like you're virtualizing the network, just like everything else is being virtualized...

The EFF launches a router project

arnd — Fri, 01 Aug 2014 12:27:18 +0000

My understanding of DPDK and ODP is that they are mostly interested in running proprietary network stacks in user space, in order to implement features there that don't exist in the kernel, but at the same time completely bypassing the kernel and the features that exist in it. The fact that Intel and some others can do DMA into caches is independent of that and you can do that in both kernel and user space based network stacks.

Note that ODP by design requires cache-coherent DMA, which does not necessarily imply doing the DMA into the cache, but it does mean that it is not portable to the typical low-end SoCs you'd find in consumer routers as opposed to the devices that Cisco and Juniper are selling to enterprise customers.

The ArmadaXP chip (used in WRT1900AC) may be an exception to that, so ODP can run on that in theory, but then you still need to program your ODP based application to talk to Marvell's network hardware through ODP.

What about hardware? (was: The EFF launches a router project)

rknight — Fri, 01 Aug 2014 04:54:57 +0000

Is it possible to do 802.11ac on miniPCI? And if so are there currently any cards available? I have a couple of Netgear ProSafe routers that apparently have Cavium Octeon processors that I've been meaning to attempt OpenWRT on.

The EFF launches a router project

mtaht — Fri, 01 Aug 2014 04:21:15 +0000

cool, thanks, good to know! The kind of numbers that intel gets out of their SDK are *amazing* - and I wasn't aware that there was a similar effort on arm. The thing is, intel gets those numbers by having DMA direct to their very large caches, which arm chips generally lack.

I keep hoping someone writing hardware will make note of all the advancements in queue theory of late and make better ethernet chips (or, in arm's case, verilog or VHDL IP).

What about hardware? (was: The EFF launches a router project)

mtaht — Fri, 01 Aug 2014 04:17:22 +0000

In more direct answer to your question - no, haven't tried the octeon against 802.11ac. No pcie bus on the devices I've seen...

The EFF launches a router project

mtaht — Fri, 01 Aug 2014 04:10:46 +0000

I don't know of any open source support for MoCa yet. I AM aware of working linux drivers for it, but so far as I know they are closed source.

What about hardware? (was: The EFF launches a router project)

mtaht — Fri, 01 Aug 2014 04:09:48 +0000

The ubiquiti (ubnt) edgerouter series of octeon based routers added fq_codel support to their QoS system as of version 1.5 of their debian/vyatta based firmware, which is based on linux 3.4.

The HUGE thread this generated is behind the beta tester forum (you have to register on the site), but it was quite edifying and informative. http://community.ubnt.com/t5/EdgeMAX-Beta/Testing-fq-code...

We ran into some issues with ingress shaping that I hope will be resolved when they switch to linux 3.10 (which may be in 1.6 but I don't know). Overall the edgerouter lite was turning in QoS numbers about 2x better than what cerowrt on the wndr3800 can do (with less cpu), and the pro, 7x. Adding BQL looks useful on the octeon ethernet drivers as well but some rework down there seems needed; I keep hoping someone else will get around to it.

Openwrt recently has gained the ability to boot via tftp on the edgerouter series, and I just saw that support for the flash and usb had been added to the main openwrt tree for it, a few days back, so openwrt barrier breaker 3.10 on octeon may be getting closer to being useful for ordinary mortals.

I have been trying to find a router that could do ingress shaping at 250mbit for quite some time now, and maybe that box can do it eventually, with a lot more work through the stack.

I LOVE the edgerouter pro (8 ports) - getting that routing the yurtlab really simplified and sped up my configuration there (as without qos enabled, it can generally forward at 8gbits)

The EFF launches a router project

mtaht — Fri, 01 Aug 2014 03:57:05 +0000

I found the tomato community rather toxic to deal with.

http://tomatousb.org/forum/t-676664/trying-to-unravel-the...

The EFF launches a router project

mtaht — Fri, 01 Aug 2014 03:55:36 +0000

dd-wrt got fq_codel for it's QoS system a while back. It's uncertain if they grokked the need to up the target parameter for it at low bandwidths (basically the fq_codel target and interval should be increased to account for a MTU's worth of packets when used at low rates (below 3mbit), which is something we didn't figure out until some testing of the real world against the behavior of the ns2 model.

I like dd-wrt, but they tend to run a bit behind openwrt (for quite a few good reasons). I figure they will have the high level of ipv6 integration that openwrt now has soon, if not already.

The EFF launches a router project

mathstuf — Thu, 31 Jul 2014 23:16:44 +0000

The guest network can also be under WPA2… Just post the password on a bulletin board or something.

The EFF launches a router project

NightMonkey — Thu, 31 Jul 2014 23:08:43 +0000

Any consideration for Tomato derivatives for this project? I'm really fond of Tomato by Shibby. http://tomato.groov.pl/

What about hardware? (was: The EFF launches a router project)

rknight — Thu, 31 Jul 2014 21:20:17 +0000

>>I agree with another poster here that it looks like switching to arm is >>sanest as none of the mips boxes I've tried can drive 802.11ac fast >>enough.

Have you tried any of the routers based on the Cavium Octeon chipset? I realize these are all currently commercial class routers, but I suspect that the multi-core Octeon chips would have no problem handling 802.11ac.

The EFF launches a router project

SiliconSlick — Thu, 31 Jul 2014 19:01:08 +0000

I bit perplexed about this as well. My Netgear WNDR3700v3 has a built-in option for wireless "Guest" networks, but I'm not about to enable it. Spammers and kiddie-porn peddlers are the first things that come to mind whenever I consider it. Who would put themselves in the situation where they could be held responsible for those miscreants' actions? I'm sure as hell not going to.

The EFF launches a router project

jhhaller — Thu, 31 Jul 2014 14:58:10 +0000

The generic activity (but driven by ARM) comparable to the Intel DPDK effort is OpenDataPlane. It's not nearly as advanced as DPDK, but may be of interest to OpenWRT and downstream projects.

The EFF launches a router project

ortalo — Thu, 31 Jul 2014 08:08:52 +0000

I second your comment and imho, that's not a simple risk but a true question: how do you comply to the (many) laws that oblige one to provide the police the necessary means to pursue their investigations (via logging, user registration, etc.)?

NB: Please, no comment only on the adequacy of these laws, that's not the question here (though I agree this is certainly the problem).

Possible alternative: Raspberry Pi + Modem

dlang — Wed, 30 Jul 2014 21:32:04 +0000

Openwrt runs on x86 systems as well, again it's not as limited as you think.

It runs on more types of devices that Fedora does.

Remember, it's still using the Linux kernel, and the userspace is still *nix. So it's not nearly as limited as you are thinking.

Possible alternative: Raspberry Pi + Modem

debacle — Wed, 30 Jul 2014 21:26:19 +0000

I just distinguished between a specialised OS such as OpenWRT, which runs only on routers, and an universal OS, which runs - more or less identical - on a server, a desktop, or an embedded device. The latter could be Debian (my choice) or Fedora, Mint, Redhat, S.u.S.E., Ubuntu etc., of course. To me it is an advantage to use the same OS for different purposes, but this depends very much on individual needs/choices.

The EFF launches a router project

dlang — Wed, 30 Jul 2014 19:36:36 +0000

I only used dd-wrt a little bit, but it looked to me like the only thing that it may be suitable for was running an AP, while OpenWRT (and derivatives) are much more general.

configuration for dd-wrt was a registry of name-value pairs while Openwrt uses config files and scripts.

There is also a very large selection of packages available in Openwrt while dd-wrt is just a router.

The EFF launches a router project

ms-tg — Wed, 30 Jul 2014 17:34:54 +0000

Thanks for posting this info.

Could anyone knowledgeable also comment on how the state of DD-WRT compares to these projects, by any chance?

What about hardware? (was: The EFF launches a router project)

fredrik — Wed, 30 Jul 2014 07:16:55 +0000

Thanks for sharing your insights into the state of router hardware today.

I hope that a future open hardware project can show the current manufacturers (linksys, etc) that it is advantageous to make router hardware that is open, and that it sells well because the consumers knows there is a community that will support it with security updates long after the router has been sold.

Possible alternative: Raspberry Pi + Modem

drag — Wed, 30 Jul 2014 05:13:35 +0000

Home routers are not really 'universal' devices. I normally see the proliferation of different Linux distros as largely a waste of time (beyond somebody who likes it as a hobby), but this is one of those cases were a custom Linux OS and unique package manager is entirely justified.

These type of devices feature specialized hardware like a programmable switch that will be very poorly handled by typical Linux distributions.

Openwrt is a fine Linux operating system and is worth learning how to operate.

The patches/improvements coming back in from Cerowrt has made the latency and performance of my home network's connection to the internet increased significantly. This has brought tremendous benefits when it comes to things like bittorrent traffic and VoIP and has laid waste to the vague suspicions of the ISP being dickheads... In the vast majority of cases it's people's home routers that suck, not the ISPs.

I really doubt that throwing Debian into the mix would yeild a superior router.

--------

Now if you really want to use Debian for this sort of thing keep in mind that the future for 'Enteprise' networks is 'Software Defined Networking'.

That is instead of screwing around with crap like 'Vlans' and struggling with managing QoS using expensive Cisco gear... you just make a 10GbE/40GbE network that is simple as can be (while maximizing 'east-west bandwidth rather then north-south in traditional network models) and then run a entirely software defined network stack on top of that. This is very effective, very cheap, and works very well. Network folks tend to be sticks-in-the-mud and love their vendors, but eventually it's going to start having a big impact outside of the large OpenStack cloud providers.

This way when you do your 'cloud' people can define the network by how they want the network to be, on the fly, via web interfaces or whatever. Just let the consumer in your business decide what sort of network they like to see and then build it for them without having to configure a single firmware or wire up a single port.

To enable this sort of stuff one of the pioneers is 'Culumus Networks' and they have developed a commercial Debian-based Linux distribution to install on various 'white box switches'. So instead of using big proprietary gear from companies like Cisco you buy generic switches from ODMs from Korea (or whatever) and install Debian on it to control the hardware backplane.

Another approach is to take a cheap 'desktop' Linux system and stuff it full of dual port NIC cards and install Debian on it. You can use OpenVswitch to manage the bridge ports, which is massively more capable then the traditional 'Linux brctl-based switch' that is very common.

From there you can take a look at things like Openflow and OVSDB to peer into the future of network management.

This sort of stuff is going to be a big deal in a couple years so while it's a lot more expensive then a openwrt-based home router it will allow you to keep using Debian and potentially get marketable skills and contribute back to Debian making it 'more universal'.

The EFF launches a router project

mtaht — Wed, 30 Jul 2014 02:31:29 +0000

I need to point out there is nearly zero cpu impact of the new queue management stuff, be it aqm or fair queuing. There are even beneficial cache effects from keeping the queues shorter with BQL, also -

the real killer for cpu is the cost software rate limiting, which accounts for like 82% of the overhead before a box flatlines. (I confirm the peak forwarding rate for the wndr3800 appears to be about 330mbit/sec with no iptables rules using fq_codel on the ethernet device, and about 50mbit with software rate limiting)

I'd like to come up with a better rate limiter, and feel that hardware assistance is going to be needed with the current generation of arm router based products as well.

The EFF launches a router project

mtaht — Wed, 30 Jul 2014 02:26:52 +0000

my benchmarks show the perceived benefits of the mips 74k cpu to be mostly illusion. To get roughly 2x more performance on simplistic benchmarks, it has a much deeper instruction pipeline, but usually has the same cache sizes as prior chips. (32K/32K). So you have worse interrupt response time, and not enough cache to do more stuff, both of which are critical things needed in a router.

An advantage of the arm chips is they generally have a L2 cache that is worthwhile and shorter pipelines. And Intel did some great things with ivy bridge direct to cache dma interface and their DLPK system that I wish the lower end routers could duplicate...

Personally I'm in love with the parallella right now (if only the 16 core co processor was more of an I/O co-processor) and its dual A9 core AND FPGA.... got enough gates to do http://jvimal.github.io/senic/ and maybe fq_codel too...

Possible alternative: Raspberry Pi + Modem

dlang — Tue, 29 Jul 2014 22:47:03 +0000

Openwrt has opkg, but if you are deciding based on the package manager, go for it.

I'd love for someone to take the time to make a installation profile that could be squeezed down to a router.

I was just taking issue with the description of OpenWRT as being a specialized distro.

you can define terms so that anything other than Debian is specialized, but if you go a bit broader (to include things like Fedora, Suse, RHEL), then OpenWRT should qualify as well.

Possible alternative: Raspberry Pi + Modem

debacle — Tue, 29 Jul 2014 22:23:53 +0000

Of course, Raspberry Pi is not the perfect HW for this, and Debian does even support hardware floating point on it (Raspbian does, however). With OpenWRT I'ld probably miss dpkg and apt, AFAIK. Nothing wrong with OpenWRT, but I like to stay with my universal OS :~)

The EFF launches a router project

dlang — Tue, 29 Jul 2014 22:17:39 +0000

Well, since there is more processing being done with active queue management, the I/O thoughput isn't the limiting factor.

I've had testing done with the 3800 with a very stripped down kernel (down to disabling connection tracking and all firewalling because it wasn't needed for the application), and the number reported back was that it topped out at 300Mb/sec

The EFF launches a router project

arnd — Tue, 29 Jul 2014 22:08:20 +0000

WNDR3700v4 has a 74Kc CPU core, so it should be a bit faster than the 24Kc on the 3800, but it won't make a lot of difference in the end, even the QCA9558/MT7620A/BCM4706 as the current high end on MIPS SoCs are probably too limited.

IPQ8064 will have a much faster CPU but the I/O path is as crippled as before by lacking cache-coherent DMA over PCI. Armada XP (from WRT1900AC) should have much higher I/O throughput, but its PJ4B CPU cores are not as fast as the Krait cores used in IPQ8064.

In terms of raw CPU performance, I'd expect doubling speed with each step AR7161->QCA9558->BCM47081->MV78230->IPQ8064, but in practice you see less than that on network transfers because they will all spend a lot of time waiting for I/O to cross the bus.

The EFF launches a router project

josh — Tue, 29 Jul 2014 21:54:06 +0000

Do any of the Open Source router projects have any support for MoCA?

Possible alternative: Raspberry Pi + Modem

dlang — Tue, 29 Jul 2014 21:42:31 +0000

> I really don't want to use any specialised distribution, such as OpenWRT

OpenWRT isn't much more specialized than Debian is

It's designed to work on much smaller, less powerful devices than Debian is, but the router hardware is smaller and less powerful only in specific ways.

a 3800 router is going to outperform a RaspberryPi at network related tasks due to the fact that the Pi has to put all it's networking through it's USB interface (which has significant bufferbloat issues remaining by the way)

I would suggest that you go look through the list of packages available for OpenWRT and then see what you need that's missing.

Possible alternative: Raspberry Pi + Modem

debacle — Tue, 29 Jul 2014 21:25:36 +0000

I thought about replacing my proprietary home DSL modem/Wifi router combo with a free software, but I really don't want to use any specialised distribution, such as OpenWRT or the new EFF project (which I welcome).

Now I bought a Raspberry Pi B+ and a separated DSL modem and hope to make it run as a replacement for my proprietary stuff just by installing Debian with PPPoE and HostAP. Let's see when I find a free weekend for this...

The EFF launches a router project

dlang — Tue, 29 Jul 2014 21:08:20 +0000

The problem with the WNDR3700v4 (and 4300 which is almost the same board) is that the flash is NAND, which requires badblock support.

That support was not available until a few months ago, and I think I saw a patch tweaking support for these models within the last week, so they are very recent entreats as far as OpenWRT is concerned.

As a like-for-like replacement of the 3800 it wouldn't be bad.

But if we are going to move CeroWRT to a new platform, it would be very nice to get something with a faster CPU because the bandwidth available to home users (in some areas) is increasing, and many of those people are the ones who would be most interested in addressing these sorts of problems

Depending on configuration and traffic, the 3800 can run out of CPU somewhere in the 10-50Mb/s range when running CeroWRT.

What about hardware? (was: The EFF launches a router project)

mtaht — Tue, 29 Jul 2014 20:50:55 +0000

The basic mips-based atheros chipset this stuff runs on has been continually updated (4 revisions over the last 6 years) and is in 100s of products currently sold. While I have CeroWrt running on several of these newer platforms, none have been compelling enough to switch to using, although several buffalo and TP-link products come close. I agree with another poster here that it looks like switching to arm is sanest as none of the mips boxes I've tried can drive 802.11ac fast enough.

OpenWrt itself runs on a lot of more current platforms than the wndr3800! - Go ahead, use those! but as CeroWrt has a rather large installed base of wndr3800s I have mostly been trying to make everything stable on that... and once that is stable... getting some of the science done on evaluating new queuing mechanisms... papers written, data sets sorted through, etc, etc.

and then porting over to a newer platform in the fall and starting new work (make-wifi-fast) on x86.

I agree that now that everything else is looking good (in openwrt and cero, anyway), that the stack is basically solid - that it would be best to try to do some sort of open source hardware for the next generation, in collaboration with the many other research oriented projects out there like project bismark, homenet, homewrt, openwireless, commotion wireless, dozens, nay hundreds of others, but it seems difficult even then - even if we could all co-operate on a RFP and buy to get to a large enough and cost-effective enough volume to be worth doing more engineering up front.

... and that requires effort and money up front that we do not have, and while we could perhaps raise enough via kickstarter it has been hard enough to express the advantages of a bufferbloat-free, blob-free, highly secure and flexible router during the project, much less beforehand. Despite the successes of projects like the rasberry pi and parallella, there would need to be some feasible long term gain and penetration into markets that linksys, buffalo, tp-link currently dominate, on razor thin margins...

Generally people only care about their home routers when they don't work. I have certainly watched smaller router-oriented ideas go by on kickstarter with something like jealousy...

I am extremely happy with where openwrt barrier breaker stands today, and I hope that the bigger manufacturers and ISPs decide to switch to it or something derived from it, on their next generations of hardware. Nothing does ipv6 as well as openwrt now does in particular, and beating bufferbloat so thoroughly has been quite satisfying.

What about hardware? (was: The EFF launches a router project)

fredrik — Tue, 29 Jul 2014 19:54:52 +0000

Is there a plan to complement this software project with some hardware building project, presumably funded on Indiegogo or Kickstarter? I really look forward to buying a router that has community support down to the last piece of physical wire on the board.

It is only too common to see free software projects like this target (workable, somewhat old) hardware, which unfortunately is more or less abandonware. I understand the need to access hardware that has open specifications, or failing that, is old enough to have been reverse engineered. But I can't help but think that all that effort spent to reverse engineer, replace binary blobs, and work around undocumented hardware bugs, might have been better spent getting it right from the beginning rather than fixing what is broken and old.

The EFF launches a router project

arnd — Tue, 29 Jul 2014 19:15:35 +0000

It seems surprisingly hard to find a good device with more than 16MB flash. Its successor WNDR3700v4 (aka WNDR4300) has 128MB NAND flash and could lift that barrier, but almost everything else that has a lot of flash also requires proprietary firmware blobs for the wireless drivers.

As I commented last week, it seems worthwhile to wait for the next generation of routers that also has dual-core ARM processors instead of single-core MIPS. OpenWRT currently has initial support for Marvell ArmadaXP and Broadcom bcm4708, but they have the same problem with wireless drivers and are not yet stable enough for a project like CeroWRT.
The new Atheros IPQ8064 (derived from Snapdragon S4 Pro) should show up in devices (Netgear R7500?) at some point this year, and that has a slightly better chance of getting free firmware, as mtaht mentioned.

The EFF launches a router project

hechacker1 — Tue, 29 Jul 2014 18:41:45 +0000

I have the 3800 with Cerowrt, and it's running the jffs2 version. It does have free space and can be provided updates. However, I don't know how much the extra EFF features take up. It has enough RAM for tmpfs to hold the next firmware image to install, and it can save settings in between installs. There's just a few bugs they are still working out.

I think the bigger problem is that cerowrt is fairly fast in releasing point patches, with the latest linux kernel, and so patch levels aren't necessarily compatible. EFF should probably freeze on a version of cerowrt and stick with it for a while, but I bet they cannot yet due to outstanding bugs in cerowrt and openwrt.

And most of the work cerowrt is already upstreamed to openwrt as well.