The EFF launches a router project [LWN.net]

The EFF launches a router project

Posted Jul 29, 2014 17:45 UTC (Tue) by dlang (guest, #313)
Parent article: The EFF launches a router project

re: upgrade issues

The big problem is that the 3800 just doesn't have that much flash available.

As a result, like many other OpenWRT devices, you end up needing to use a read-only compressed filesystem to hold the OS software. This gets coupled with an overlay of a read-write filesystem, but the compression is poor (or non-existant) on this overlay. The difference in image size between using the read-only compressed image vs the read-write image can be about 10x

This means that when you install a update to a package that was in the base OS, you don't free the space the original took up, and the new version takes a lot more space than the old one did.

As a result, you can't start with the highly compressed image and update it.

New devices are shipping with much more flash, so this may end up changing and OpenWRT will need to adapt it's scripts and packages to deal with the upgrade in place process better, but support for these new devices has been lagging (in part because the new devices show bad flash sectors to the OS, so the filesystem needs to be able to deal with them and with the possibility of loosing eraseblocks as they are written), but it is starting to show up now.

to post comments

The EFF launches a router project

Posted Jul 29, 2014 18:41 UTC (Tue) by hechacker1 (guest, #82466) [Link]

I have the 3800 with Cerowrt, and it's running the jffs2 version. It does have free space and can be provided updates. However, I don't know how much the extra EFF features take up. It has enough RAM for tmpfs to hold the next firmware image to install, and it can save settings in between installs. There's just a few bugs they are still working out.

I think the bigger problem is that cerowrt is fairly fast in releasing point patches, with the latest linux kernel, and so patch levels aren't necessarily compatible. EFF should probably freeze on a version of cerowrt and stick with it for a while, but I bet they cannot yet due to outstanding bugs in cerowrt and openwrt.

And most of the work cerowrt is already upstreamed to openwrt as well.

The EFF launches a router project

Posted Jul 29, 2014 19:15 UTC (Tue) by arnd (guest, #8866) [Link] (9 responses)

It seems surprisingly hard to find a good device with more than 16MB flash. Its successor WNDR3700v4 (aka WNDR4300) has 128MB NAND flash and could lift that barrier, but almost everything else that has a lot of flash also requires proprietary firmware blobs for the wireless drivers.

As I commented last week, it seems worthwhile to wait for the next generation of routers that also has dual-core ARM processors instead of single-core MIPS. OpenWRT currently has initial support for Marvell ArmadaXP and Broadcom bcm4708, but they have the same problem with wireless drivers and are not yet stable enough for a project like CeroWRT.
The new Atheros IPQ8064 (derived from Snapdragon S4 Pro) should show up in devices (Netgear R7500?) at some point this year, and that has a slightly better chance of getting free firmware, as mtaht mentioned.

The EFF launches a router project

Posted Jul 29, 2014 21:08 UTC (Tue) by dlang (guest, #313) [Link] (8 responses)

The problem with the WNDR3700v4 (and 4300 which is almost the same board) is that the flash is NAND, which requires badblock support.

That support was not available until a few months ago, and I think I saw a patch tweaking support for these models within the last week, so they are very recent entreats as far as OpenWRT is concerned.

As a like-for-like replacement of the 3800 it wouldn't be bad.

But if we are going to move CeroWRT to a new platform, it would be very nice to get something with a faster CPU because the bandwidth available to home users (in some areas) is increasing, and many of those people are the ones who would be most interested in addressing these sorts of problems

Depending on configuration and traffic, the 3800 can run out of CPU somewhere in the 10-50Mb/s range when running CeroWRT.

The EFF launches a router project

Posted Jul 29, 2014 22:08 UTC (Tue) by arnd (guest, #8866) [Link] (7 responses)

WNDR3700v4 has a 74Kc CPU core, so it should be a bit faster than the 24Kc on the 3800, but it won't make a lot of difference in the end, even the QCA9558/MT7620A/BCM4706 as the current high end on MIPS SoCs are probably too limited.

IPQ8064 will have a much faster CPU but the I/O path is as crippled as before by lacking cache-coherent DMA over PCI. Armada XP (from WRT1900AC) should have much higher I/O throughput, but its PJ4B CPU cores are not as fast as the Krait cores used in IPQ8064.

In terms of raw CPU performance, I'd expect doubling speed with each step AR7161->QCA9558->BCM47081->MV78230->IPQ8064, but in practice you see less than that on network transfers because they will all spend a lot of time waiting for I/O to cross the bus.

The EFF launches a router project

Posted Jul 29, 2014 22:17 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

Well, since there is more processing being done with active queue management, the I/O thoughput isn't the limiting factor.

I've had testing done with the 3800 with a very stripped down kernel (down to disabling connection tracking and all firewalling because it wasn't needed for the application), and the number reported back was that it topped out at 300Mb/sec

The EFF launches a router project

Posted Jul 30, 2014 2:31 UTC (Wed) by mtaht (guest, #11087) [Link]

I need to point out there is nearly zero cpu impact of the new queue management stuff, be it aqm or fair queuing. There are even beneficial cache effects from keeping the queues shorter with BQL, also -

the real killer for cpu is the cost software rate limiting, which accounts for like 82% of the overhead before a box flatlines. (I confirm the peak forwarding rate for the wndr3800 appears to be about 330mbit/sec with no iptables rules using fq_codel on the ethernet device, and about 50mbit with software rate limiting)

I'd like to come up with a better rate limiter, and feel that hardware assistance is going to be needed with the current generation of arm router based products as well.

The EFF launches a router project

Posted Jul 30, 2014 2:26 UTC (Wed) by mtaht (guest, #11087) [Link] (4 responses)

my benchmarks show the perceived benefits of the mips 74k cpu to be mostly illusion. To get roughly 2x more performance on simplistic benchmarks, it has a much deeper instruction pipeline, but usually has the same cache sizes as prior chips. (32K/32K). So you have worse interrupt response time, and not enough cache to do more stuff, both of which are critical things needed in a router.

An advantage of the arm chips is they generally have a L2 cache that is worthwhile and shorter pipelines. And Intel did some great things with ivy bridge direct to cache dma interface and their DLPK system that I wish the lower end routers could duplicate...

Personally I'm in love with the parallella right now (if only the 16 core co processor was more of an I/O co-processor) and its dual A9 core AND FPGA.... got enough gates to do http://jvimal.github.io/senic/ and maybe fq_codel too...

The EFF launches a router project

Posted Jul 31, 2014 14:58 UTC (Thu) by jhhaller (guest, #56103) [Link] (3 responses)

The generic activity (but driven by ARM) comparable to the Intel DPDK effort is OpenDataPlane. It's not nearly as advanced as DPDK, but may be of interest to OpenWRT and downstream projects.

The EFF launches a router project

Posted Aug 1, 2014 4:21 UTC (Fri) by mtaht (guest, #11087) [Link] (2 responses)

cool, thanks, good to know! The kind of numbers that intel gets out of their SDK are *amazing* - and I wasn't aware that there was a similar effort on arm. The thing is, intel gets those numbers by having DMA direct to their very large caches, which arm chips generally lack.

I keep hoping someone writing hardware will make note of all the advancements in queue theory of late and make better ethernet chips (or, in arm's case, verilog or VHDL IP).

The EFF launches a router project

Posted Aug 1, 2014 12:27 UTC (Fri) by arnd (guest, #8866) [Link] (1 responses)

My understanding of DPDK and ODP is that they are mostly interested in running proprietary network stacks in user space, in order to implement features there that don't exist in the kernel, but at the same time completely bypassing the kernel and the features that exist in it. The fact that Intel and some others can do DMA into caches is independent of that and you can do that in both kernel and user space based network stacks.

Note that ODP by design requires cache-coherent DMA, which does not necessarily imply doing the DMA into the cache, but it does mean that it is not portable to the typical low-end SoCs you'd find in consumer routers as opposed to the devices that Cisco and Juniper are selling to enterprise customers.

The ArmadaXP chip (used in WRT1900AC) may be an exception to that, so ODP can run on that in theory, but then you still need to program your ODP based application to talk to Marvell's network hardware through ODP.

The EFF launches a router project

Posted Feb 15, 2016 13:13 UTC (Mon) by nysan (subscriber, #81015) [Link]

"My understanding of DPDK and ODP is that they are mostly interested in running proprietary network stacks in user space, in order to implement features there that don't exist in the kernel, but at the same time completely bypassing the kernel and the features that exist in it. The fact that Intel and some others can do DMA into caches is independent of that and you can do that in both kernel and user space based network stacks."

Have a look at openfastpath on ODP.
Linux is still used as controlplane with --enable-sp, ARP, RIPv2 et.c. is sent to the kernel stack.
Standard cmds "arp", "ip" et.c. are used to configure parts of the stack thats common between both.
Control plane stack state is mirrored via netlink for fast access by packet churning cores, i.e. ARP table, FIB et.c.