Some Cerowrt updates

[Posted December 7, 2011 by corbet]

For those who are looking forward to the 1.0 release of the Cerowrt bufferbloat-fighting router distribution, Dave Täht has posted a status update. "In summary, it IS possible to build a device far superior to anything shipped today, one that can set an example for the rest of the industry, save everyone time, and hassle, and do it for very low cost, AND consisting almost purely of software that can be applied to any hardware with the same capabilities, thus introducing competition to drive the costs down while holding the desirable feature set to a whole new standard." There is discussion of reducing the feature set to make the distribution more manageable, though; possible "puppies to shoot" include bind9 and ISC-ntp, mesh networking, and extensive IPv6 support.

to post comments

Some Cerowrt updates

Posted Dec 7, 2011 19:38 UTC (Wed) by mtaht (guest, #11087) [Link] (25 responses)

The recent adoption of Byte Queue Limits (BQL) into the kernel, successes regarding using things like HTB, QFQ, and RED to bound latency better, and the current discussions involving using Time in Queue, make me feel as though we're about to turn back the tide of bufferbloat, linux-wide. Far more thought, engineering, and testing will be required, of course...

That's what's driving me to think about shooting the other technological puppies, and/or adopting different ones.

I'm glad we now have - as we needed so badly back in june - a reliable, fully debugged, well understood wireless test routing platform in the wndr3700v2 and wndr3800.

However almost all the good stuff that happened this year has been pushed up into openwrt and/or into the mainline kernel. I'd like to go lean and mean with cerowrt, and try to track the BQL development process closely next quarter.

At this point, the only things I'm certain will be in cerowrt-rc8 (or whatever we end up calling it, as having 'release candidates' at this point seems to be resembling 'Duke Nukem Forever') are BQL, and some advanced AQMs.

http://www.bufferbloat.net/issues/305

Everything else currently in it, is on the table.

BTW:

The uptime on the router where we first got DNSSEC to work is currently:

21:57:14 up 213 days, 13:39, load average: 0.00, 0.01, 0.04

It was a wonderful weekend, when we did that. People said it couldn't be done. Proof of concept, on just that one concept, achieved!

Not a lot of progress on DNSSEC since, as it turns out that getting this ONE new BIT in DNS handled correctly is going to be a PITA. The 'right' way to do it is to create a standard for getaddrinfo that makes sense, and I'm not the guy to do it.

I'm hoping that there is someone out there that thinks DNSSEC is important enough to more fully deploy. I think it's very important, but at the moment, I'd rather rest up over the holiday, and then go back to beating the bloat.

Details on what the 'right thing' to do for DNSSEC are at:

http://www.bufferbloat.net/issues/113

Some Cerowrt updates

Posted Dec 7, 2011 20:13 UTC (Wed) by mtaht (guest, #11087) [Link]

I have to correct something I just said.

Jim and I and Jon and a bunch of other people wrote a lot, gave a bunch of talks, read a lot of papers, tried a lot of stuff, and cajoled, bribed, convinced, and sometimes even nagged a whole lot of engineers to take a hard look at and then fix what we thought were problems.

*We* didn't push stuff upstream.

Stuff got *done*, and ended up upstream because it was the right thing.

And for all that, I misunderstood what tom herbert's BQL might be able to do for months, because I was too heads down in cerowrt.

The past couple weeks have seen an explosion of additional stuff 'getting done'.

My hat is off to those doing actual engineering. That list of talented folk is far too long for me to thank all the individuals involved at this point. And then there were those that organized those talks, or loaned us facilities and office space and hardware, and that list is also too long at this point to go into. And then there were those that gave advice, or did testing, or contributed packages, or participated on the bloat list - and that list is too long to go into.

It's been a great year, and I'm looking forward to the next in whatever form it may take. Virtual egg-nog all round.

Some Cerowrt updates

Posted Dec 7, 2011 21:25 UTC (Wed) by Lennie (subscriber, #49641) [Link] (23 responses)

Did they tell you DNSSEC couldn't be done ?

For a home router, that would mostly be recursion, which is easy, install Unbound with auto-key and NTP, done. ;-)

On the authoritive side it has been much harder to get everything done, lots of automation and very strict checking is needed for key rollover.

Anyway, I think IPv6 is probably more important now. Still far to little provider networks have IPv6 deployed.

In 2012 IPv6 deployment will be inevitable, every network that does not do it will start to lose customers.

IPv6 on these little boxes in people's homes

Posted Dec 7, 2011 21:59 UTC (Wed) by tialaramex (subscriber, #21167) [Link] (10 responses)

If you ignore niche "DIY" solutions like running a full-blown Linux x86-64 desktop machine as the endpoint, or flashing your own firmware onto a cheap MIPS system that wouldn't do it out of the box, there is no consumer level IPv6 capable CPE, whether in the form of a "DSL router" or "Wireless router" or whatever.

The big step forward for 2011 over last decade is that several suppliers _tried_ to sell such things, but they all struggled to deliver something reliable enough and cheap enough to be acceptable to even a generous, technically competent and IPv6-enthused ISP, let alone the cut-price ISPs supplying most users.

Even if one of these (or another previously unknown) manufacturers actually gets the formula right in 2012, there's no way most home customers will have IPv6 capability from their CPE in 2012, it's too big a deployment task. After the "Cliff ahead, divert now" signs ran out and the engine went quiet, that wasn't because the danger was averted, it's because we went off the cliff. The next sound will be the crash as we hit the bottom. Strategies involving braking or turning the wheel to avert disaster are now irrelevant.

IPv6 on these little boxes in people's homes

Posted Dec 7, 2011 22:04 UTC (Wed) by paravoid (subscriber, #32869) [Link] (6 responses)

You really should have a look at RIPE's IPv6 CPE survey¹ and ARIN's Broadband CPE² page before making such claims.

¹: http://labs.ripe.net/content/ipv6-cpe-survey
²: http://getipv6.info/index.php/Broadband_CPE

IPv6 on these little boxes in people's homes

Posted Dec 8, 2011 2:29 UTC (Thu) by tialaramex (subscriber, #21167) [Link] (5 responses)

Both these pull two tricks which are helpful if you want a warm glowing feeling about IPv6 availability or are convincing the boss that it's a viable choice for your business, but are irrelevant in terms of mass deployment to the homes of ordinary people.

The first is "Well, Cisco do it". Yes, and so do several other high-end switch/ router vendors. But they sell into the enterprise, where the guy installing the CPE knows what DHCP is and whether they want to enable NAT, not into homes where you need a diagram just to explain how to switch it on. Most importantly, they charge a LOT of money for these products.

The second is "This device requires you to plug it into an existing router". Ignoring for a moment the reality that this means you're not really doing native IPv6 but some kind of tunnelling (and so you will probably pay in terms of reliability and performance) it means an extra box per customer. That breaks the economics for the ISP. It's viable for a few fanatics, but it isn't a way to get millions of ordinary households

[ Digital broadcast TV is a useful comparison here. It was _possible_ to build boxes that turn a digital TV multiplex into an old-school PAL-encoded analogue signal to feed into a pre-SCART era television, and in fact such boxes were deployed where needed. But for the vast majority of people the digital TV upgrade happened when they purchased a new, better television which had the local DTV standard baked in ]

The only ISP in my country that actually has ordinary household IPv6 customers right now, so far as I can tell, has despaired about finding a way to deliver CPE without these tricks. Some of their customers can afford to buy an actual Cisco DSL device, and others are happy to burn custom firmware or run some unsupported beta, but to roll it out to the rest of their customers, as they'd like to do, they need a real solution, one that works, is properly supported, and is available (not "in beta") at non-luxury prices and they haven't found one. Right now (as-in, since November) they're experimenting with Technicolor, but this is only the latest in a series of experiments, and most of the previous ones were abandoned, because a manufacturer says "How hard can it be?" and promises to support a test rollout then they discover the answer is "Actually it's not as easy as we thought" and that's it. Hardware vendors are really bad at software.

Of course it will all fix itself somehow, just don't believe the people who say it's going to be painless.

IPv6 on these little boxes in people's homes

Posted Dec 8, 2011 5:52 UTC (Thu) by mtaht (guest, #11087) [Link] (4 responses)

I am certainly NOT someone to stand up and say getting IPv6 to the home is painless. In fact one of the goals of the whole cerowrt R&D project was to clearly identify how painful it would be, and what sorts of investments in what subsystems and applications would be required to make ipv6 work better in that environment.

I like to think we've identified a few more areas in IPv6 that need serious work.

Upselling to NAT

Posted Dec 8, 2011 18:59 UTC (Thu) by dmarti (subscriber, #11625) [Link] (3 responses)

Moving a home user's service from a real IPv4 address to a NAT IPv4 address is a problem if the user is an EFF member, extreme gamer, Bittorrent user, or some kind of nerd.

As a big ISP, you could re-brand the change as an upgrade to "Family Friendly" or "High-Security Firewalled" service. Start out offering it as a paid upgrade, then make it "free for a limited time", then the default for new users. If it breaks some network service, just say that the other end doesn't like Families or Security.

This is so much easier than IPv6 that I don't see why the ISPs won't do this instead. (In Karachi, Pakistan, an ISP will sell you 100Mbps Ethernet to your neighbors, decent backhaul, and NAT for $8/mo. This works fine for Skype, Facebook, and YouTube.)

Upselling to NAT

Posted Dec 8, 2011 22:13 UTC (Thu) by drag (guest, #31333) [Link] (2 responses)

Many ISPs in the USA are providing NAT only IP addresses. I know people that have private-only IP addresses. The problems are more then just inconvenience for bittorrent users. It makes a lot of things suck.

They don't upsell, they just don't talk about it. They just do it. Then performance and reliability drops somewhat and irritates the people that depend on certain VPN software or VoIP for their work or whatever.

The problem ISPs now face is not enough public IPs for their customers, but not enough private IPs for their networks. When your a company like Comcast and you are forced to use multiple duplicate 10.0.0.0/8 networks you know you are running into some serious addressing limitations.

What we have going on right now is that ISPs are doing things like tunneling IPv4 over IPv4 in order to provide the network infrastructure necessary to deliver public addresses to customers. If you are going to give private addresses you are going to have to do some really crazy stuff like combination of tunning NAT connections some multiple NAT'd networks to get the TCP connections to customers. Tunneling IPv4 over IPv4 over Ipv4 type stuff.

On top of this NAT is not some sort of panacea that is going to buy you a lot of time.

The number of ports you have is limited to 16 bits minus whatever needs to be reserved. Each connection on a NAT router takes a port. So I'm guesstimating that leaves you about 30,000 useful TCP connections per public IP address. I am sure that things start to break down before that. Just doing a simple google about "linux" and clicking on a wikipedia link I get about 20 TCP connections started. So for every public facing IP address you can serve 1000-1500 active customers. And this is not really something you can ramp up for peak usage or anything.. it's a hard limit. Once customers start banging against connection number limits then things are going to suck for them.

What is going to happen, what I am guessing is happening is this:

ISPs are going to switch their networks entirely over to IPv6. They will be tunneling IPv4 over IPv6 to their customers. I expect that the larger ISPs are already well on their way. Once the roll out of the migration from IPv6 to IPv4 is complete then they will start fazing out support for DOCSIS 2.x modems and whatever the equivalent for DSL is. The newer 3.0 modems have IPv6 built in as a requirement for the protocol. Then they will require customers to purchase new routers. These things are probably going be pure IPv6 with a single IPv4 address mapped to it that tunnels over the IPv6 network.

'Computer Appliances' will start requiring IPv6 for various things. They won't really be advertising it as such they will just say things like ISP compatibility and DOCSIS requirements and such. It'll be a headache, but it will need to be done. They will require the higher performance that needs the newer level 1 protocols anyways. Blueray players, 'Smart' Televisions, IP-based cable boxes, PS3/XBox/etc, that sort of thing.

One thing that is important to keep in mind is that you don't really need IPv6 support on your 'home NAT router' to have IPv6 support in your appliances and in your OS. Any modern system can have full IPv6 internet access very easily on any "IPv4-only" NAT network. Full stack access. No firewalls, no port limitations, nothing. 100% unfiltered access right through any common NAT firewall. I think things like PS3 support this already. Microsoft already uses it for Windows 7.. it's requirement now for file sharing to work properly nowadays. Linux it's a bit more of a headache, but it's slowly catching up to Microsoft.

So it's not really necessary for customers to upgrade their home routers, but it will make things better.

While all this is happening it will slowly start to dawn on businesses in the USA that they will not be able to do business in certain parts of the world unless they have some sort of IPv6 connectivity support. Especially with China.

As IPv6 gradually moves to mainstream it will free up more and more IPv4 addresses to extend the useful life of that protocol.

Upselling to NAT

Posted Dec 8, 2011 22:26 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

> The number of ports you have is limited to 16 bits minus whatever needs to be reserved. Each connection on a NAT router takes a port. So I'm guesstimating that leaves you about 30,000 useful TCP connections per public IP address. I am sure that things start to break down before that. Just doing a simple google about "linux" and clicking on a wikipedia link I get about 20 TCP connections started. So for every public facing IP address you can serve 1000-1500 active customers. And this is not really something you can ramp up for peak usage or anything.. it's a hard limit. Once customers start banging against connection number limits then things are going to suck for them.

actually, a connection is the set

source IP, source port, destination IP, destination port

so you can re-use the same source IP and source port if you have a different destination IP and/or destination port.

the OS doesn't re-use the source port by default, but it could. so there is really no long-term reason for the NAT boxes to run out of the ability to handle connections.

Upselling to NAT

Posted Dec 9, 2011 16:39 UTC (Fri) by ncm (guest, #165) [Link]

Amusingly, that's been patented.

IPv6 on these little boxes in people's homes

Posted Dec 7, 2011 22:10 UTC (Wed) by dlang (guest, #313) [Link]

a lot of things depend on how you define 'disaster'

I fully expect that additional layers of NAT and a market in IPv4 addresses is going to make the crash a lot softer than many people are predicting (and I think soft enough that most people won't notice it)

however, the existence of NAT is enough to qualify as a 'diaster' for many IPv6 advocates

IPv6 on these little boxes in people's homes

Posted Dec 8, 2011 6:10 UTC (Thu) by mtaht (guest, #11087) [Link]

'After the "Cliff ahead, divert now" signs ran out and the engine went quiet, that wasn't because the danger was averted, it's because we went off the cliff'.

Well said. The first generation of manufacturers that tried to produce 'IPv6 certified' gear tried to make their old kernels (usually 2.6.16 based) do it, on hardware designs and assumptions that were equally old, and failed.

The only solution to having gone off that cliff that jg and I could come up with was to try and prove that the modern kernel development process was the fastest and best way to swap out the wheels and engine, and install wings before we hit bottom.

IPv6 on these little boxes in people's homes

Posted Dec 10, 2011 17:02 UTC (Sat) by ballombe (subscriber, #9523) [Link]

One of the largest French ISP provides IPV6 out of their ADSL box with automatic client-side configuration. I use it all the time.

DNSSEC

Posted Dec 8, 2011 5:47 UTC (Thu) by mtaht (guest, #11087) [Link] (11 responses)

At the homenet meeting back in august, someone stood up and said "bind9 and DNSSEC will never run on a home router"...

... and we'd already had it up and running for quite some time at that point.

however, what you write above strongly implies that you haven't tried to get DNSSEC to work right on cheap CPE. You see, DNSSEC requires not that time be accurate, but accurate to within an hour.

Cheap CPE does not come with a battery backed up clock. So you end up with this circular dependency on getting time, for which you need DNS. There are related failure modes where basically you DO want to be able to check if it's DNS failing or DNSSEC and do the right thing.

See bug 113.

DNSSEC

Posted Dec 8, 2011 11:56 UTC (Thu) by Lennie (subscriber, #49641) [Link] (10 responses)

Yes, I know of the circular dependency of DNSSEC and NTP.

I just didn't know a cheap CPE did not have a battery.

Thanks for clearing that up.

DNSSEC

Posted Dec 8, 2011 13:43 UTC (Thu) by job (guest, #670) [Link] (9 responses)

It's probably stupid to run your DNS resolver anyway. Most end users rely on their ISP to do it for them. (And that's why DNS blacklists can work.)

DNSSEC

Posted Dec 8, 2011 14:53 UTC (Thu) by mtaht (guest, #11087) [Link] (8 responses)

Running your own dns resolver is useful.

It cuts 10s of ms of latency off your provider's resolver for cached data.

It lets you have your own DNS for inside your home.

It lets you have trustworthy DNS data when your provider does not provide it - for example, playing with NXDOMAIN.

DNSSEC

Posted Dec 8, 2011 19:33 UTC (Thu) by job (guest, #670) [Link] (7 responses)

The super cheap hardware discussed here won't cut down on latency compared to your neighboring DNS server, especially then the CPU is taxed by routing many streams at once. The bigger DNS server will also have better caching, and more users to keep said cache warm. That's probably why most consumer broadband boxes just forward queries and is not really a resolver in its own right.

DNSSEC

Posted Dec 8, 2011 20:27 UTC (Thu) by mtaht (guest, #11087) [Link] (4 responses)

Um, er, no, a 680 mhz 128MB wireless router caches dns traffic, routes 500+Mbit streams, does a bit of AQM, and does various other things just fine without melting down. Since most the time it's running a heck of a lot slower than that, caching DNS locally is a win.

Secondly, you still can (and should) use the upstream provider as a forwarder DNS, for example, comcast's DNSSEC servers are usually less than 10 ms away. But in the home, your DNS server is .02ms away.

DNSSEC

Posted Dec 8, 2011 22:24 UTC (Thu) by Simetrical (guest, #53439) [Link] (3 responses)

Hmm. But the OS DNS cache is 0.001 ms away. Why isn't that good enough? I can see why you'd benefit from your own DNS server if you have a whole bunch of machines behind it, so they can pool the cache. But if it's just two or three clients, like in a typical home, what purpose does having a DNS server in the router serve?

DNSSEC

Posted Dec 9, 2011 0:57 UTC (Fri) by zlynx (guest, #2285) [Link] (2 responses)

Windows has an OS DNS cache. Oddly enough, it seems most Linux distros do not install a DNS cache by default. I'm not sure about Android.

I'm going to blame NetworkManager for this Linux situation. It used to be pretty easy to modify the network scripts to always point DNS to localhost. NetworkManager seems it makes it far too difficult to configure a local DNS cache.

If you do figure that you need to add dns=dnsmasq to the configuration file, it turns out that dnsmasq is the only supported local cache, and then you find out that it couldn't possibly have been tested, as it crashes NetworkManager randomly (or possibly when two interfaces come up, or it might have something to do with VPNs, or maybe suspend/resume).

Really, the whole caching DNS is a lot easier to set up on the router.

DNSSEC

Posted Dec 11, 2011 19:08 UTC (Sun) by niner (guest, #26151) [Link] (1 responses)

> I'm going to blame NetworkManager for this Linux situation. It used to be pretty easy to modify the network scripts to always point DNS to localhost.
> NetworkManager seems it makes it far too difficult to configure a local DNS cache.

On the KDE Network Management Plasmoid I just edit the connection and change the Method from "Automatic (DHCP)" to "Automatic (DHCP) addresses only" and type the IP address of my DNS server into the DNS Servers field, hit OK and be done. I find this much simpler than with the configuration method I used before NetworkManager. And I can change this per connection so that at home I use my local DNS while at work I use whatever the DHCP server gives me.

DNSSEC

Posted Dec 12, 2011 3:35 UTC (Mon) by zlynx (guest, #2285) [Link]

But how do you update the forwarding address in your DNS cache daemon?

DNSSEC

Posted Dec 8, 2011 20:53 UTC (Thu) by zlynx (guest, #2285) [Link] (1 responses)

Agree with mtaht.

I used to run my home network router on a dual Pentium 166. It did routing with IPv6 tunnels and QoS, Squid, DHCP, web server and BitTorrent.

It worked very well. The CPU in modern home routers (well, above the $60 level anyway) is much faster than that P166 was, so I expect it will do an even better job.

DNSSEC

Posted Dec 8, 2011 22:08 UTC (Thu) by dlang (guest, #313) [Link]

there are some $30 single-band wifi routers with 400MHz cpus and 64M ram, they are probably fast enough to do this as well.

CPE

Posted Dec 8, 2011 8:22 UTC (Thu) by ncm (guest, #165) [Link] (1 responses)

Apparently CPE means "customer premises equipment".

It's nice to learn that the real reason IPv6 isn't well deployed is that it doesn't work well enough to deploy. But if it doesn't work well enough, what is it that Cisco is selling? Is supporting it properly just too resource-intensive for a 200MHz MIPS with 4M RAM, so the specs need to provide a way to offload work to more-expensive routers upstream?

Heaven forfend that a router manufacturer might be obliged to ship with as much RAM as is found in any cereal-box-prize MP3 player.

CPE

Posted Dec 9, 2011 0:32 UTC (Fri) by tialaramex (subscriber, #21167) [Link]

Unlike in the core, where it was about money, at the edges it's mostly about lack of interest. The big ISPs who can order a Taiwanese-built $50 combination DSL modem, Ethernet switch, WiFi access point, Sandwich toaster and Christmas tree decoration in six figure quantities do not care about IPv6. In fact, they don't care about very much. Broken NAT, terrible latency (this is the cerowrt thread, remember?) awful configuration UIs, devices that need rebooting every few days... As a result, the Sandwich toaster / whatever supplied by your new ISP not supporting IPv6 is just one of many ways in which it will likely turn out to suck.

Customers have yet to wise up to this on the whole. The minority who understand that the ISP-provided device sucks will buy a $100-200 device from an electronics store, based on online reviews or personal recommendation. But even most of them don't care about IPv6. So if you're making home network devices it just doesn't make economic sense to prioritise IPv6 support, even if you strongly suspect that every device without IPv6 support will be obsolete in a few years.

If the idiots are queuing up on December 18th to buy 2011 calendars, who are you, as a humble calendar maker, to refuse to sell them and offer only 2012 calendars? The customer is always right, after all.