LWN: Comments on "Networking on tiny machines"

Networking on tiny machines

dlang — Sat, 31 May 2014 07:56:42 +0000

The question of schedulers has been discussed many times. This is one of those things that Linus has very strong feelings on.

He's seen other systems start to have vastly different schedulers for different purposes, and the end result is that there is a lot of confusion and each one is tailored for a specific use case, but that use case doesn't match what the users are actually needing to do (and/or users end up needing to do a little bit of multiple different types of work that each scheduler developer says "well, if you want to do that, don't use my scheduler")

and I'll say that the current disk schedulers seem to have this problem, although there is a little bit of a difference in that sometimes the disk controller does some of the work for the kernel, so there is more of a case for the noop scheduler, but there isn't the equivalent for the task scheduler)

Networking on tiny machines

garyamort — Sat, 31 May 2014 04:22:48 +0000

"any new features have to be workable across the entire gamut" - no they don't. In the case of things like various schedulers and power control management - as long as the userspace API is identical they don't need to work across all systems.

The i/o scheduler for example has many different patterns, some of which work better under different circumstances.

Multiple processor schedulers could be included in the kernel as well - rather then requiring constantly updated patch sets for every new kernel.

For an area I am familiar with, just browse through the ARM directories:
https://github.com/torvalds/linux/tree/master/arch/arm

Lots of kernel bits there for custom chips which are only found in specific hardware devices.

Also since there seems to be some confusion, I'm not saying a lot of kernel devs are hostile to mobile phones or whatnot - I'm saying that for the most part, their personally concerned with large, multiprocessor servers and high end hardware. If a new feature doesn't have some affect on what their interested in, it is quite often dismissed. Consider the comments quoted for this article.

Changes which would provide real benefits today but "don't further" some future goal, which historically can take years to be implemented, and which don't benefit the personal use cases of the maintainer - rejected.

Changes which provide performance benefits in real use cases by the submitter are rejected because the maintainer guesses that they won't apply to others - while at the same time admitting that he wouldn't use Linux for any of those use cases so he is not in a good position to judge.

Networking on tiny machines

smurf — Thu, 29 May 2014 07:43:02 +0000

That being said, IMHO the demand for shrinking the kernel back down to something that works well on single-digit MByte machines is a reasonable goal.

It'll probably live in its own tree before getting into mainline, but then the linux-tiny patchset of earlier times did the same thing. So did/do the RT patches, for that matter.

Networking on tiny machines

raven667 — Wed, 28 May 2014 14:43:29 +0000

You are right that the reason these features were rejected as-is is not due to mobile devices small resources but it has a lot to do with the fact that the kernel has many many constituencies a large number of which are big iron and any new features have to be workable across the entire gamut of places where Linux is used, so features have to integrate well with the overall scheme, layout and style of the kernel, can't cause too many regressions on other completely unrelated systems and should be generally useful to those other systems as well which is why the specific code wasn't pulled in, as it was too much bolted on and single use, but the fact that these features were written provides an example of what is possible and what is needed so they were re-implemented in ways that had broader consensus across the Linux kernel ecosystem.

Networking on tiny machines

smurf — Wed, 28 May 2014 06:03:05 +0000

There are a couple of reasons why wakelocks or the Binder haven't been accepted as they are, and those have nothing to do with mobile phones' small footprints, memory or otherwise.

Don't twist history.

Networking on tiny machines

dlang — Wed, 28 May 2014 00:01:31 +0000

well, the OP talks about replacing functions instead of maintining patches against libraries.

what libraries does the kernel maintain?

It's far from obvious that this is talking about kernel functions.

Networking on tiny machines

renox — Tue, 27 May 2014 22:09:29 +0000

> the huge changes the Android devs have been making aren't in the kernel, they are in userspace.

Uh? These changes are not relevant here, as the GP was talking about Linux.

> There have been a handful of features that they've added to the kernel, and after a lot of discussion those are getting integrated (or replaced with something the upstream devs and the android devs can agree on). So at best you are mixing issues.

Given the time and the difficulty for to have these 'handful of features' integrated into the mainline, I'd say that he is right..

Networking on tiny machines

dlang — Tue, 27 May 2014 19:08:24 +0000

the huge changes the Android devs have been making aren't in the kernel, they are in userspace.

There have been a handful of features that they've added to the kernel, and after a lot of discussion those are getting integrated (or replaced with something the upstream devs and the android devs can agree on)

So at best you are mixing issues.

Networking on tiny machines

garyamort — Tue, 27 May 2014 18:17:29 +0000

I think it's been pretty clear for quite a while that the gatekeepers for Linux have been pretty much focused primarily on servers and the bright shiny 16-cpu their employers buy them for quite a number of years.

BFS was declared a failure because it didn't handle well on 16+ cpu's....which while that may have been common for linux kernel coders back in 2009 - it's not common for most people.

Android devs realized quite early that enhancements to make Linux function well on embedded devices would be difficult to get accepted - so they made the sensible choice of writing complete replacements for linux functions rather then try to maintain a patch set against libraries which kernel devs could change and break the patches. Which seems to have upset some kernel devs that instead of playing catch up and stroking their ego's - android devs just side stepped them.

It's really not that big of a deal though - in the end all that really happens is Linux gets forked into a specialized version - and if the distro becomes popular enough, kernel devs eat crow and grudgingly accept the code they previously rejected.

Sure, in the end a lot of kernel devs end up looking like incompetent twits when their 'reasoned arguments' turn out to be a bunch of narcissistic ego stroking, but Linux itself continually improves by absorbing the code that works well while at the same time avoiding having the handle all the fiddly debugging.

Networking on tiny machines

Cyberax — Tue, 27 May 2014 16:49:11 +0000

Hmm... And guided high-velocity ammunition seems to be a great solution for both of them!

Networking on tiny machines

mathstuf — Tue, 27 May 2014 16:41:14 +0000

Maybe "[aircraft] carrier-sized gnat" problems?

Networking on tiny machines

marcH — Tue, 27 May 2014 15:54:50 +0000

So, "carrier-grade NAT" (really need to find a more appropriate name for this kludge, preferably a funny one) is doomed?

Networking on tiny machines

krakensden — Tue, 27 May 2014 15:34:14 +0000

> some decentralized game

Many multiplayer console games- like Call of Duty- are, it saves on hosting costs. It's mostly invisible to the players though.

Networking on tiny machines

krakensden — Tue, 27 May 2014 15:32:25 +0000

Comcast claims they're up to 16%:

http://www.comcast6.net/index.php/8-ipv6-trial-news-and-i...

Networking on tiny machines

quanstro — Thu, 22 May 2014 14:16:49 +0000

it might be a mistake to over generalize from a targeted attack on a high-value asset.

Networking on tiny machines

dlang — Fri, 16 May 2014 18:41:23 +0000

that's already available, that's what LWIP is that Andi mentioned in his original post. It's not that small, and it only takes a few apps before it becomes larger than the kernel version (and only a couple to be larger than Andi's stripped down version)

Networking on tiny machines

piotrjurkiewicz — Fri, 16 May 2014 18:36:11 +0000

Luigi Rizzo's netmap goes that way: http://info.iet.unipi.it/~luigi/netmap/

I think such an approach could be a solution for the problems mentioned in the article too. I mean to implement some kind of lightweight kernel interface between NIC and userspace. Than user would be able to disable kernel networking and use his own userspace networking stack (either tiny or performance-oriented one) in extreme cases.

Networking on tiny machines

Wol — Thu, 15 May 2014 22:08:27 +0000

People returned them in droves? I think you've been taken in by propaganda!

There was the highly publicised example of one manufacturer. I think they claimed "the majority of our linux netbooks have been returned", only for somebody else to point out they didn't make any!!!

And when I asked (at ASDA, our Walmart subsidiary) I was told that they'd had pretty much NO returns at all. That was where I bought my netbook (which unfortunately I broke - I ended up thanks to wifely pressure trying to install XP on it. Somehow that trashed the BIOS :-(

No - there was a lot of propaganda to try and make linux netbooks look bad, but as far as I can make out, every time anybody dug into the figures they discovered somebody was lying with statistics ...

Cheers,
Wol

Networking on tiny machines

dlang — Tue, 13 May 2014 19:47:47 +0000

well, if turning on networking by itself is 400K, that's 40% of your 1M kernel before you do anything else. So it's going to be pretty easy to hit that limit.

Networking on tiny machines

mwsealey — Tue, 13 May 2014 19:40:58 +0000

Question: 2MB of writable memory or 2MB of XIP flash?

Can we even think of an application that justifies a 2MB kernel (or even a 1MB kernel!?) that wouldn't also take up more than 2MB?

Networking on tiny machines

rahvin — Tue, 13 May 2014 00:33:42 +0000

You are correct abandoning ipv4 isn't the right course, in retrospect what I should have said was more along the lines of popping up a big scary warning that tells the user to call their ISP.

I'm sure half their users calling support because Google told them their is something wrong with their internet would do two things, the first is make the ISP hate Google with a passion and the second is cause the ISP to ensure ipv6 is implemented and being used in preference to ipv4.

It's frustrating for me because I'm on Comcast business, just a month ago I finally got a free modem upgrade to support Docsis 3 and ipv6 (I had to specifically request this upgrade), when I inquired about ipv6 support which their own tools say are fully deployed on my CMTS I was told it's in beta on the business side and the beta is closed. That Beta was open to users last year. In other words the only way I can use ipv6 is if I had a modem that supported it and I requested to be part of a "beta" a year ago when I didn't have a modem that supported it. I'd take sweet relish in Google or Facebook doing that to Comcast.

Networking on tiny machines

eean — Sun, 11 May 2014 00:28:35 +0000

Talk to Iranian nuclear researchers about air wall security. :p

Mostly while the Linux kernel shouldn't be insecure itself, its role in providing security is for sure not always needed.

Networking on tiny machines

raven667 — Sat, 10 May 2014 22:30:46 +0000

> But I don't know how Windows fared in that transition.

This is a good point I hadn't seen mentioned before. What happened was that people ran Windows on both laptops and desktops, so Windows didn't suffer here, but people were highly award that laptops were inferior in almost every way so it is laptop sales which suffered, only people who valued portability above all else used them. Laptops only became generally popular in the mid 2000s when the specs finally caught up to desktops, or at least when single-core speeds leveled off.

Since we are on the subject it should be mentioned that the whole net book category of computers was designed for Linux first and in the competitive marketplace traditional Linux desktops lost fair and square, people returned them in droves.

Networking on tiny machines

marcH — Sat, 10 May 2014 21:26:48 +0000

> Feature management is done best by config options not branches.

It's one of the most classic configuration management trade-off. Only simple problems have simple answers here.

Networking on tiny machines

marcH — Sat, 10 May 2014 21:23:50 +0000

> It is interesting that an idea of a user-space IP stack is useful not only for tiny systems, but for those that wants to handle 1e6 connections with minimal latency.

Done at least here: http://www.shenick.com/products/diversifeye/ ; possibly also elsewhere.

Networking on tiny machines

giraffedata — Sat, 10 May 2014 21:23:15 +0000

Even some micro-controllers today are more powerful than the first systems Windows ran on...

True, but I don't know how that's relevant here, because the claim was that Microsoft bet that computers would continually get more powerful, not that they would always be more powerful than the first ones Windows ran on.

Actually, thinking about it some more, I realize that one of these backward steps happened well before the netbooks and tablets. It happened with laptop computers. The first laptops were far less powerful in every dimension than previously typical computers they replaced. They didn't even have color displays.

But I don't know how Windows fared in that transition.

Networking on tiny machines

marcH — Sat, 10 May 2014 20:56:25 +0000

Even some micro-controllers today are more powerful than the first systems Windows ran on...

Networking on tiny machines

giraffedata — Sat, 10 May 2014 20:44:37 +0000

One only needs to look at what has happened to Windows. They made the bet that computers would only become more powerful and have struggled for years with netbooks and tablets, etc.
Computers have obviously become much more powerful.

You seem to have missed the point, because computers have not become universally more powerful. Several entire classes of computers less powerful than previously common ones have been introduced and taken over much of the workload of those previous ones. So even if Microsoft hadn't added any bloat at all, it still would have lost market share. To stay even, it would have had to do what this article reports is proposed for Linux: take stuff out.

Networking on tiny machines

marcH — Sat, 10 May 2014 14:42:46 +0000

> The designers of devices and bills of material (BOMs) will fight for every last expense, and will not change their planned amounts of memory or storage when they can change software to be more efficient instead.

For small software changes you are right. Good luck explaining the concept of "upstreaming code" to some hardware engineers...

If on the other hand you talk about switching to a different operating system, which is effectively a major product change, then of course everyone from top to bottom listens.

Networking on tiny machines

marcH — Sat, 10 May 2014 14:38:45 +0000

> I think I must be misreading those numbers: between 1/25 and 1/2 of a square mm, really? Depending on what? Process geometry? Amount of RAM? Cache size?

For pure comparisons between features/blocks I think it's easier to look at number of gates - independent from the process.

Networking on tiny machines

marcH — Sat, 10 May 2014 14:36:29 +0000

> One only needs to look at what has happened to Windows. They made the bet that computers would only become more powerful and have struggled for years with netbooks and tablets, etc.

Computers have obviously become much more powerful. The problem is that Microsoft made a much bigger bet than this: they made the bet that the power of computers would grow much FASTER than Windows bloat.

With Windows phone it looks like they finally learned how to trim bloat. But too little, too late. Also, very poor choice of a name. They were probably too proud of it which shows how they've lost it.

Networking on tiny machines

marcH — Sat, 10 May 2014 14:17:37 +0000

> I expect these two factors to drive IPv6 for home users, it should be cheaper for ISPs to provision than expanding the NAT and it should be lower latency for customers where that matters like VoIP and gaming.

Peer to peer.

The one thing that crazy/triple NATs break is peer to peer.

I would only take a couple of successful peer-to-peer applications (think Napster, Skype, some decentralized game,...) to force ISPs to implement IPv6.

So what is very effectively delaying IPv6 (forever?) is... "cloud computing".

Networking on tiny machines

marcH — Sat, 10 May 2014 14:12:27 +0000

> 3% is hardly "people started switching en-masse"

It's not en-masse but it's millions: enough to prove it works on a massive scale.

> That's still in the "early adopters" combined with a little bit of "people don't realize they're using it"

I think the vast majority of people start using IPv6 when their ISP (and new Android version...) starts, which means they indeed don't even realize it.

Networking on tiny machines

mgedmin — Sat, 10 May 2014 11:38:47 +0000

Thank you, this was very interesting.

Networking on tiny machines

lacos — Fri, 09 May 2014 16:28:34 +0000

> The designers of devices and bills of material (BOMs) will fight for
> every last expense, and will not change their planned amounts of memory
> or storage when they can change software to be more efficient instead.

Indeed, there's a reason why they're called *hard*ware and *soft*ware.

Networking on tiny machines

raven667 — Fri, 09 May 2014 15:43:26 +0000

> moved their services to IPv6 and refused to provide ipv4 services

That's not going to happen, not now not ever. That's just not how the world works.

> Google/Facebook with a lot of IP space and basically a key component of the internet needs to step up and move completely to IPV6

This has pretty much happened, both Google and Facebook are dual-stack for their public facing properties, as are some of the major CDNs, Netflix and YouTube as well. There is a long tail of IPv4-only services that will exist for the next decade or two but all the highest traffic services are ready and waiting for clients to convert over.

> there is still a lot of equipment out there that's not ipv6 compatible

Not true of end-user devices like computers and phones but is true for many consumer routers, even though the cable DOCSIS 3 standard mandates IPv6 support for the modem, a lot of routers will have to be replaced (good business opportunity for router makers really, hopefully we can shoe-horn CoDel in the new deployment as well)

> Carrier grade NAT is a reality

Even my organization is moving forward with a large NAT system but we are tying the deployment of NAT with the deployment of IPv6 because everything which routes directly (all the most popular web properties mentioned above) doesn't have to go through the NAT which greatly reduces the expense of it.

I expect these two factors to drive IPv6 for home users, it should be cheaper for ISPs to provision than expanding the NAT and it should be lower latency for customers where that matters like VoIP and gaming.

Networking on tiny machines

jem — Fri, 09 May 2014 10:47:16 +0000

Check out the numbers on https://www.google.com/intl/en/ipv6/

The growth is steady, and there is a chance the for the global percentage to jump to 6-7 before the end of the year. The 3 % figure is for the whole Internet; the percentages for some countries are much bigger, e.g. USA 7.14%, Germany 8.38%, France 5.23%, Belgium 16.93%.

Networking on tiny machines

dlang — Fri, 09 May 2014 08:31:04 +0000

3% is hardly "people started switching en-masse" That's still in the "early adopters" combined with a little bit of "people don't realize they're using it" where some network people setup IPv6 because "it's the right thing to do" (as opposed to any push from the users)

now, if your last paragraph was written in future mode rather than in past tense, then I could possibly agree with you. But I think that there is a LOT more room for 'temporary' fixes (including sales of IPv4 addresses) in the meantime.

Networking on tiny machines

khim — Fri, 09 May 2014 08:24:43 +0000

last I heard, IPv6 traffic is somewhere in the 3-6% range

3.01% by latest Google's estimate. But it's grows quite strongly: the same time last year it barely crossed 1% mark.

It's well-known fact that Internet only just works. This time (as every time previosly) all attempts to postpone the switch were used in the very same way they were used in the past: to push switch back few years and do nothing in the meanwhile.

Only when screams “Aaargh. I need, really need XX IPv4 addresses or else my whole company will go down in flames” started getting calm “Oh, I'm so sorry that your company is going down in flames. Nice weather, isn't it?” response people started switching en-masse to IPv6.

Networking on tiny machines

dlang — Fri, 09 May 2014 07:56:07 +0000

so you are saying that some critical part of the internet should make itself inaccessible to 90%+ [1] of the internet for some unknown timeframe (but probably years) to force everyone to upgrade.

somehow I think that the competiton would just step in and the company would go out of business in the meantime.

[1] last I heard, IPv6 traffic is somewhere in the 3-6% range, and every network stack I've heard of uses IPv6 in preference to IPv4 if it works