LWN: Comments on "Multipath TCP: an overview"

Personal transit options: US

Duncan — 2013-04-06T10:37:21+00:00

Of course some USians (for that is what I assume you meant, I'd guess "Americans" in general are far more used to non-personal-automobile transit than the average USian) took pause at that parenthetical as well... While public transit isn't as widespread or as commonly used and therefore generally assumed useful in the US as it tends to be in many other nations, it's certainly there in the larger cities, and many people do depend on it, to the extent of no longer having a personal automobile, finding renting more cost effective for that rare occasion, or even no longer having a(n unexpired) driver's license.

Multipath TCP: an overview

eternaleye — 2013-04-04T07:45:32+00:00

I thought it might be worthwhile to point out the main page of the Tng group, who created Minion: http://dedis.cs.yale.edu/2009/tng/

They also have a 'cross-layer negotiation' design, allowing the decision of *which* minion is used to be made on a per-connection basis.

Their general concept of splitting the Transport layer up is (in my view) a very useful one. Even though the IETF passed on codifying it into the literature, it's a very useful pattern in designing protocols - layer an endpoint layer, a congestion layer, an isolation layer, and a semantic layer.

By using UDP as the endpoint layer, you can (potentially) avoid some of the problems MPTCP had to engage in heroics to deal with.

By using (say) a portless variant of DCCP on top of that as congestion control, you have something that could still be within the kernel (after all, there are various protocols implemented in the kernel that go on top of UDP or TCP) but because of the benefits of using UDP as the endpoint layer you don't have the horrific barriers to deployment DCCP proper ran aground on. Plus, this makes per-link congestion control feasible without breaking end-to-end reliability - which has been a significant problem due to the absolutely atrocious interaction of TCP, wireless LAN, and multiple layers of retransmission latency.

Layering the isolation layer here (rather than on top of the semantics) is genius. Aside from that it means you only need a design for datagrams, and not streams (so you can use DTLS regardless of the application-level semantics), it also means that the (intelligent in braindead ways) middleboxes only know information that they might have a valid reason to touch. Beyond that, they basically get told to push off.

Finally, you can build just about any semantics you like on top, by using the resulting building block of secure, point-to-point, CC-friendly unreliable datagram channels.

Multipath TCP: an overview

christophpaasch — 2013-04-04T06:24:22+00:00

It returns the info from the initial TCP subflow.

So, even if the host has moved away (e.g., moved away from its WiFi access-point and MPTCP switched to 3G), the syscalls still return the old address of the WiFi interface.

Multipath TCP: an overview

kevinm — 2013-04-04T04:04:57+00:00

What do the getsockname() and getpeername() APIs do on a MPTCP connection?

Fundamentally modifications to TCP are necessary

giraffedata — 2013-04-02T16:15:07+00:00

An IP address conflates two things that should be separate: location (where) and identity (who). The latter should be in layer 4.

I'm with you on there being a need to separate location and identity, and to do it by layers, but it looks like all part of layer 3 (network layer) to me. One should be able to direct any IP packet to an identity, not just a TCP stream.

That's just speaking of ideals, of course. I'm not saying that's the direction we should be going now.

There is a layering issue between TCP and IP in that the TCP port address shouldn't be in the IP packet header. I wouldn't want to confuse that with this.

Multipath TCP: an overview

wookey — 2013-04-02T14:09:26+00:00

That did grate rather.

But one has to make allowances for Americans, even ones as well-adjusted as our esteemed editor.

Fundamentally modifications to TCP are necessary

paulj — 2013-04-02T12:17:11+00:00

I don't think IP conflates these. It just wasn't an issue on the horizon in the design of IP. That said, the original designers of IP did envision that further addressing schemes (e.g the "associative addressing" Cerf & Kahn referred to in their '74 paper) might be layered over TCP.

Sadly, the designers and implementors that followed chose to prioritise short-term performance concerns over the long-term flexibility of IP. It became effectively impossible to insert new protocols between IP and TCP (in the sense of it having an IP protocol number != TCP).

It might still be possible to insert an identity layer. The lower 64 bits of the IPv6 address could be used for this. Unfortunately though:

a) There's no guarantee IPv6 will succeed

b) Even if it does, there are (as usual) short-sighted people out there pushing to abolish the split in IPv6 addresses between network and host ID portions ("Why should we limit the hierarchical network space to 64 bits? Why do we need 64 bits for a host?").

So we shall see if this is possible. Otherwise, it has to be done in TCP.

Identity is a very complex issue. It can mean different things to different people/processes at different times. Think about the identity for an email address, or an SSL cert, or a web page - you can surely think of many different scenarios and distinct issues for each. At the network layer, it is very hard to come up with a universal meaning of identity other than "the location in the network". Identity is an issue that really can't be solved at the network layer, other than equating it with location. Even a shim protocol between TCP and IP can't really say more than "these 2 network locations appear to be controlled by the same entity, around this time".

Anyway... :)

Fundamentally modifications to TCP are necessary

marcH — 2013-04-02T07:25:08+00:00

An IP address conflates two things that should be separate: location (where) and identity (who). The latter should be in layer 4.

You don't ask the entire Post office to update ZIP codes when you move house.

It's not me saying it but the whole research community. Look at M-TCP, HIP, GTP (GPRS tunnelling), dynamic load balancing,... they are all try to somehow retrofit this separation in a backward, half-compatible way. Because it's too late it tends not to be pretty.

"As simple as possible, but not simpler" - too simple this time.

Fundamentally modifications to TCP are necessary

giraffedata — 2013-04-02T02:44:24+00:00

Layering doesn't require TCP to have endpoint addressing that doesn't involve IP addresses, because it isn't that kind of layer.

Now if TCP were a network topology layer, then it would need its own addressing and could easily do the kind of routing we're talking about. But I would not expect anyone to have designed TCP that way, because it would be redundant. The basic architecture of the Internet says routing packets to whatever ephemeral link happens to be up now is what the IP layer is for. A TCP driver is supposed to be blissfully ignorant of paths and concentrate on turning a blizzard of packets into an ordered, ungranular, reliable stream.

The problem as I see it is just that IP hasn't evolved in a way that its routing protocols are sufficent for the needs of millions of handsets hopping from one wireless network to another. Considering that the original routing protocols were hardcoded files on each node, that's not surprising.

Multipath TCP: an overview

martinfick — 2013-04-01T23:31:59+00:00

Are you sure? I suspect most firewalls, load-balancers, and other middle boxes which do port forwarding or filtering are filtering by protocol, so they will filter out UDP if only the TCP port is open/forwarded.

Fundamentally modifications to TCP are necessary

dlang — 2013-04-01T23:14:10+00:00

In theory you could propagate routes for every host to the entire Internet and all hosts could move dynamically.

In practice it just doesn't scale, the overhead of letting the routing tables get that large just doesn't work at acceptable speeds.

Someday this may change.

It has changed for phones. It used to be that the area code and first three digits of the number routed you to specific buildings and then the last digits routed you out from that. so while there are still large chunks of landlines that mostly follow this model, phone number portability and cell phones make it so that any phone number can appear anywhere on the network.

now, the phone system only needs to find this to setup the conversation, not for each packet. This is the "smart network vs dumb network" discussion from above.

Given that the "smart network" of the phone system now tends to run on top of the "dumb network" of the Internet, I think it's pretty clear that the Internet has shows itself to be far superior

If you think about it, the Internet already has the layer of indirection you are talking about, DNS. The problem is that looking things up in DNS is far too slow and updates far too infrequently for it to be used in routing decisions for every packet.

Fundamentally modifications to TCP are necessary

marcH — 2013-04-01T22:41:32+00:00

> So, if you want to make use of multiple, disparate, network attachments, you need to do this above IP. Either you must modify TCP, or you must insert another protocol between IP and TCP (e.g. a shim, see shim6).

The original sin is actually quite simple: it's the lack of layering in TCP/IP. TCP should not "steal" and rely on IP addresses directly. Just like every other layer, the TCP layer should have a its own "host address" and some indirection logic to resolve it to IP layer address(es). This indirection logic would be the most natural place to implement mobility and all the other features Multipath TCP is offering. I'm sure there are a few good research papers explaining this.

Now of course it's way too late for such a dramatic change but keeping in mind the core shortcoming and the very theoretical but "correct" design helps understand all the numerous and more complex workarounds that keep being offered. I find.

Multipath TCP: an overview

marcH — 2013-04-01T22:11:57+00:00

Just FYI: my home Wifi is routed to... a 3G, metered connection.

Far from ideal, but less and less unusual:

https://www.google.ie/search?q=3g+hotspot

Multipath TCP: an overview

marcH — 2013-04-01T22:03:24+00:00

> ... and it doesn't exist yet. ;)

I know at least Skype and AnyConnect which already have _some_ of the abilities of my fictional libMTR++.

However they are only designed for internal use, i.e.: working around network limitations and barely reporting them.

Multipath TCP: an overview

marcH — 2013-04-01T21:57:34+00:00

To be honest most "middle boxes" I've seen rejecting UDP were just... web proxies and thus not letting *anything* go through.

Of course any intermediate kind of network crippling is possible in theory.

Multipath TCP: an overview

christophpaasch — 2013-04-01T21:08:38+00:00

The protocol allows to not use 3G for data-traffic with the "backup-flag" (MP_PRIO-option in RFC 6824).

Or, it is also possible to simply not open up a subflow across 3G and only create it after the phone lost the WiFi connection. MPTCP supports break-before-make scenarios.

It is simply a question of instructing the kernel to do one or the other through some kind of configuration interface (may it be sockopts, sysctl,...)

Multipath TCP: an overview

cesarb — 2013-04-01T20:55:39+00:00

It is not "more traffic on wifi". If your 3G is metered and you are connected to wifi, _all_ traffic should be on wifi.

Multipath TCP: an overview

intgr — 2013-04-01T15:46:10+00:00

> End-station MTU discovery has been proven to be fragile, and often does not work, due to stupid intermediate nodes blocking the required signalling

Sounds like PMTU discovery has the same enemies: "smart" middleboxes (NAT) and enterprise admins (firewalls).

While I agree that protocol designers should have seen it coming, I still think the larger part of the blame lies with device vendors not implementing existing protocols to the specification and not releasing updates even when those problems are pointed out to them -- basically holding the whole industry hostage because the cost is externalized.

Which brings me to a crazy idea -- perhaps we need an "Acid test of consumer routers" -- following the model of acidtests.org, which pressured several browser vendors to improve on their standards conformance, because every savvy user could easily see how broken their choice was. Has anyone attempted something like this already?

Multipath TCP: an overview

paulj — 2013-04-01T13:44:11+00:00

Note though "the network should be as simple as it needs to be, but no simpler." - to paraphrase someone.

While complexity generally should be pushed to the end-stations and kept out of the network, as much as possible, that does not mean *all* complexity should be kept out of the network. There are certain basic functions that *should* be kept in the network, along with whatever complexity is required to support those functions.

E.g. the original designers of TCPIP believed it was very important for the network to abstract away differences in MTUs. They argued that otherwise it would be difficult for end-stations to reliably work-around differences in MTU, so that it would become difficult for the network to make use of larger MTUs. This would block the use of techniques such encapsulation, extra-addressing, etc., by end-stations to solve problems such as mobility, etc. For this reason they made TCPIP support fragmentation at the network level.

Sadly, later implementors and stewards of IP concluded differently. They moved to deprecate fragmentation from IPv4, and excluded it completely from IPv6. Instead, it was left to the end-stations to negotiate the correct MTU. Unfortunately however the original designers of IP have been proven correct. End-station MTU discovery has been proven to be fragile, and often does not work, due to stupid intermediate nodes blocking the required signalling (which is out of band). At the best of times it probably adds more latency than network level fragmentation.

As a result, the Internet is crippled with a 1500 MTU. :( The success of IPv6 would likely set this in stone.

See my blog: http://pjakma.wordpress.com/2011/06/28/cerf-and-kahn-on-w...

Multipath TCP: an overview

intgr — 2013-04-01T13:24:17+00:00

> Pushing these problems out to the edges of the network - ideally to the end-stations - avoids the need for complex, fragile networks

Sorry to hijack this thread, but I wanted to stress this point. This is exactly what's wrong with NAT: it builds too much smarts into devices whose real purpose is just juggling packets around.

And you see the pains every time someone wants to add an extension to TCP (like MPTCP, ECN and others) or introduce a new transport protocol (SCTP, DCCP). In fact, even application developers suffer and have to come up with clever workarounds (STUN, TURN) or less clever (UPnP). People have to spend heroic amounts of time to work through all the potential issues these "smart" middleboxes introduce, with every vendor bringing their own quirks and bugs on the table.

In the end, innovation suffers and robustness of networks in general suffers due to all the workarounds that have to be employed.

It drives me nuts when people deploy IPv6 and still don't get it, "we're an enterprise, we need NAT". No you don't, you need a firewall.

</rant>

Multipath TCP: an overview

dlang — 2013-03-31T21:20:53+00:00

But the server support could be completely in the OS, transparent to the application couldn't it?

Multipath TCP: an overview

martinfick — 2013-03-31T16:35:39+00:00

Well it isn't completely transparent, you still need MPTCP support on the server.

Multipath TCP: an overview

paulj — 2013-03-31T07:43:36+00:00

There are a lot of trade-offs in multi-homing, because the different available connections may have different properties, e.g. in terms of cost, latency, throughput. The appropriate choice of trade-offs may well differ at different times, and/or for different users, and/or for different applications.

Trying to make the network handle choosing these trade-offs would require either building lots of complex logic into it, to try dynamically second-guess the needs of each user, and/or lots of signalling, to allow the user to instruct the network what it wants (with attendant state overheads in the network).

Pushing these problems out to the edges of the network - ideally to the end-stations - avoids the need for complex, fragile networks. Complex networks also tend to have an ossifying effect on development of new applications. Have a read of "The rise of the stupid network", mirrored at, e.g,: http://www.rageboy.com/stupidnet.html .

Basically, the network should be as simple as it needs to be, but no simpler. ;)

Fundamentally modifications to TCP are necessary

giraffedata — 2013-03-31T01:01:30+00:00

Thanks; that explains it. I forgot that every router along the way would be independently choosing a route to the destination.

Fundamentally modifications to TCP are necessary

paulj — 2013-03-30T22:30:02+00:00

So does the intermediary node with the "4.3.2.1 is also a destination for 1.2.3.4" routing entry also send copies of the packet to 4.3.2.1? Or just to 1? If copies, what happens if the copies also go through intermediaries that create copies?

Is this more efficient than just letting the end-stations create additional flows between any additional IP addresses? If yes, how?

Multipath TCP: an overview

dlang — 2013-03-30T22:02:21+00:00

remember that the server application/port can remain the same if the client has multiple connections. It's only in the case where the server has multiple connections that the it needs multiple IP addresses, and in that case I don't see any way around the issue.

I also don't expect servers to expose multiple connections very frequently, they don't need to and if they are serving substantial numbers of clients, they probably don't want to have any one client be able to use a substantial amount of bandwidth.

I see two categories where multipath TCP makes lots of sense

1. multi-homed clients (wifi + cell, DSL + cable + cell backup, etc)

Here, only the client has multiple IP addresses

2. huge file transfers within one organization on specially setup paths

Here both the client and server may have multiple IP addresses, and even with just a single IP at both ends, they may want to use many subconnections to be more resistant to the effects of packet loss.

Multipath TCP: an overview

roblucid — 2013-03-30T21:12:59+00:00

True, but wouldn't it be simpler and cleaner to have this policy as part of a virtual router in the device which actually kept the application IP/port the same?

Why should applications, worry & account for multi-homing an so on? You still end up adminstering firewalls to set a policy at a more feasible level.

Fundamentally modifications to TCP are necessary

dlang — 2013-03-30T20:42:29+00:00

Clarifying one point that I skipped

Routing on the Internet is done one hop at a time.

your default routers don't talk directly to the server, they talk to other routers that talk to other routers.... that talk to the server.

It's common to have 10-20 routers in the path for some connections (do a traceroute to the server to see the routers that your traffic to the server goes through, and keep in mind that the traffic from the server back to you may go through a different series of routers)

Fundamentally modifications to TCP are necessary

dlang — 2013-03-30T20:39:39+00:00

The problem is that when the other side needs to reply to you, it sends packets out with a destination of your 1.2.3.4 IP address

every router along the way will have one best path to get to 1.2.3.4, and so all the traffic will go down that path.

It's this best path priority routing that lets the Internet work as well as it does, but it means that if you have two completely different connections, and you want to use them both, you have to split the traffic between two different IP addresses, one for each connection, so that the traffic to you will get split between those connections.

The nice thing with multipath TCP is that it can do this under the covers (in a library or in the kernel).

With multipath TCP widely available, you could even have this implemented in a router. The router would have multiple connections, and proxy the TCP connection into multipath TCP, utilizing the multiple connections that it has completely transparently to the endpoint machine.

Fundamentally modifications to TCP are necessary

giraffedata — 2013-03-30T20:25:53+00:00

There is no way in IP to treat multiple IP addresses as one logical IP [address]

The MPTCP alternative I'm thinking about wouldn't do that. The client would use one IP address and the overall IP network would route via either the 3G or wifi network based on its own policies.

For example, the handset has IP address 1.2.3.4 on the wifi network and 4.3.2.1 on the 3G network. All outgoing packets from the handset bear source IP address 1.2.3.4. All packets the server sends to the handset bear destination IP address 1.2.3.4. The server's routing table (or the routing table in its default router) shows that in addition to the normal routes, 4.3.2.1 is a route to 1.2.3.4.

Does that work?

Fundamentally modifications to TCP are necessary

paulj — 2013-03-30T10:26:38+00:00

At the IP level, nodes are identified by IP addresses and the IP network routes between addresses. This may mean that different packets take different routes, even though they travel from the same source to the same destination IP address. TCP, indeed, generally knows nothing about this.

However, some hosts have multiple network connections. E.g., a common case today is having both a 3G and a wifi connection, each to a completely different network.

There is no way in IP to treat multiple IP addresses as one logical IP, and have IP handle the routing between them. Well, there are some ways (Mobile IP). Unfortunately these solutions rely on tunnelling packets, and the lack of foresight of IP designers and network operators in crippling the Internet to a fixed MTU means tunnelling is not generally practical with the Internet today. (And, sadly, IPv6 makes things worse in this regard :( ).

So, if you want to make use of multiple, disparate, network attachments, you need to do this above IP. Either you must modify TCP, or you must insert another protocol between IP and TCP (e.g. a shim, see shim6). Unfortunately, again down to lack of foresight and (most of all) stupidity of network operators, new IP protocols are not generally practical with the Internet today - too many idiotic middle-boxes will filter it out. So at the TCP level it has to be.

(There's a recurring theme here. If IPv6 fails - and it might - the next generation IP protocol will have to have a fixed preamble that looks exactly like the beginning of an IPv4 + TCP + HTTP packet. Sigh).

Multipath TCP: an overview

gdt — 2013-03-30T08:37:04+00:00

There are no end of those protocols. I work in academic networking and I've experienced my fair share of them. Usually they are to compensate for poor single-flow performance, a problem Linux generally doesn't have if you simply give enough memory to the autotuned TCP buffers.

MPTCP is the idea implemented well. That's a good thing, as it gives us a place to point users who think they might do better implementing this idea themselves.

Multipath TCP: an overview

gdt — 2013-03-30T08:28:12+00:00

That wouldn't be a good idea. Almost all IP-using protocols maintain an estimate of the "path round-trip time". Using multiple paths at the IP layer would lead to high variability in the estimate of the round trip time. Thus those protocols wouldn't be able to optimise throughput, flow control or buffer usage.

Fundamentally modifications to TCP are necessary

giraffedata — 2013-03-29T20:22:25+00:00

However if you want to use more than one of your available paths at once TCP must be changed.

I believe that is the question that some of us have. Why does TCP have to change? The IP I learned long ago has multipath routing built in and the TCP driver couldn't stop it if it wanted to. If TCP has to be involved in routing, then is part of MPTCP giving the layer above IP the ability to constrain how IP routes the packets?

I don't know anything about route advertisement protocols, but assuming there isn't one today that lets the handset merely tell the server that it can reach 1.2.3.4 via 4.3.2.1 (two of the handset's IP addresses), would it be harder to add that than to add the ability for the server to recognize two TCP destinations as the same place (MPTCP)?

Multipath TCP: an overview

paulj — 2013-03-29T13:51:45+00:00

I don't quite remember the details, but I sat through a presentation by someone involved with MP-TCP. They were a mathematician who'd been brought on to work on the congestion control side and formulate a mechanism that would work and be fair to both MP-TCP and plain TCP flows.

Not vouching for their success in this, but they do appear to have thought about congestion control.

Multipath TCP: an overview

christophpaasch — 2013-03-28T17:44:43+00:00

Does your friend you are mentioning may share the references of his claims?

MPTCP has no issues with head-of-line blocking, because the receive-window is not on a per-subflow level but rather on the data-level [1].

I don't see, how MPTCP could cause extreme congestion, because the coupled congestion control makes sure that MPTCP is fair to other flows across shared bottlenecks [2].

Cheers,
Christoph

[1] https://datatracker.ietf.org/doc/rfc6824/
[2] https://datatracker.ietf.org/doc/rfc6356/

Multipath TCP: an overview

jnareb — 2013-03-28T17:31:22+00:00

But mosh requires that you have mosh-server installed on remote server, isn't it?

MPTCP is to be completely transparent.

Multipath TCP: an overview

kov — 2013-03-28T17:13:57+00:00

Doing that transparently by default may be a problem for most mobile use cases. Consider a smartphone with a metered cellular connection and a WiFi connection. Most applications would probably not want to download big chunks of data from the cellular connection, even though it could be used to speed things up. That could be controlled at a system level, I imagine, but you'd either need a way of distinguishing at the system-level which applications should be allowed to use multi-path and which not. Or (what I think would be more likely) you'd disable it by default and have applications which really want to use it request it explicitly.

Multipath TCP: an overview

raven667 — 2013-03-28T16:53:06+00:00

I don't have the background to comment on this intelligently but I showed this to someone with more experience and they pointed out that they had seen many such research projects in the '80s but that they always failed because they were unable to account for all the complex interactions between the multiple streams and balancing data across them. Essentially they get bogged down in new and impressive corner cases where traffic stops or where it causes extreme congestion. This may be something that is only useful on a local managed network and not across the open Internet.

Fundamentally modifications to TCP are necessary

Cyberax — 2013-03-28T16:33:14+00:00

LISP looks more and more hopeless. They are still keeping their head in a very dark place - lots of WG members are still thinking about LISP as a replacement for BGP for stub networks.