A two-part series on LXC networking (Flockport Labs)

> Why do people have this fascination with L2 networking

Because it requires less state on the endpoint server host, there are standard discovery protocols to find layer2 neighbors and safely pass traffic on a layer2 path without loops, you can easily move an IP between nodes and even sites and the network will almost instantly relearn the path without the application needing to do anything or even lose TCP connections. Learning a different IP for a service requires much more interaction from higher layers in the stack such as the application and only makes sense if you think you should be writing a bunch of new code rather than just using the well worn working systems that already exist.

> So maybe even if they move the container to an other machine they need to keep the original IP-address ?

Yes, absolutely, this needs to work reliably not only across a single cluster but across different datacenters as well for maximum effect.

I hate to say it, but most developers actually don't understand networking.

And the problem with using some discovery process is that the discovery is expensive, and so the result is always going to be cached somewhere (and frequently it ends up being cached at multiple levels), even if it's not cached, there's a race condition between updating the discovery source of truth and moving the service.

So moving the service and keeping it on the same IP address is a very practical win in terms of simplicity and reliability.

"Is it because infrastructure people think developers don't understand networking and want to simplify it by using L2-networking ? Or think developers are lazy"

No (I for example don't) but they do have a habit of creating things that want to live in L2 land. eg Elasticsearch systems automatically cluster if you rub them together on the same collision domain.

Anyway, I don't think this is a bad thing but it's nice to have the flexibility to do either or both.

"Stretched trombone" anyone 8)

Slightly repeating points made earlier: doing discovery of other services implies some sort of database that keeps track of where everything is - dynamic DNS at a minimum. That isn't necessarily the problem. The problem comes much more when you try to scale to lots of machines with different providers, in geographically very different locations, dynamic provisioning of machines, changing IPs etc, and you want: i) all private traffic between containers on these machines kept private (i.e. encrypted); ii) a few public IPs that actually represent the service being offered but are internally routed to a changing set of machines; iii) the ability to scale the various different subservices up and down without manually editing configuration files. Really, at the end of the day it just comes down to another layer of indirection and abstraction: you can hide some of the changing complexity of the exterior and avoid those messy details leaking into the configuration of your services. It gets you decoupled from the specifics of your deployment.

Also, sometimes you need features that the "real" network isn't providing. For example, with weave, you end up with an overlay network that supports multicast even if the "real" network doesn't, copes with different MTUs in different parts of the "real" network very well, does encryption at the network layer so for some applications you won't need to configure every service with its own crypto, does multi-hop routing, and copes with failure very well.

There are numerous advantages to using L2 networking if your software switch properly understands datacenter bridging.

Of course since very few software engineers understand advanced switch theory it is not surprising that most softswitches share that lack of understanding.