The container orchestrator landscape

Posted Aug 23, 2022 23:19 UTC (Tue) by bartoc (guest, #124262)
Parent article: The container orchestrator landscape

systemd also has a built-in container runtime and (sorta) orchestrator, which is a lot simpler than k8s and makes it easier to have tasks that are just normal systemd units.

One thing that always really annoyed me about k8s is the whole networking stack and networking requirements. My servers have real ipv6 addresses, that are routable from everywhere and I really, really do not want to deal with some insane BGP overlay. Each host can good and well get (at least) a /60 that can be further subdivided for each container.

The whole process just felt like figuring out exactly how the k8s people had reimplemented any number of existing utilities. It all gave me the impression the whole thing was abstraction for abstraction's sake. I feel the same way about stuff like ansible, so maybe I just really care about what code is actually executing on my servers more than most people.

I found Hashicorp's offerings (in general tbh, not just Nomad) to be a lot of shiny websites on top of very basic tools that ended up adding relatively little value compared to just using whatever it was they were abstracting over.

The container orchestrator landscape

Posted Aug 24, 2022 1:10 UTC (Wed) by jordan (subscriber, #110573) [Link] (3 responses)

I find myself mourning fleet with some regularity.

The container orchestrator landscape

Posted Aug 26, 2022 14:37 UTC (Fri) by mdaverde (guest, #151459) [Link] (2 responses)

With systemd-nspawn, systemd-sysext & portable services, it really does feel like there's a space to be explored for a new systemd-based orchestrator.

I believe Facebook/Meta's infra is heavily systemd-based with their in-house Twine cluster manager but I don't know how much of the internals are available.

The container orchestrator landscape

Posted Aug 26, 2022 15:20 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

How close to fleet[1] do you think this might turn out to be?

[1]https://github.com/coreos/fleet

The container orchestrator landscape

Posted Aug 26, 2022 15:46 UTC (Fri) by paulj (subscriber, #341) [Link]

Systemd would be used for setting up the containers / cgroups yes. But, it's a very small, localised implementation detail really.

Twine (the external name, but more often called 'Tupperware' - probably the better name to use in searches) would be hard-to-impossible to make available to non-FB use, and probably mostly pointless. It is very heavily integrated in with lots of other Facebook infrastructure, from the CI system, to the automated fleet roll-out system of services, to the service discovery and routing system, etc., etc.

The container orchestrator landscape

Posted Aug 24, 2022 1:28 UTC (Wed) by rjones (subscriber, #159862) [Link] (6 responses)

> The whole process just felt like figuring out exactly how the k8s people had reimplemented any number of existing utilities. It all gave me the impression the whole thing was abstraction for abstraction's sake.

Kubernetes and it's networking stack complexity is the result of the original target for these sorts of clusters.

The idea is that you needed to have a way for Kubernetes to easily adapt to a wide variety of different cloud architectures. The people that are running them don't have control over the addresses they get, addresses are very expensive, and they don't have control over any of the network infrastructure. Ipv6 isn't even close to a option for most of these types of setup.

So it makes a lot of sense to take advantage of Tunnelling over TCP for the internal networking. This way it works completely independent of any physical or logical network configuration the kubernetes might be hosted on. You can even make it work between multiple cloud providers if you want.

> One thing that always really annoyed me about k8s is the whole networking stack and networking requirements. My servers have real ipv6 addresses, that are routable from everywhere and I really, really do not want to deal with some insane BGP overlay. Each host can good and well get (at least) a /60 that can be further subdivided for each container.

You don't have to use the tunneling network approach if you want. For example if you have physical servers with multiple network ports you can just use those separate lans instead.

Generally speaking you'll want to have 3 LANs. One for the pod network, one for the service network, and one for external network. More sophisticated setups might want to have a dedicated network for storage on top of that, and I am sure that people can find uses for even more then that.

I don't know how mature K8s IPv6 support is nowadays, but I can see why that would be preferable.

> It all gave me the impression the whole thing was abstraction for abstraction's sake. I feel the same way about stuff like ansible, so maybe I just really care about what code is actually executing on my servers more than most people.

It could be that a lot of people are not in a position to micro-manage things on that level and must depend on the expertise of other people to accomplish things in a reasonable manner.

The container orchestrator landscape

Posted Aug 24, 2022 6:55 UTC (Wed) by dw (subscriber, #12017) [Link] (2 responses)

I don't mean to keep jumping into your replies, but I feel I can see what stage in the cycle you're at with Kubernetes and it's probably worth pointing out something that might not immediately be obvious: in all the rush to absorb the design complexity of the system, it's very easy to forget that there are numerous ways to achieve the flexibility it offers, and the variants Kubernetes chose to bake in are only one instantiation, and IMHO usually far from the right one.

Take as a simple example the network abstraction, it's maybe 20%+ of the the whole Kubernetes conceptual overhead. K8 more or less mandates some kind of mapping at the IP and naming layers, so you usually have at a minimum some variation of a custom DNS server and a few hundred ip/nf/xdp rules or whatnot to implement routing. Docker's solution to the same problem was simply a convention for dumping network addresses into environment variables. No custom DNS, no networking nonsense.

It's one of a thousand baked-in choices made in k8s that really didn't need to be that way. The design itself is bad.

No conversation of Kubernetes complexity is complete without mention of their obsolescent-by-design approach to API contracts. We've just entered a period where Ingresses went from marked beta, to stable, to about-to-be-deprecated by gateways. How many million lines of YAML toil across all k8s users needed trivial updates when the interface became stable, and how many million more will be wasted by the time gateways are fashionable? How long will gateways survive? That's a meta-design problem, and a huge red flag. Once you see it in a team you can expect it time and time again. Not only is it overcomplicated by design, it's also quicksand, and nothing you build on it can be expected to have any permanence.

The container orchestrator landscape

Posted Aug 25, 2022 18:44 UTC (Thu) by Depereo (guest, #104565) [Link]

Having lived some of the issues with maintaining an inhouse distribution of kubernetes (non-certified), I would agree with the shifting sands analogy.

It's quite frustrating to go from the infrastructure world of VMs, which are extremely backwards and forwards compatible, to kubernetes, where the necessary major upgrades every few months will break several deployment pipelines, or deprecate APIs, or do various other things that require your clients to endlessly scramble to 'keep up'. And you're right, it's usually to do with network requirements (or sometimes storage which is somewhat related to network design anyway).

Committing to deployment on k8s is a commitment to a much higher degree of required ongoing updates for and probably unexpected issues with deployment than I'm used to with for example virtual machine orchestration. Unless you're at a certain and very large size I have come to think it's not worth it at all.

The container orchestrator landscape

Posted Aug 26, 2022 1:38 UTC (Fri) by thockin (guest, #158217) [Link]

> K8 more or less mandates some kind of mapping at the IP and naming layers, so you usually have at a minimum some variation of a custom DNS server and a few hundred ip/nf/xdp rules or whatnot to implement routing. Docker's solution to the same problem was simply a convention for dumping network addresses into environment variables. No custom DNS, no networking nonsense.

Last I looked in depth, docker had a DNS server built in, too. Publishing IPs via env vars is a TERRIBLE solution for a bunch of reasons. DNS is better, but still has a lot of historical problems (and yeah, kube sort of tickles it wrong sometimes). DNS + VIP is much better, which is what k8s implements. Perfect? No. But pretty functional.

> No conversation of Kubernetes complexity is complete without mention of their obsolescent-by-design approach to API contracts. We've just entered a period where Ingresses went from marked beta, to stable, to about-to-be-deprecated by gateways.

I know of no plan to formally deprecate Ingress, and I would be the approver of that, so....FUD. Also, deprecate != EOL. We have made a public statement that we have NO PLANS to remove GA APIs. Perhaps some future circumstance could cause us to re-evaluate that, but for now, no.

> How many million lines of YAML toil across all k8s users needed trivial updates when the interface became stable

The long-beta of Ingress is a charge I will accept. That sucked and we have taken action to prevent that from ever happening again.

> and how many million more will be wasted by the time gateways are fashionable?

Nobody HAS to adopt gateway, but hopefully they will want to. It's a much more functional API than Ingress.

> How long will gateways survive? That's a meta-design problem, and a huge red flag.

APIs are forever. That's how long. Once it hits GA, we will keep supporting it. No FUD required.

> nothing you build on it can be expected to have any permanence.

We have a WHOLE LOT of evidence to the contrary. If you have specific issues, I'd love to hear them.

I don't claim kubernetes is perfect or fits every need, but you seem to have had a bad experience that is not quite the norm.

The container orchestrator landscape

Posted Aug 24, 2022 7:58 UTC (Wed) by bartoc (guest, #124262) [Link] (1 responses)

> The idea is that you needed to have a way for Kubernetes to easily adapt to a wide variety of different cloud architectures. The people that are running them don't have control over the addresses they get, addresses are very expensive, and they don't have control over any of the network infrastructure. Ipv6 isn't even close to a option for most of these types of setup.

Well, I don't care about any cloud architectures except mine :). More seriously though the people running clouds absolutely do have control over the addresses they get! And tunneling works just as well if you want to provide access to the ipv6 internet on container hosts that only have ipv4, except in that situation you have some hope of getting rid of the tunnels once you no longer need ipv4.

> Generally speaking you'll want to have 3 LANs. One for the pod network, one for the service network, and one for external network. More sophisticated setups might want to have a dedicated network for storage on top of that, and I am sure that people can find uses for even more then that.

IMO this is _nuts_, I want _ONE_ network and I want that network to be the internet (with stateful firewalls, obviously).

The container orchestrator landscape

Posted Aug 26, 2022 1:41 UTC (Fri) by thockin (guest, #158217) [Link]

This is FUD. In general you need one flexible network or one node network plus a cluster-centric overlay system.

You DO need to think about addressing and how you want you cluster(s) to interact with everything else.

The container orchestrator landscape

Posted Aug 26, 2022 1:26 UTC (Fri) by thockin (guest, #158217) [Link]

> I don't know how mature K8s IPv6 support is nowadays

Should work fine.

The container orchestrator landscape

Posted Aug 26, 2022 1:25 UTC (Fri) by thockin (guest, #158217) [Link] (1 responses)

You don't need an overlay if you already have a decent sized range of IPs per node. Just use those IPs.

I don't know where the idea that you NEED an overlay comes from. If you have IPs, just use those. That's what it was designed for.

The container orchestrator landscape

Posted Aug 26, 2022 23:18 UTC (Fri) by bartoc (guest, #124262) [Link]

When I was setting this up I found the supporting infrastructure for just using real ipv6 was really poor. You at least need something to read out whatever prefix you got from DHCP-PD and set up forwarding rules, and either set addresses on the VMs, or start a copy of radvd facing the containers and tell it about said prefix.

Or you could use a virtual switch, that would probably "just work"