The container orchestrator landscape [LWN.net]

The container orchestrator landscape

Posted Aug 30, 2022 12:57 UTC (Tue) by kleptog (subscriber, #1183)
In reply to: The container orchestrator landscape by jordan
Parent article: The container orchestrator landscape

I've used a lot of Docker in production over the years, most of it with Swarm and while it's a lot better than it was there seem to be some core structural issues which make it unreliable, at least for us. The overlay network corrupts itself often enough (maybe due to having 150+ of them) and while restarting services seems to fix it usually, it's just frustrating to deal with. It's unfortunate, because as a conceptual design Swarm is very nice (well, except for secrets), just the implementation lets it down.

Our next projects will not use Swarm. We've experimented with K8s (on EKS) and you can make it do amazing things. But ECS is really easy to use and basically does what you want, just like Swarm. Nomad is something to look into.

to post comments

The container orchestrator landscape

Posted Aug 31, 2022 20:54 UTC (Wed) by rorycl (guest, #151214) [Link] (4 responses)

> I've used a lot of Docker in production over the years, most of it with Swarm and while it's a lot better than it was there seem to be some core structural issues which make it unreliable

Our SaaS outfit is considering moving from a traditional Linux environment across around a few 10s of servers to use containerisation predominantly to allow a better development experience and testing, particularly for groups of covalent apps, but also to help divorce os and machine maintenance from app deployment.

Having built our business on reading the classic O'Reilly texts to pick up both concepts and implementation details, that combination seems difficult to find in books about orchestration. That is probably the fault of old age, but perhaps the proprietary beginnings of some of these technologies means marketing has confused purpose.

A guru pointed me to the Poulton "Docker Deep Dive" book (I read the May 2020 edition) and the last few chapters are devoted to Swarm. Despite the curious dissimilarities between Compose and Swarm, Swarm seems perfect for our sort of environment and a reasonable translation from our familiar linux setup in production, but where the Swarm manager acts to make hosts act like one large host by utilizing overlay networks on which apps can conveniently be scaled.

For a smallish outfit the benefits of Swarm seems straight-forward. Poulton summarises the situation like this: "Docker Swarm competes directly with Kubernetes -- they both orchestrate containerized applications. While it's true that Kubernetes has more momentum and a more active community and ecosystem, Docker Swarm is an excellent technology and a lot easier to configure and deploy. It's an excellent technology for small to medium businesses and application deployments".

Apart from concerns such as @kleptog's, it isn't clear to me why many more businesses aren't using Swarm.

The container orchestrator landscape

Posted Sep 1, 2022 13:01 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> Apart from concerns such as @kleptog's, it isn't clear to me why many more businesses aren't using Swarm.

Because Swarm is too simplistic. It's kinda like writing in BASIC. It's OK for beginners, but you quickly reach its limits once you start using it seriously.

So people avoid it and jump straight into a more complex solution.

The container orchestrator landscape

Posted Sep 2, 2022 14:47 UTC (Fri) by rorycl (guest, #151214) [Link] (1 responses)

> Because Swarm is too simplistic

...for what, for example?

(I've made a longer comment below, by the way.)

The container orchestrator landscape

Posted Sep 3, 2022 5:00 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Integration with load balancers for traffic ingress and management of stateful resources are among the most problematic.

The container orchestrator landscape

Posted Sep 4, 2022 11:35 UTC (Sun) by kleptog (subscriber, #1183) [Link]

I was pretty enthusiastic about Swarm in the beginning as has a good model for managing containers. But if you're deploying production applications with it you get into a point that others have noted: all the management of permanent resources is done in Swarm itself. And the API is just the Docker API with no authentication.

So if you have a complicated application where the resources (say networks, or services) depend on configuration settings, you have to write a kind of wrapper which reads the configuration and then uses that to update the Swarm configuration. And that configuration is stored separately to Swarm itself. This is annoying and error prone. Because the tool to do this is complex you get the situation where you distribute the tool in a container and then start it up passing the Docker control socket in.

So Swarm can work well if your application is simple enough to deploy via a Docker Compose file. But if you're getting to the point where you're thinking "I need to make a tool to generate the Compose file for me" you're basically at the point where you need something more powerful than Swarm can offer.

That said: for our CI environment, and local testing Swarm works fine. But for production it's too weak. Fortunately for containers, they don't care what tool they're running under.