The container orchestrator landscape

Speaking as a Google SRE, looking at k8s "from the other side" as it were, my main problem with it is actually not complexity, but terminology. It's *almost* 1:1 equivalent to Borg, except everything has a different name and it's differently opinionated about certain things. For example, in k8s you have pods which own or manage containers, and then you have a ReplicaSet which owns or manages one or more pods. This is all very sensible and reasonable. In Borg, we would say that you have alloc instances (pods) which own or manage tasks (containers), and then you have an alloc (ReplicaSet) which describes one or more alloc instances. The problem is that Borg also describes the set of tasks as a "job," and this has no direct equivalent in k8s.* Worse, Borg lets you dispense with the alloc altogether, so you can just have a "naked" job that consists of a bunch of tasks (containers) with no pod-like abstraction over them. On the one hand, this means that we don't have to configure the alloc if we don't want or need to. On the other hand, it means that we have two ways of doing things. But the real problem is that, when we're talking about Borg informally, we often say "job" instead of "job or alloc" - which is the one Borg term that doesn't really have a clean equivalent in k8s.

The reverse also happens. Borg doesn't let you submit individual alloc instances (pods) or tasks (containers) without wrapping them up in an alloc (ReplicaSet) or job (see above), so if you just want one copy of something, you have to give Borg a template and say "make one copy of it" instead of submitting the individual object directly, and so in practice we mostly speak of "jobs and allocs" rather than "tasks and alloc instances." But in k8s, you can configure one pod at a time if you really want to.

(For more specifics on how Borg works, read the paper: https://research.google/pubs/pub43438/)

* k8s also defines something called a "job," but it's a completely different thing, not relevant here.