The container orchestrator landscape [LWN.net]

The container orchestrator landscape

Posted Aug 24, 2022 0:26 UTC (Wed) by denton (guest, #159595)
Parent article: The container orchestrator landscape

Does Nomad enable you to extend the control plane with your own custom types?

One thing that k8s got right was it gave the ability for users to define new resource types, via CustomResourceDefinitions (CRDs). So for example, if you wanted a Postgres database in your k8s cluster, you could install a CRD + Postgres Controller and have access to that new API. It's led to a large number of Operators that can enable advanced functionality in the cluster, without the user needing to understand how they work. This is similar to managed services on cloud providers, like Aurora or RDS in AWS.

I'm wondering if nomad has a similar functionality?

The container orchestrator landscape

Posted Aug 24, 2022 17:09 UTC (Wed) by schmichael (guest, #160476) [Link]

> I'm wondering if nomad has a similar functionality?

No, Nomad has chosen not to implement CRDs/Controllers/Operators as seen in Kubernetes. Many users use the Nomad API to build their own service control planes, and the Nomad Autoscaler - https://github.com/hashicorp/nomad-autoscaler/ - is an example of a generic version of this: it's a completely external project and service that runs in your Nomad cluster to provide autoscaling of your other Nomad managed services and their infrastructure. Projects like Patroni also work with Nomad, so similar projects to controllers due exist: https://github.com/ccakes/nomad-pgsql-patroni

The reason (pros) for this decision is largely that it lets Nomad focus on core scheduling problems. Many of our users build a platform on top of Nomad and appreciate the clear distinction between Nomad placing workloads and their higher level platform tooling managing the specific orchestration needs of their systems using Nomad's APIs. This should feel similar to the programming principles of encapsulation and composition.

The cons we've observed are: (1) you likely have to manage state for your control plane ... somewhere ... this makes it difficult to write generic open source controllers, and (2) your API will be distinct from Nomad's and require its own security, discovery, UI, etc.

I don't want to diminish the pain of forcing our users to solve those themselves. I could absolutely see Nomad gaining CRD-like capabilities someday, but in the short term you should plan on having to manage controller state and APIs yourself.

Disclaimer: I am the HashiCorp Nomad Engineering Team Lead