Monitoring with Prometheus 2.0

Posted Jan 17, 2018 20:44 UTC (Wed) by anarcat (subscriber, #66354)
In reply to: Monitoring with Prometheus 2.0 by bitfehler
Parent article: Monitoring with Prometheus 2.0

"But for monitoring endpoints, this is potentially hundreds of services that are available publicly without any protection." That part is a misunderstanding. For scraping, Prometheus supports all kinds of security, including regular TLS, client certificates (https://prometheus.io/docs/prometheus/latest/configuratio...) as well as HTTP basic auth (https://prometheus.io/docs/prometheus/latest/configuratio...).

Sure: prom supports scraping HTTPS targets. But by default, the node_exporter (and in fact most exporters as well) do not export their metrics through HTTPS. Users are told to install a TLS proxy in front to enable end-to-end security.

And even then: this doesn't authenticate the collecting server against the metrics target. For that you need yet another authentication layer. Furthermore, many container deployments do not use HTTPS internally: it's all plain text, and then HTTPS is added on the edges, which means a lot of this traffic goes in the clear. So I think it's a fairly accurate description. It doesn't mean it's catastrophic: many organizations have been running Munin exactly that way forever. But it's something to keep in mind when deploying Prometheus: it's not magic.

The security guide is great, in that regard: honest, and to the point. Thank you for that.

Besides that, nice overview. The criticism is valid, however in my experience the benefits start to outweigh the downsides at a certain scale, e.g. at some point the flexibility and interoperability with other components becomes a major feature (e.g. "having to" use Grafana is nice because we show data from other sources than just Prometheus, etc). I am sure more "out-of-the-box" solutions will show up eventually.

Yep. Note that in the last paragraph, i suggest sysadmins should wait befor converting existing infrastructures, but I would probably use prometheus to monitor any new infrastructure I would setup in the future. My only concern is disk space and downsampling, but I will be touching on that subject more in the next article, which should come out next week. Stay tuned! :)