Monitoring with Prometheus 2.0

Posted Feb 2, 2018 7:53 UTC (Fri) by faxm0dem (guest, #92265)
Parent article: Monitoring with Prometheus 2.0

Long-term storage is very important to us, but RRDTool doesn't scale unless you DIY. The solution we came up with is to pre-aggregate the data into multiple resolutions using configurable consolidation functions, just like RRDTool does, but on top of a modern storage. We wrote a riemann plugin that does the aggregation in realtime, and then indexes the results into Elasticsearch. It also handles the aliases so that the access to the data is transparent to the user (highest resolution available gets higher priority) and curates old data automatically so that storage usage doesn't increase in time.