prometheus/documentation/prometheus-mixin
Julien Duchesne 8855c2e626
Add prometheus_tsdb_clean_start metric (#8824)
Add cleanup of the lockfile when the db is cleanly closed

The metric describes the status of the lockfile on startup
0: Already existed
1: Did not exist
-1: Disabled

Therefore, if the min value over time of this metric is 0, that means that executions have exited uncleanly
We can then use that metric to have a much lower threshold on the crashlooping alert:

If the metric exists and it has been zero, two restarts is enough to trigger the alarm
If it does not exist (old prom version for example), the current five restarts threshold remains

Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>

* Change metric name + set unset value to -1

Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>

* Only check the last value of the clean start alert

Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>

* Fix test + nit

Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
2021-06-16 15:03:02 +05:30
..
.gitignore Add .gitignore file 2019-06-26 15:22:23 +02:00
alerts.jsonnet Add Makefile and suitable jsonnet files 2019-06-26 15:30:55 +02:00
alerts.libsonnet Add prometheus_tsdb_clean_start metric (#8824) 2021-06-16 15:03:02 +05:30
config.libsonnet mixin: Scope grafana config 2020-12-30 17:50:34 +01:00
dashboards.jsonnet Fix prometheus-mixin dashboards to use grafanaDashboards 2019-07-11 15:40:26 +02:00
dashboards.libsonnet Add body_size_limit to prevent bad targets response large body cause Prometheus server OOM (#8827) 2021-05-29 07:05:42 +08:00
go.mod .circleci/config.yml: check mixins (#6895) 2020-08-25 15:59:41 +02:00
go.sum .circleci/config.yml: check mixins (#6895) 2020-08-25 15:59:41 +02:00
jsonnetfile.json Use absolute jsonnet import paths 2020-10-20 11:42:30 +02:00
Makefile Adjust to jsonnet v0.13 2019-06-26 16:22:21 +02:00
mixin.libsonnet Basic Prometheus dashboard. 2018-11-16 17:17:47 +00:00
README.md Update Readme since jsonnetfmt is available in the jsonnet go implementation since v0.16.0 2020-06-16 10:41:58 +01:00
tools.go .circleci/config.yml: check mixins (#6895) 2020-08-25 15:59:41 +02:00

Prometheus Mixin

This is work in progress. We aim for it to become a good role model for alerts and dashboards eventually, but it is not quite there yet.

The Prometheus Mixin is a set of configurable, reusable, and extensible alerts and dashboards for Prometheus.

To use them, you need to have jsonnet (v0.13+) and jb installed. If you have a working Go development environment, it's easiest to run the following:

$ go get github.com/google/go-jsonnet/cmd/jsonnet
$ go get github.com/google/go-jsonnet/cmd/jsonnetfmt 
$ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

Note: The make targets lint and fmt need the jsonnetfmt binary, which is available from v.0.16.0 in the Go implementation of jsonnet. If your jsonnet version is older than 0.16.0 you have to either upgrade or install the C++ version of jsonnetfmt if you want to use make lint or make fmt.

Next, install the dependencies by running the following command in this directory:

$ jb install

You can then build a prometheus_alerts.yaml with the alerts and a directory dashboards_out with the Grafana dashboard JSON files:

$ make prometheus_alerts.yaml
$ make dashboards_out

For more advanced uses of mixins, see https://github.com/monitoring-mixins/docs.