2017-10-26 06:53:27 -07:00
|
|
|
---
|
|
|
|
title: Recording rules
|
2017-10-27 00:47:38 -07:00
|
|
|
sort_rank: 2
|
2017-10-26 06:53:27 -07:00
|
|
|
---
|
|
|
|
|
|
|
|
# Defining recording rules
|
|
|
|
|
|
|
|
## Configuring rules
|
|
|
|
|
|
|
|
Prometheus supports two types of rules which may be configured and then
|
|
|
|
evaluated at regular intervals: recording rules and [alerting
|
2017-10-27 00:47:38 -07:00
|
|
|
rules](alerting_rules.md). To include rules in Prometheus, create a file
|
|
|
|
containing the necessary rule statements and have Prometheus load the file via
|
|
|
|
the `rule_files` field in the [Prometheus configuration](configuration.md).
|
2017-11-01 05:58:32 -07:00
|
|
|
Rule files use YAML.
|
2017-10-26 06:53:27 -07:00
|
|
|
|
|
|
|
The rule files can be reloaded at runtime by sending `SIGHUP` to the Prometheus
|
|
|
|
process. The changes are only applied if all rule files are well-formatted.
|
|
|
|
|
2023-01-12 00:05:41 -08:00
|
|
|
_Note about native histograms (experimental feature): Native histogram are always
|
|
|
|
recorded as gauge histograms (for now). Most cases will create gauge histograms
|
|
|
|
naturally, e.g. after `rate()`._
|
2022-10-14 04:05:27 -07:00
|
|
|
|
2017-10-26 06:53:27 -07:00
|
|
|
## Syntax-checking rules
|
|
|
|
|
|
|
|
To quickly check whether a rule file is syntactically correct without starting
|
2021-02-15 12:59:32 -08:00
|
|
|
a Prometheus server, you can use Prometheus's `promtool` command-line utility
|
|
|
|
tool:
|
2017-10-26 06:53:27 -07:00
|
|
|
|
|
|
|
```bash
|
2017-11-01 05:58:32 -07:00
|
|
|
promtool check rules /path/to/example.rules.yml
|
2017-10-26 06:53:27 -07:00
|
|
|
```
|
|
|
|
|
2021-02-15 12:59:32 -08:00
|
|
|
The `promtool` binary is part of the `prometheus` archive offered on the
|
|
|
|
project's [download page](https://prometheus.io/download/).
|
|
|
|
|
2017-10-26 06:53:27 -07:00
|
|
|
When the file is syntactically valid, the checker prints a textual
|
|
|
|
representation of the parsed rules to standard output and then exits with
|
|
|
|
a `0` return status.
|
|
|
|
|
2017-11-01 08:35:50 -07:00
|
|
|
If there are any syntax errors or invalid input arguments, it prints an error
|
|
|
|
message to standard error and exits with a `1` return status.
|
2017-10-26 06:53:27 -07:00
|
|
|
|
|
|
|
## Recording rules
|
|
|
|
|
|
|
|
Recording rules allow you to precompute frequently needed or computationally
|
|
|
|
expensive expressions and save their result as a new set of time series.
|
|
|
|
Querying the precomputed result will then often be much faster than executing
|
|
|
|
the original expression every time it is needed. This is especially useful for
|
|
|
|
dashboards, which need to query the same expression repeatedly every time they
|
|
|
|
refresh.
|
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
Recording and alerting rules exist in a rule group. Rules within a group are
|
2020-12-03 02:32:10 -08:00
|
|
|
run sequentially at a regular interval, with the same evaluation time.
|
|
|
|
The names of recording rules must be
|
|
|
|
[valid metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
2020-11-18 11:29:01 -08:00
|
|
|
The names of alerting rules must be
|
|
|
|
[valid label values](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
2017-10-26 06:53:27 -07:00
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
The syntax of a rule file is:
|
2017-10-26 06:53:27 -07:00
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
```yaml
|
|
|
|
groups:
|
|
|
|
[ - <rule_group> ]
|
|
|
|
```
|
|
|
|
|
|
|
|
A simple example rules file would be:
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
groups:
|
|
|
|
- name: example
|
2017-11-14 07:22:25 -08:00
|
|
|
rules:
|
2023-01-09 16:36:07 -08:00
|
|
|
- record: code:prometheus_http_requests_total:sum
|
|
|
|
expr: sum by (code) (prometheus_http_requests_total)
|
2017-11-01 05:58:32 -07:00
|
|
|
```
|
|
|
|
|
|
|
|
### `<rule_group>`
|
|
|
|
```
|
|
|
|
# The name of the group. Must be unique within a file.
|
|
|
|
name: <string>
|
|
|
|
|
|
|
|
# How often rules in the group are evaluated.
|
|
|
|
[ interval: <duration> | default = global.evaluation_interval ]
|
|
|
|
|
2021-10-21 14:14:17 -07:00
|
|
|
# Limit the number of alerts an alerting rule and series a recording
|
|
|
|
# rule can produce. 0 is no limit.
|
2021-09-15 00:48:26 -07:00
|
|
|
[ limit: <int> | default = 0 ]
|
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
rules:
|
|
|
|
[ - <rule> ... ]
|
|
|
|
```
|
|
|
|
|
|
|
|
### `<rule>`
|
2017-10-26 06:53:27 -07:00
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
The syntax for recording rules is:
|
2017-11-08 02:38:03 -08:00
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
```
|
|
|
|
# The name of the time series to output to. Must be a valid metric name.
|
|
|
|
record: <string>
|
|
|
|
|
|
|
|
# The PromQL expression to evaluate. Every evaluation cycle this is
|
|
|
|
# evaluated at the current time, and the result recorded as a new set of
|
|
|
|
# time series with the metric name as given by 'record'.
|
|
|
|
expr: <string>
|
|
|
|
|
|
|
|
# Labels to add or overwrite before storing the result.
|
|
|
|
labels:
|
|
|
|
[ <labelname>: <labelvalue> ]
|
|
|
|
```
|
|
|
|
|
|
|
|
The syntax for alerting rules is:
|
2017-11-08 02:38:03 -08:00
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
```
|
2020-07-06 14:00:16 -07:00
|
|
|
# The name of the alert. Must be a valid label value.
|
2017-11-01 05:58:32 -07:00
|
|
|
alert: <string>
|
|
|
|
|
|
|
|
# The PromQL expression to evaluate. Every evaluation cycle this is
|
|
|
|
# evaluated at the current time, and all resultant time series become
|
|
|
|
# pending/firing alerts.
|
|
|
|
expr: <string>
|
|
|
|
|
|
|
|
# Alerts are considered firing once they have been returned for this long.
|
|
|
|
# Alerts which have not yet fired for long enough are considered pending.
|
|
|
|
[ for: <duration> | default = 0s ]
|
2017-10-26 06:53:27 -07:00
|
|
|
|
2023-01-09 03:21:38 -08:00
|
|
|
# How long an alert will continue firing after the condition that triggered it
|
|
|
|
# has cleared.
|
|
|
|
[ keep_firing_for: <duration> | default = 0s ]
|
|
|
|
|
2017-11-01 05:58:32 -07:00
|
|
|
# Labels to add or overwrite for each alert.
|
|
|
|
labels:
|
|
|
|
[ <labelname>: <tmpl_string> ]
|
|
|
|
|
|
|
|
# Annotations to add to each alert.
|
|
|
|
annotations:
|
|
|
|
[ <labelname>: <tmpl_string> ]
|
|
|
|
```
|
2017-10-26 06:53:27 -07:00
|
|
|
|
2022-12-12 07:08:45 -08:00
|
|
|
See also the
|
|
|
|
[best practices for naming metrics created by recording rules](https://prometheus.io/docs/practices/rules/#recording-rules).
|
|
|
|
|
2021-10-21 14:14:17 -07:00
|
|
|
# Limiting alerts and series
|
|
|
|
|
|
|
|
A limit for alerts produced by alerting rules and series produced recording rules
|
|
|
|
can be configured per-group. When the limit is exceeded, _all_ series produced
|
|
|
|
by the rule are discarded, and if it's an alerting rule, _all_ alerts for
|
|
|
|
the rule, active, pending, or inactive, are cleared as well. The event will be
|
|
|
|
recorded as an error in the evaluation, and as such no stale markers are
|
|
|
|
written.
|