mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-18 03:24:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
prometheus/docs/configuration/configuration.md

1304 lines
49 KiB
Markdown
Raw Normal View History

Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
---
title: Configuration
Consolidate configuration and rules docs in docs/configuration/
2017-10-27 00:47:38 -07:00
sort_rank: 1
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
---
# Configuration
Prometheus is configured via command-line flags and a configuration file. While
the command-line flags configure immutable system parameters (such as storage
locations, amount of data to keep on disk and in memory, etc.), the
configuration file defines everything related to scraping [jobs and their
Added another full link to the configuration docs (#3553)
2017-12-07 00:31:15 -08:00
instances](https://prometheus.io/docs/concepts/jobs_instances/), as well as
Consolidate configuration and rules docs in docs/configuration/
2017-10-27 00:47:38 -07:00
which [rule files to load](recording_rules.md#configuring-rules).
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
docs: Fix minor issues with the docs. (#3389) Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-01 08:35:50 -07:00
To view all available command-line flags, run `./prometheus -h`.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
Prometheus can reload its configuration at runtime. If the new configuration
is not well-formed, the changes will not be applied.
A configuration reload is triggered by sending a `SIGHUP` to the Prometheus process or
docs: Fix minor issues with the docs. (#3389) Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-01 08:35:50 -07:00
sending a HTTP POST request to the `/-/reload` endpoint (when the `--web.enable-lifecycle` flag is enabled).
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
This will also reload any configured rule files.
## Configuration file
docs: Fix minor issues with the docs. (#3389) Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-01 08:35:50 -07:00
To specify which configuration file to load, use the `--config.file` flag.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
The file is written in [YAML format](http://en.wikipedia.org/wiki/YAML),
defined by the scheme described below.
Brackets indicate that a parameter is optional. For non-list parameters the
value is set to the specified default.
Generic placeholders are defined as follows:
* `<boolean>`: a boolean that can take the values `true` or `false`
* `<duration>`: a duration matching the regular expression `[0-9]+(ms|[smhdwy])`
* `<labelname>`: a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*`
* `<labelvalue>`: a string of unicode characters
* `<filename>`: a valid path in the current working directory
* `<host>`: a valid string consisting of a hostname or IP followed by an optional port number
* `<path>`: a valid URL path
* `<scheme>`: a string that can take the values `http` or `https`
* `<string>`: a regular string
* `<secret>`: a regular string that is a secret, such as a password
docs: Document new recording rule format (#3378)
2017-11-01 05:58:32 -07:00
* `<tmpl_string>`: a string which is template-expanded before usage
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
The other placeholders are specified separately.
A valid example file can be found [here](/config/testdata/conf.good.yml).
The global configuration specifies parameters that are valid in all other configuration
contexts. They also serve as defaults for other configuration sections.
```yaml
global:
# How frequently to scrape targets by default.
[ scrape_interval: <duration> | default = 1m ]
# How long until a scrape request times out.
[ scrape_timeout: <duration> | default = 10s ]
# How frequently to evaluate rules.
[ evaluation_interval: <duration> | default = 1m ]
# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
[ <labelname>: <labelvalue> ... ]
# Rule files specifies a list of globs. Rules and alerts are read from
# all matching files.
rule_files:
[ - <filepath_glob> ... ]
# A list of scrape configurations.
scrape_configs:
[ - <scrape_config> ... ]
# Alerting specifies settings related to the Alertmanager.
alerting:
alert_relabel_configs:
[ - <relabel_config> ... ]
alertmanagers:
[ - <alertmanager_config> ... ]
Add 2.0 stability guarantees (#3484) As discussed generally consider SDs as unstable, as realistically they are never going to be. Drop the words "experimental/beta" from most places in the docs, as users are getting the wrong impression from this.
2017-12-14 04:54:32 -08:00
# Settings related to the remote write feature.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
remote_write:
[ - <remote_write> ... ]
Add 2.0 stability guarantees (#3484) As discussed generally consider SDs as unstable, as realistically they are never going to be. Drop the words "experimental/beta" from most places in the docs, as users are getting the wrong impression from this.
2017-12-14 04:54:32 -08:00
# Settings related to the remote read feature.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
remote_read:
[ - <remote_read> ... ]
```
### `<scrape_config>`
A `scrape_config` section specifies a set of targets and parameters describing how
to scrape them. In the general case, one scrape configuration specifies a single
job. In advanced configurations, this may change.
Targets may be statically configured via the `static_configs` parameter or
dynamically discovered using one of the supported service-discovery mechanisms.
Additionally, `relabel_configs` allow advanced modifications to any
target and its labels before scraping.
```yaml
# The job name assigned to scraped metrics by default.
job_name: <job_name>
# How frequently to scrape targets from this job.
[ scrape_interval: <duration> | default = <global_config.scrape_interval> ]
# Per-scrape timeout when scraping this job.
[ scrape_timeout: <duration> | default = <global_config.scrape_timeout> ]
# The HTTP resource path on which to fetch metrics from targets.
[ metrics_path: <path> | default = /metrics ]
# honor_labels controls how Prometheus handles conflicts between labels that are
# already present in scraped data and labels that Prometheus would attach
# server-side ("job" and "instance" labels, manually configured target
# labels, and labels generated by service discovery implementations).
#
# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels. This is useful for use cases such as federation, where all labels
# specified in the target should be preserved.
#
# Note that any globally configured "external_labels" are unaffected by this
# setting. In communication with external systems, they are always applied only
# when a time series does not have a given label yet and are ignored otherwise.
[ honor_labels: <boolean> | default = false ]
# Configures the protocol scheme used for requests.
[ scheme: <scheme> | default = http ]
# Optional HTTP URL parameters.
params:
[ <string>: [<string>, ...] ]
# Sets the `Authorization` header on every scrape request with the
# configured username and password.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
basic_auth:
[ username: <string> ]
[ password: <secret> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Sets the `Authorization` header on every scrape request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <secret> ]
# Sets the `Authorization` header on every scrape request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the scrape request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
# List of Azure service discovery configurations.
azure_sd_configs:
[ - <azure_sd_config> ... ]
# List of Consul service discovery configurations.
consul_sd_configs:
[ - <consul_sd_config> ... ]
# List of DNS service discovery configurations.
dns_sd_configs:
[ - <dns_sd_config> ... ]
# List of EC2 service discovery configurations.
ec2_sd_configs:
[ - <ec2_sd_config> ... ]
# List of OpenStack service discovery configurations.
openstack_sd_configs:
[ - <openstack_sd_config> ... ]
# List of file service discovery configurations.
file_sd_configs:
[ - <file_sd_config> ... ]
# List of GCE service discovery configurations.
gce_sd_configs:
[ - <gce_sd_config> ... ]
# List of Kubernetes service discovery configurations.
kubernetes_sd_configs:
[ - <kubernetes_sd_config> ... ]
# List of Marathon service discovery configurations.
marathon_sd_configs:
[ - <marathon_sd_config> ... ]
# List of AirBnB's Nerve service discovery configurations.
nerve_sd_configs:
[ - <nerve_sd_config> ... ]
# List of Zookeeper Serverset service discovery configurations.
serverset_sd_configs:
[ - <serverset_sd_config> ... ]
# List of Triton service discovery configurations.
triton_sd_configs:
[ - <triton_sd_config> ... ]
# List of labeled statically configured targets for this job.
static_configs:
[ - <static_config> ... ]
# List of target relabel configurations.
relabel_configs:
[ - <relabel_config> ... ]
# List of metric relabel configurations.
metric_relabel_configs:
[ - <relabel_config> ... ]
# Per-scrape limit on number of scraped samples that will be accepted.
# If more than this number of samples are present after metric relabelling
# the entire scrape will be treated as failed. 0 means no limit.
[ sample_limit: <int> | default = 0 ]
```
Where `<job_name>` must be unique across all scrape configurations.
### `<tls_config>`
A `tls_config` allows configuring TLS connections.
```yaml
# CA certificate to validate API server certificate with.
[ ca_file: <filename> ]
# Certificate and key files for client cert authentication to the server.
[ cert_file: <filename> ]
[ key_file: <filename> ]
# ServerName extension to indicate the name of the server.
# http://tools.ietf.org/html/rfc4366#section-3.1
[ server_name: <string> ]
# Disable validation of the server certificate.
[ insecure_skip_verify: <boolean> ]
```
### `<azure_sd_config>`
Azure SD configurations allow retrieving scrape targets from Azure VMs.
The following meta labels are available on targets during relabeling:
* `__meta_azure_machine_id`: the machine ID
* `__meta_azure_machine_location`: the location the machine runs in
* `__meta_azure_machine_name`: the machine name
Add OS type meta label to Azure SD (#3863) There is currently no way to differentiate Windows instances from Linux ones. This is needed when you have a mix of node_exporters / wmi_exporters for OS-level metrics and you want to have them in separate scrape jobs. This change allows you to do just that. Example: ``` - job_name: 'node' azure_sd_configs: - <azure_sd_config> relabel_configs: - source_labels: [__meta_azure_machine_os_type] regex: Linux action: keep ``` The way the vendor'd AzureSDK provides to get the OsType is a bit awkward - as far as I can tell, this information can only be gotten from the startup disk. Newer versions of the SDK appear to improve this a bit (by having OS information in the InstanceView), but the current way still works.
2018-02-19 07:40:57 -08:00
* `__meta_azure_machine_os_type`: the machine operating system
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_azure_machine_private_ip`: the machine's private IP
* `__meta_azure_machine_resource_group`: the machine's resource group
* `__meta_azure_machine_tag_<tagname>`: each tag value of the machine
Inital support for Azure VMSS (#4202) * Inital support for Azure VMSS Signed-off-by: Johannes Scheuermann <johannes.scheuermann@inovex.de> * Add documentation for the newly introduced label Signed-off-by: Johannes M. Scheuermann <joh.scheuer@gmail.com>
2018-08-01 04:52:21 -07:00
* `__meta_azure_machine_scale_set`: the name of the scale set which the vm is part of (this value is only set if you are using a [scale set](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/))
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
See below for the configuration options for Azure discovery:
```yaml
# The information to access the Azure API.
Adding support for multiple azure environments (#4569) Signed-off-by: Tariq Ibrahim <tariq.ibrahim@microsoft.com>
2018-09-04 08:55:40 -07:00
# The Azure environment.
[ environment: <string> | default = AzurePublicCloud ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# The subscription ID.
subscription_id: <string>
# The tenant ID.
tenant_id: <string>
# The client ID.
client_id: <string>
# The client secret.
client_secret: <secret>
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 300s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
```
### `<consul_sd_config>`
Consul SD configurations allow retrieving scrape targets from [Consul's](https://www.consul.io)
Catalog API.
The following meta labels are available on targets during [relabeling](#relabel_config):
* `__meta_consul_address`: the address of the target
* `__meta_consul_dc`: the datacenter name for the target
consul: improve consul service discovery (#3814) * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.
2018-03-23 07:48:43 -07:00
* `__meta_consul_metadata_<key>`: each node metadata key value of the target
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_consul_node`: the node name defined for the target
* `__meta_consul_service_address`: the service address of the target
* `__meta_consul_service_id`: the service ID of the target
Discovery consul service meta (#4280) * Upgrade Consul client * Add ServiceMeta to the labels in ConsulSD Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>
2018-07-17 21:06:56 -07:00
* `__meta_consul_service_metadata_<key>`: each service metadata key value of the target
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_consul_service_port`: the service port of the target
* `__meta_consul_service`: the name of the service the target belongs to
* `__meta_consul_tags`: the list of tags of the target joined by the tag separator
```yaml
# The information to access the Consul API. It is to be defined
# as the Consul documentation requires.
Set consul server default to localhost:8500 (#3703)
2018-01-24 04:14:32 -08:00
[ server: <host> | default = "localhost:8500" ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
[ token: <secret> ]
[ datacenter: <string> ]
Set consul server default to localhost:8500 (#3703)
2018-01-24 04:14:32 -08:00
[ scheme: <string> | default = "http" ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
[ username: <string> ]
[ password: <secret> ]
Document consul sd tls_config (#3440) Fixes https://github.com/prometheus/docs/issues/681
2017-11-08 08:36:08 -08:00
tls_config:
[ <tls_config> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# A list of services for which targets are retrieved. If omitted, all services
# are scraped.
services:
[ - <string> ]
consul: improve consul service discovery (#3814) * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.
2018-03-23 07:48:43 -07:00
# See https://www.consul.io/api/catalog.html#list-nodes-for-service to know more
# about the possible filters that can be used.
# An optional tag used to filter nodes for a given service.
[ tag: <string> ]
# Node metadata used to filter nodes for a given service.
[ node_meta:
[ <name>: <value> ... ] ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# The string by which Consul tags are joined into the tag label.
[ tag_separator: <string> | default = , ]
consul: improve consul service discovery (#3814) * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.
2018-03-23 07:48:43 -07:00
# Allow stale Consul results (see https://www.consul.io/api/index.html#consistency-modes). Will reduce load on Consul.
[ allow_stale: <bool> ]
# The time after which the provided names are refreshed.
# On large setup it might be a good idea to increase this value because the catalog will change all the time.
[ refresh_interval: <duration> | default = 30s ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```
Note that the IP number and port used to scrape the targets is assembled as
`<__meta_consul_address>:<__meta_consul_service_port>`. However, in some
Consul setups, the relevant address is in `__meta_consul_service_address`.
In those cases, you can use the [relabel](#relabel_config)
feature to replace the special `__address__` label.
consul: improve consul service discovery (#3814) * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.
2018-03-23 07:48:43 -07:00
The [relabeling phase](#relabel_config) is the preferred and more powerful
way to filter services or nodes for a service based on arbitrary labels. For
users with thousands of services it can be more efficient to use the Consul API
directly which has basic support for filtering nodes (currently by node
metadata and a single tag).
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
### `<dns_sd_config>`
A DNS-based service discovery configuration allows specifying a set of DNS
domain names which are periodically queried to discover a list of targets. The
DNS servers to be contacted are read from `/etc/resolv.conf`.
This service discovery method only supports basic DNS A, AAAA and SRV record
queries, but not the advanced DNS-SD approach specified in
[RFC6763](https://tools.ietf.org/html/rfc6763).
During the [relabeling phase](#relabel_config), the meta label
`__meta_dns_name` is available on each target and is set to the
record name that produced the discovered target.
```yaml
# A list of DNS domain names to be queried.
names:
[ - <domain_name> ]
# The type of DNS query to perform.
[ type: <query_type> | default = 'SRV' ]
# The port number used if the query type is not SRV.
[ port: <number>]
# The time after which the provided names are refreshed.
[ refresh_interval: <duration> | default = 30s ]
```
Where `<domain_name>` is a valid DNS domain name.
Where `<query_type>` is `SRV`, `A`, or `AAAA`.
### `<ec2_sd_config>`
EC2 SD configurations allow retrieving scrape targets from AWS EC2
instances. The private IP address is used by default, but may be changed to
the public IP address with relabeling.
The following meta labels are available on targets during [relabeling](#relabel_config):
* `__meta_ec2_availability_zone`: the availability zone in which the instance is running
* `__meta_ec2_instance_id`: the EC2 instance ID
* `__meta_ec2_instance_state`: the state of the EC2 instance
* `__meta_ec2_instance_type`: the type of the EC2 instance
Expose EC2 instance owner as a discovery label. This exposes the OwnerID field of the DescribeInstances respons as . Signed-off-by: Javier Kohen <jkohen@google.com>
2018-08-17 08:30:18 -07:00
* `__meta_ec2_owner_id`: the ID of the AWS account that owns the EC2 instance
discovery/ec2: Add primary_subnet_id label Signed-off-by: José Martínez <xosemp@gmail.com>
2018-07-25 00:38:14 -07:00
* `__meta_ec2_primary_subnet_id`: the subnet ID of the primary network interface, if available
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_ec2_private_ip`: the private IP address of the instance, if present
* `__meta_ec2_public_dns_name`: the public DNS name of the instance, if available
* `__meta_ec2_public_ip`: the public IP address of the instance, if available
* `__meta_ec2_subnet_id`: comma separated list of subnets IDs in which the instance is running, if available
* `__meta_ec2_tag_<tagkey>`: each tag value of the instance
* `__meta_ec2_vpc_id`: the ID of the VPC in which the instance is running, if available
See below for the configuration options for EC2 discovery:
```yaml
# The information to access the EC2 API.
# The AWS Region.
region: <string>
EC2 Discovery: Allow to set a custom endpoint (#4333) Allowing to set a custom endpoint makes it easy to monitor targets on non AWS providers with EC2 compliant APIs. Signed-off-by: Jannick Fahlbusch <git@jf-projects.de>
2018-07-18 02:48:14 -07:00
# Custom endpoint to be used.
[ endpoint: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# The AWS API keys. If blank, the environment variables `AWS_ACCESS_KEY_ID`
# and `AWS_SECRET_ACCESS_KEY` are used.
[ access_key: <string> ]
[ secret_key: <secret> ]
# Named AWS profile used to connect to the API.
[ profile: <string> ]
Include 1.8 changes in configuration docs
2017-10-10 10:55:29 -07:00
# AWS Role ARN, an alternative to using AWS API keys.
[ role_arn: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
discovery/ec2: Support filtering instances in discovery (#4011)
2018-03-30 23:51:11 -07:00
# Filters can be used optionally to filter the instance list by other criteria.
# Available filter criteria can be found here:
# https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html
# Filter API documentation: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Filter.html
filters:
[ - name: <string>
values: <string>, [...] ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```
discovery/ec2: Support filtering instances in discovery (#4011)
2018-03-30 23:51:11 -07:00
The [relabeling phase](#relabel_config) is the preferred and more powerful
way to filter targets based on arbitrary labels. For users with thousands of
instances it can be more efficient to use the EC2 API directly which has
support for filtering instances.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
### `<openstack_sd_config>`
Add 2.0 stability guarantees (#3484) As discussed generally consider SDs as unstable, as realistically they are never going to be. Drop the words "experimental/beta" from most places in the docs, as users are getting the wrong impression from this.
2017-12-14 04:54:32 -08:00
OpenStack SD configurations allow retrieving scrape targets from OpenStack Nova
instances.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
docs: fix OpenStack SD for the hypervisor role Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-07-13 07:48:54 -07:00
One of the following `<openstack_role>` types can be configured to discover targets:
#### `hypervisor`
The `hypervisor` role discovers one target per Nova hypervisor node. The target
address defaults to the `host_ip` attribute of the hypervisor.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
The following meta labels are available on targets during [relabeling](#relabel_config):
docs: fix OpenStack SD for the hypervisor role Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-07-13 07:48:54 -07:00
* `__meta_openstack_hypervisor_host_ip`: the hypervisor node's IP address.
* `__meta_openstack_hypervisor_name`: the hypervisor node's name.
* `__meta_openstack_hypervisor_state`: the hypervisor node's state.
* `__meta_openstack_hypervisor_status`: the hypervisor node's status.
* `__meta_openstack_hypervisor_type`: the hypervisor node's type.
Update the openstack_sd_config section openstack_sd_config requires a 'role' parameter which wasn't documented.
2017-12-08 07:00:03 -08:00
#### `instance`
discovery/openstack: discover all interfaces (#4649) * discovery/openstack: discover all interfaces * Add address pool label Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-10-09 08:17:08 -07:00
The `instance` role discovers one target per network interface of Nova
instance. The target address defaults to the private IP address of the network
interface.
Update the openstack_sd_config section openstack_sd_config requires a 'role' parameter which wasn't documented.
2017-12-08 07:00:03 -08:00
The following meta labels are available on targets during [relabeling](#relabel_config):
* `__meta_openstack_instance_id`: the OpenStack instance ID.
* `__meta_openstack_instance_name`: the OpenStack instance name.
* `__meta_openstack_instance_status`: the status of the OpenStack instance.
* `__meta_openstack_instance_flavor`: the flavor of the OpenStack instance.
* `__meta_openstack_public_ip`: the public IP of the OpenStack instance.
* `__meta_openstack_private_ip`: the private IP of the OpenStack instance.
discovery/openstack: discover all interfaces (#4649) * discovery/openstack: discover all interfaces * Add address pool label Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-10-09 08:17:08 -07:00
* `__meta_openstack_address_pool`: the pool of the private IP.
Update the openstack_sd_config section openstack_sd_config requires a 'role' parameter which wasn't documented.
2017-12-08 07:00:03 -08:00
* `__meta_openstack_tag_<tagkey>`: each tag value of the instance.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
See below for the configuration options for OpenStack discovery:
```yaml
# The information to access the OpenStack API.
Update the openstack_sd_config section openstack_sd_config requires a 'role' parameter which wasn't documented.
2017-12-08 07:00:03 -08:00
# The OpenStack role of entities that should be discovered.
docs: fix OpenStack SD for the hypervisor role Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-07-13 07:48:54 -07:00
role: <openstack_role>
Update the openstack_sd_config section openstack_sd_config requires a 'role' parameter which wasn't documented.
2017-12-08 07:00:03 -08:00
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# The OpenStack Region.
region: <string>
# identity_endpoint specifies the HTTP endpoint that is required to work with
# the Identity API of the appropriate version. While it's ultimately needed by
# all of the identity services, it will often be populated by a provider-level
# function.
[ identity_endpoint: <string> ]
# username is required if using Identity V2 API. Consult with your provider's
# control panel to discover your account's username. In Identity V3, either
# userid or a combination of username and domain_id or domain_name are needed.
[ username: <string> ]
[ userid: <string> ]
# password for the Identity V2 and V3 APIs. Consult with your provider's
# control panel to discover your account's preferred method of authentication.
[ password: <secret> ]
# At most one of domain_id and domain_name must be provided if using username
# with Identity V3. Otherwise, either are optional.
[ domain_name: <string> ]
[ domain_id: <string> ]
# The project_id and project_name fields are optional for the Identity V2 API.
# Some providers allow you to specify a project_name instead of the project_id.
# Some require both. Your provider's authentication policies will determine
# how these fields influence authentication.
[ project_name: <string> ]
[ project_id: <string> ]
Support discover instances from all projects (#4682) By default, OpenStack SD only queries for instances from specified project. To discover instances from other projects, users have to add more openstack_sd_configs for each project. This patch adds `all_tenants` <bool> options to openstack_sd_configs. For example: - job_name: 'openstack_all_instances' openstack_sd_configs: - role: instance region: RegionOne identity_endpoint: http://<identity_server>/identity/v3 username: <username> password: <super_secret_password> domain_name: Default all_tenants: true Co-authored-by: Kien Nguyen <kiennt2609@gmail.com> Signed-off-by: dmatosl <danielmatos.lima@gmail.com>
2018-10-17 05:01:33 -07:00
# Whether the service discovery should list all instances for all projects.
# It is only relevant for the 'instance' role and usually requires admin permissions.
[ all_tenants: <boolean> | default: false ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Refresh interval to re-read the instance list.
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
discovery/openstack: support tls_config Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-24 07:08:42 -07:00
# TLS configuration.
tls_config:
[ <tls_config> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```
### `<file_sd_config>`
File-based service discovery provides a more generic way to configure static targets
and serves as an interface to plug in custom service discovery mechanisms.
It reads a set of files containing a list of zero or more
`<static_config>`s. Changes to all defined files are detected via disk watches
and applied immediately. Files may be provided in YAML or JSON format. Only
changes resulting in well-formed target groups are applied.
The JSON file must contain a list of static configs, using this format:
```yaml
[
{
"targets": [ "<host>", ... ],
"labels": {
"<labelname>": "<labelvalue>", ...
}
},
...
]
```
As a fallback, the file contents are also re-read periodically at the specified
refresh interval.
Each target has a meta label `__meta_filepath` during the
[relabeling phase](#relabel_config). Its value is set to the
filepath from which the target was extracted.
There is a list of
Add prefix to relative 3 URLs (#3551)
2017-12-06 13:16:53 -08:00
[integrations](https://prometheus.io/docs/operating/integrations/#file-service-discovery) with this
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
discovery mechanism.
```yaml
# Patterns for files from which target groups are extracted.
files:
[ - <filename_pattern> ... ]
# Refresh interval to re-read the files.
[ refresh_interval: <duration> | default = 5m ]
```
Where `<filename_pattern>` may be a path ending in `.json`, `.yml` or `.yaml`. The last path segment
may contain a single `*` that matches any character sequence, e.g. `my/path/tg_*.json`.
### `<gce_sd_config>`
[GCE](https://cloud.google.com/compute/) SD configurations allow retrieving scrape targets from GCP GCE instances.
The private IP address is used by default, but may be changed to the public IP
address with relabeling.
The following meta labels are available on targets during [relabeling](#relabel_config):
Document the new __meta_gce_instance_id discovery label. Signed-off-by: Javier Kohen <jkohen@google.com>
2018-08-10 08:59:22 -07:00
* `__meta_gce_instance_id`: the numeric id of the instance
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_gce_instance_name`: the name of the instance
Include GCE labels during service discovery. Updated vendor files for Google API. (#4150) Signed-off-by: Nathan Graves <nathan.graves@kofile.us>
2018-05-08 09:37:47 -07:00
* `__meta_gce_label_<name>`: each GCE label of the instance
Add a machine type label to GCE SD (#4032)
2018-03-31 01:20:19 -07:00
* `__meta_gce_machine_type`: full or partial URL of the machine type of the instance
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_gce_metadata_<name>`: each metadata item of the instance
* `__meta_gce_network`: the network URL of the instance
* `__meta_gce_private_ip`: the private IP address of the instance
* `__meta_gce_project`: the GCP project in which the instance is running
* `__meta_gce_public_ip`: the public IP address of the instance, if present
* `__meta_gce_subnetwork`: the subnetwork URL of the instance
* `__meta_gce_tags`: comma separated list of instance tags
* `__meta_gce_zone`: the GCE zone URL in which the instance is running
See below for the configuration options for GCE discovery:
```yaml
# The information to access the GCE API.
# The GCP Project
project: <string>
# The zone of the scrape targets. If you need multiple zones use multiple
# gce_sd_configs.
zone: <string>
# Filter can be used optionally to filter the instance list by other criteria
Integrate changes from prometheus/docs
2017-10-26 06:42:07 -07:00
# Syntax of this filter string is described here in the filter query parameter section:
# https://cloud.google.com/compute/docs/reference/latest/instances/list
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
[ filter: <string> ]
# Refresh interval to re-read the instance list
[ refresh_interval: <duration> | default = 60s ]
# The port to scrape metrics from. If using the public IP address, this must
# instead be specified in the relabeling rule.
[ port: <int> | default = 80 ]
# The tag separator is used to separate the tags on concatenation
[ tag_separator: <string> | default = , ]
```
Credentials are discovered by the Google Cloud SDK default client by looking
in the following places, preferring the first location found:
1. a JSON file specified by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
2. a JSON file in the well-known path `$HOME/.config/gcloud/application_default_credentials.json`
3. fetched from the GCE metadata server
If Prometheus is running within GCE, the service account associated with the
instance it is running on should have at least read-only permissions to the
compute resources. If running outside of GCE make sure to create an appropriate
service account and place the credential file in one of the expected locations.
### `<kubernetes_sd_config>`
Kubernetes SD configurations allow retrieving scrape targets from
[Kubernetes'](http://kubernetes.io/) REST API and always staying synchronized with
the cluster state.
One of the following `role` types can be configured to discover targets:
#### `node`
The `node` role discovers one target per cluster node with the address defaulting
to the Kubelet's HTTP port.
The target address defaults to the first existing address of the Kubernetes
node object in the address type order of `NodeInternalIP`, `NodeExternalIP`,
`NodeLegacyHostIP`, and `NodeHostName`.
Available meta labels:
* `__meta_kubernetes_node_name`: The name of the node object.
* `__meta_kubernetes_node_label_<labelname>`: Each label from the node object.
* `__meta_kubernetes_node_annotation_<annotationname>`: Each annotation from the node object.
* `__meta_kubernetes_node_address_<address_type>`: The first address for each node address type, if it exists.
In addition, the `instance` label for the node will be set to the node name
as retrieved from the API server.
#### `service`
The `service` role discovers a target for each service port for each service.
This is generally useful for blackbox monitoring of a service.
The address will be set to the Kubernetes DNS name of the service and respective
service port.
Available meta labels:
* `__meta_kubernetes_namespace`: The namespace of the service object.
* `__meta_kubernetes_service_name`: The name of the service object.
* `__meta_kubernetes_service_label_<labelname>`: The label of the service object.
* `__meta_kubernetes_service_annotation_<annotationname>`: The annotation of the service object.
* `__meta_kubernetes_service_port_name`: Name of the service port for the target.
* `__meta_kubernetes_service_port_number`: Number of the service port for the target.
* `__meta_kubernetes_service_port_protocol`: Protocol of the service port for the target.
#### `pod`
The `pod` role discovers all pods and exposes their containers as targets. For each declared
port of a container, a single target is generated. If a container has no specified ports,
a port-free target per container is created for manually adding a port via relabeling.
Available meta labels:
* `__meta_kubernetes_namespace`: The namespace of the pod object.
* `__meta_kubernetes_pod_name`: The name of the pod object.
* `__meta_kubernetes_pod_ip`: The pod IP of the pod object.
* `__meta_kubernetes_pod_label_<labelname>`: The label of the pod object.
* `__meta_kubernetes_pod_annotation_<annotationname>`: The annotation of the pod object.
* `__meta_kubernetes_pod_container_name`: Name of the container the target address points to.
* `__meta_kubernetes_pod_container_port_name`: Name of the container port.
* `__meta_kubernetes_pod_container_port_number`: Number of the container port.
* `__meta_kubernetes_pod_container_port_protocol`: Protocol of the container port.
* `__meta_kubernetes_pod_ready`: Set to `true` or `false` for the pod's ready state.
* `__meta_kubernetes_pod_node_name`: The name of the node the pod is scheduled onto.
* `__meta_kubernetes_pod_host_ip`: The current host IP of the pod object.
Update docs for __meta_kubernetes_pod_uid
2017-11-24 07:02:53 -08:00
* `__meta_kubernetes_pod_uid`: The UID of the pod object.
Expose controller kind and name to labelling rules Relabelling rules can use this information to attach the name of the controller that has created a pod. In turn, this can be used to slice metrics by workload at query time, ie. "Give me all metrics that have been created by the $name Deployment" Signed-off-by: Damien Lespiau <damien@weave.works>
2018-04-06 15:27:39 -07:00
* `__meta_kubernetes_pod_controller_kind`: Object kind of the pod controller.
* `__meta_kubernetes_pod_controller_name`: Name of the pod controller.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
#### `endpoints`
The `endpoints` role discovers targets from listed endpoints of a service. For each endpoint
address one target is discovered per port. If the endpoint is backed by a pod, all
additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
Available meta labels:
* `__meta_kubernetes_namespace`: The namespace of the endpoints object.
* `__meta_kubernetes_endpoints_name`: The names of the endpoints object.
* For all targets discovered directly from the endpoints list (those not additionally inferred
from underlying pods), the following labels are attached:
* `__meta_kubernetes_endpoint_ready`: Set to `true` or `false` for the endpoint's ready state.
* `__meta_kubernetes_endpoint_port_name`: Name of the endpoint port.
* `__meta_kubernetes_endpoint_port_protocol`: Protocol of the endpoint port.
Map target infos of endpoints to prometheus meta labels. (#3770)
2018-03-09 02:07:00 -08:00
* `__meta_kubernetes_endpoint_address_target_kind`: Kind of the endpoint address target.
* `__meta_kubernetes_endpoint_address_target_name`: Name of the endpoint address target.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* If the endpoints belong to a service, all labels of the `role: service` discovery are attached.
* For all targets backed by a pod, all labels of the `role: pod` discovery are attached.
Include 1.8 changes in configuration docs
2017-10-10 10:55:29 -07:00
#### `ingress`
The `ingress` role discovers a target for each path of each ingress.
This is generally useful for blackbox monitoring of an ingress.
The address will be set to the host specified in the ingress spec.
Available meta labels:
* `__meta_kubernetes_namespace`: The namespace of the ingress object.
* `__meta_kubernetes_ingress_name`: The name of the ingress object.
* `__meta_kubernetes_ingress_label_<labelname>`: The label of the ingress object.
* `__meta_kubernetes_ingress_annotation_<annotationname>`: The annotation of the ingress object.
* `__meta_kubernetes_ingress_scheme`: Protocol scheme of ingress, `https` if TLS
config is set. Defaults to `http`.
* `__meta_kubernetes_ingress_path`: Path from ingress spec. Defaults to `/`.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
See below for the configuration options for Kubernetes discovery:
```yaml
# The information to access the Kubernetes API.
# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]
# The Kubernetes role of entities that should be discovered.
role: <role>
# Optional authentication information used to authenticate to the API server.
# Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
# mutually exclusive.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Optional HTTP basic authentication information.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Optional bearer token authentication information.
[ bearer_token: <secret> ]
# Optional bearer token file authentication information.
[ bearer_token_file: <filename> ]
# TLS configuration.
tls_config:
[ <tls_config> ]
# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
names:
[ - <string> ]
```
Added missing ingress from role list (#3666)
2018-01-08 13:23:01 -08:00
Where `<role>` must be `endpoints`, `service`, `pod`, `node`, or
`ingress`.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
See [this example Prometheus configuration file](/documentation/examples/prometheus-kubernetes.yml)
for a detailed example of configuring Prometheus for Kubernetes.
You may wish to check out the 3rd party [Prometheus Operator](https://github.com/coreos/prometheus-operator),
which automates the Prometheus setup on top of Kubernetes.
### `<marathon_sd_config>`
Marathon SD configurations allow retrieving scrape targets using the
[Marathon](https://mesosphere.github.io/marathon/) REST API. Prometheus
will periodically check the REST endpoint for currently running tasks and
create a target group for every app that has at least one healthy task.
The following meta labels are available on targets during [relabeling](#relabel_config):
* `__meta_marathon_app`: the name of the app (with slashes replaced by dashes)
* `__meta_marathon_image`: the name of the Docker image used (if available)
* `__meta_marathon_task`: the ID of the Mesos task
* `__meta_marathon_app_label_<labelname>`: any Marathon labels attached to the app
* `__meta_marathon_port_definition_label_<labelname>`: the port definition labels
* `__meta_marathon_port_mapping_label_<labelname>`: the port mapping labels
Include 1.8 changes in configuration docs
2017-10-10 10:55:29 -07:00
* `__meta_marathon_port_index`: the port index number (e.g. `1` for `PORT1`)
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
See below for the configuration options for Marathon discovery:
```yaml
# List of URLs to be used to contact Marathon servers.
Tweak marathon wording around clustering.
2017-11-02 06:03:19 -07:00
# You need to provide at least one server URL.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
servers:
- <string>
# Polling interval
[ refresh_interval: <duration> | default = 30s ]
Use common HTTPClientConfig for marathon_sd configuration (#4009) This adds support for basic authentication which closes #3090 The support for specifying the client timeout was removed as discussed in https://github.com/prometheus/common/pull/123. Marathon was the only sd mechanism doing this and configuring the timeout is done through `Context`. DC/OS uses a custom `Authorization` header for authenticating. This adds 2 new configuration properties to reflect this. Existing configuration files that use the bearer token will no longer work. More work is required to make this backwards compatible.
2018-04-05 01:08:18 -07:00
# Optional authentication information for token-based authentication
# https://docs.mesosphere.com/1.11/security/ent/iam-api/#passing-an-authentication-token
# It is mutually exclusive with `auth_token_file` and other authentication mechanisms.
[ auth_token: <secret> ]
# Optional authentication information for token-based authentication
# https://docs.mesosphere.com/1.11/security/ent/iam-api/#passing-an-authentication-token
# It is mutually exclusive with `auth_token` and other authentication mechanisms.
[ auth_token_file: <filename> ]
# Sets the `Authorization` header on every request with the
# configured username and password.
# This is mutually exclusive with other authentication mechanisms.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Use common HTTPClientConfig for marathon_sd configuration (#4009) This adds support for basic authentication which closes #3090 The support for specifying the client timeout was removed as discussed in https://github.com/prometheus/common/pull/123. Marathon was the only sd mechanism doing this and configuring the timeout is done through `Context`. DC/OS uses a custom `Authorization` header for authenticating. This adds 2 new configuration properties to reflect this. Existing configuration files that use the bearer token will no longer work. More work is required to make this backwards compatible.
2018-04-05 01:08:18 -07:00
basic_auth:
[ username: <string> ]
[ password: <string> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Use common HTTPClientConfig for marathon_sd configuration (#4009) This adds support for basic authentication which closes #3090 The support for specifying the client timeout was removed as discussed in https://github.com/prometheus/common/pull/123. Marathon was the only sd mechanism doing this and configuring the timeout is done through `Context`. DC/OS uses a custom `Authorization` header for authenticating. This adds 2 new configuration properties to reflect this. Existing configuration files that use the bearer token will no longer work. More work is required to make this backwards compatible.
2018-04-05 01:08:18 -07:00
# Sets the `Authorization` header on every request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file` and other authentication mechanisms.
# NOTE: The current version of DC/OS marathon (v1.11.0) does not support standard Bearer token authentication. Use `auth_token` instead.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token` and other authentication mechanisms.
# NOTE: The current version of DC/OS marathon (v1.11.0) does not support standard Bearer token authentication. Use `auth_token_file` instead.
[ bearer_token_file: /path/to/bearer/token/file ]
# TLS configuration for connecting to marathon servers
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```
By default every app listed in Marathon will be scraped by Prometheus. If not all
of your services provide Prometheus metrics, you can use a Marathon label and
docs - refer to documentation/examples/prometheus-marathon.yml Signed-off-by: Timo Beckers <timo@incline.eu>
2018-10-25 08:56:19 -07:00
Prometheus relabeling to control which instances will actually be scraped.
See [the Prometheus marathon-sd configuration file](/documentation/examples/prometheus-marathon.yml)
for a practical example on how to set up your Marathon app and your Prometheus
configuration.
By default, all apps will show up as a single job in Prometheus (the one specified
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
in the configuration file), which can also be changed using relabeling.
### `<nerve_sd_config>`
Integrate changes from prometheus/docs
2017-10-26 06:42:07 -07:00
Nerve SD configurations allow retrieving scrape targets from [AirBnB's Nerve]
(https://github.com/airbnb/nerve) which are stored in
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
[Zookeeper](https://zookeeper.apache.org/).
The following meta labels are available on targets during [relabeling](#relabel_config):
* `__meta_nerve_path`: the full path to the endpoint node in Zookeeper
* `__meta_nerve_endpoint_host`: the host of the endpoint
* `__meta_nerve_endpoint_port`: the port of the endpoint
* `__meta_nerve_endpoint_name`: the name of the endpoint
```yaml
# The Zookeeper servers.
servers:
- <host>
# Paths can point to a single service, or the root of a tree of services.
paths:
- <string>
[ timeout: <duration> | default = 10s ]
```
### `<serverset_sd_config>`
Integrate changes from prometheus/docs
2017-10-26 06:42:07 -07:00
Serverset SD configurations allow retrieving scrape targets from [Serversets]
(https://github.com/twitter/finagle/tree/master/finagle-serversets) which are
stored in [Zookeeper](https://zookeeper.apache.org/). Serversets are commonly
used by [Finagle](https://twitter.github.io/finagle/) and
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
[Aurora](http://aurora.apache.org/).
The following meta labels are available on targets during relabeling:
* `__meta_serverset_path`: the full path to the serverset member node in Zookeeper
* `__meta_serverset_endpoint_host`: the host of the default endpoint
* `__meta_serverset_endpoint_port`: the port of the default endpoint
* `__meta_serverset_endpoint_host_<endpoint>`: the host of the given endpoint
* `__meta_serverset_endpoint_port_<endpoint>`: the port of the given endpoint
* `__meta_serverset_shard`: the shard number of the member
* `__meta_serverset_status`: the status of the member
```yaml
# The Zookeeper servers.
servers:
- <host>
# Paths can point to a single serverset, or the root of a tree of serversets.
paths:
- <string>
[ timeout: <duration> | default = 10s ]
```
Serverset data must be in the JSON format, the Thrift format is not currently supported.
### `<triton_sd_config>`
[Triton](https://github.com/joyent/triton) SD configurations allow retrieving
scrape targets from [Container Monitor](https://github.com/joyent/rfd/blob/master/rfd/0027/README.md)
discovery endpoints.
The following meta labels are available on targets during relabeling:
Add ability to filter triton_sd targets by pre-defined groups (#4701) Additionally, add triton groups metadata to the discovery reponse and correct a documentation error regarding the triton server id metadata. Signed-off-by: Richard Kiene <richard.kiene@joyent.com>
2018-10-10 02:03:34 -07:00
* `__meta_triton_groups`: the list of groups belonging to the target joined by a comma separator
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_triton_machine_alias`: the alias of the target container
Add ability to filter triton_sd targets by pre-defined groups (#4701) Additionally, add triton groups metadata to the discovery reponse and correct a documentation error regarding the triton server id metadata. Signed-off-by: Richard Kiene <richard.kiene@joyent.com>
2018-10-10 02:03:34 -07:00
* `__meta_triton_machine_brand`: the brand of the target container
* `__meta_triton_machine_id`: the UUID of the target container
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
* `__meta_triton_machine_image`: the target containers image type
Add ability to filter triton_sd targets by pre-defined groups (#4701) Additionally, add triton groups metadata to the discovery reponse and correct a documentation error regarding the triton server id metadata. Signed-off-by: Richard Kiene <richard.kiene@joyent.com>
2018-10-10 02:03:34 -07:00
* `__meta_triton_server_id`: the server UUID for the target container
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```yaml
# The information to access the Triton discovery API.
# The account to use for discovering new target containers.
account: <string>
# The DNS suffix which should be applied to target containers.
dns_suffix: <string>
# The Triton discovery endpoint (e.g. 'cmon.us-east-3b.triton.zone'). This is
# often the same value as dns_suffix.
endpoint: <string>
Add ability to filter triton_sd targets by pre-defined groups (#4701) Additionally, add triton groups metadata to the discovery reponse and correct a documentation error regarding the triton server id metadata. Signed-off-by: Richard Kiene <richard.kiene@joyent.com>
2018-10-10 02:03:34 -07:00
# A list of groups for which targets are retrieved. If omitted, all containers
# available to the requesting account are scraped.
groups:
[ - <string> ... ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# The port to use for discovery and metric scraping.
[ port: <int> | default = 9163 ]
# The interval which should should be used for refreshing target containers.
[ refresh_interval: <duration> | default = 60s ]
# The Triton discovery API version.
[ version: <int> | default = 1 ]
# TLS configuration.
tls_config:
[ <tls_config> ]
```
### `<static_config>`
A `static_config` allows specifying a list of targets and a common label set
for them. It is the canonical way to specify static targets in a scrape
configuration.
```yaml
# The targets specified by the static config.
targets:
[ - '<host>' ]
# Labels assigned to all metrics scraped from the targets.
labels:
[ <labelname>: <labelvalue> ... ]
```
### `<relabel_config>`
Relabeling is a powerful tool to dynamically rewrite the label set of a target before
it gets scraped. Multiple relabeling steps can be configured per scrape configuration.
They are applied to the label set of each target in order of their appearance
in the configuration file.
Initially, aside from the configured per-target labels, a target's `job`
label is set to the `job_name` value of the respective scrape configuration.
The `__address__` label is set to the `<host>:<port>` address of the target.
After relabeling, the `instance` label is set to the value of `__address__` by default if
it was not set during relabeling. The `__scheme__` and `__metrics_path__` labels
are set to the scheme and metrics path of the target respectively. The `__param_<name>`
label is set to the value of the first passed URL parameter called `<name>`.
Additional labels prefixed with `__meta_` may be available during the
relabeling phase. They are set by the service discovery mechanism that provided
the target and vary between mechanisms.
Labels starting with `__` will be removed from the label set after relabeling is completed.
If a relabeling step needs to store a label value only temporarily (as the
input to a subsequent relabeling step), use the `__tmp` label name prefix. This
prefix is guaranteed to never be used by Prometheus itself.
```yaml
# The source labels select values from existing labels. Their content is concatenated
# using the configured separator and matched against the configured regular expression
# for the replace, keep, and drop actions.
[ source_labels: '[' <labelname> [, ...] ']' ]
# Separator placed between concatenated source label values.
[ separator: <string> | default = ; ]
# Label to which the resulting value is written in a replace action.
# It is mandatory for replace actions. Regex capture groups are available.
[ target_label: <labelname> ]
# Regular expression against which the extracted value is matched.
[ regex: <regex> | default = (.*) ]
# Modulus to take of the hash of the source label values.
[ modulus: <uint64> ]
# Replacement value against which a regex replace is performed if the
# regular expression matches. Regex capture groups are available.
[ replacement: <string> | default = $1 ]
# Action to perform based on regex matching.
[ action: <relabel_action> | default = replace ]
```
`<regex>` is any valid
[RE2 regular expression](https://github.com/google/re2/wiki/Syntax). It is
required for the `replace`, `keep`, `drop`, `labelmap`,`labeldrop` and `labelkeep` actions. The regex is
anchored on both ends. To un-anchor the regex, use `.*<regex>.*`.
`<relabel_action>` determines the relabeling action to take:
* `replace`: Match `regex` against the concatenated `source_labels`. Then, set
`target_label` to `replacement`, with match group references
(`${1}`, `${2}`, ...) in `replacement` substituted by their value. If `regex`
does not match, no replacement takes place.
* `keep`: Drop targets for which `regex` does not match the concatenated `source_labels`.
* `drop`: Drop targets for which `regex` matches the concatenated `source_labels`.
* `hashmod`: Set `target_label` to the `modulus` of a hash of the concatenated `source_labels`.
* `labelmap`: Match `regex` against all label names. Then copy the values of the matching labels
to label names given by `replacement` with match group references
(`${1}`, `${2}`, ...) in `replacement` substituted by their value.
* `labeldrop`: Match `regex` against all label names. Any label that matches will be
removed from the set of labels.
* `labelkeep`: Match `regex` against all label names. Any label that does not match will be
removed from the set of labels.
Care must be taken with `labeldrop` and `labelkeep` to ensure that metrics are still uniquely labeled
once the labels are removed.
### `<metric_relabel_configs>`
Metric relabeling is applied to samples as the last step before ingestion. It
has the same configuration format and actions as target relabeling. Metric
relabeling does not apply to automatically generated timeseries such as `up`.
One use for this is to blacklist time series that are too expensive to ingest.
### `<alert_relabel_configs>`
Alert relabeling is applied to alerts before they are sent to the Alertmanager.
It has the same configuration format and actions as target relabeling. Alert
relabeling is applied after external labels.
One use for this is ensuring a HA pair of Prometheus servers with different
external labels send identical alerts.
### `<alertmanager_config>`
An `alertmanager_config` section specifies Alertmanager instances the Prometheus server sends
alerts to. It also provides parameters to configure how to communicate with these Alertmanagers.
Alertmanagers may be statically configured via the `static_configs` parameter or
dynamically discovered using one of the supported service-discovery mechanisms.
Additionally, `relabel_configs` allow selecting Alertmanagers from discovered
entities and provide advanced modifications to the used API path, which is exposed
through the `__alerts_path__` label.
```yaml
# Per-target Alertmanager timeout when pushing alerts.
[ timeout: <duration> | default = 10s ]
# Prefix for the HTTP path alerts are pushed to.
[ path_prefix: <path> | default = / ]
# Configures the protocol scheme used for requests.
[ scheme: <scheme> | default = http ]
# Sets the `Authorization` header on every request with the
# configured username and password.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
basic_auth:
[ username: <string> ]
[ password: <string> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Sets the `Authorization` header on every request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the scrape request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
# List of Azure service discovery configurations.
azure_sd_configs:
[ - <azure_sd_config> ... ]
# List of Consul service discovery configurations.
consul_sd_configs:
[ - <consul_sd_config> ... ]
# List of DNS service discovery configurations.
dns_sd_configs:
[ - <dns_sd_config> ... ]
# List of EC2 service discovery configurations.
ec2_sd_configs:
[ - <ec2_sd_config> ... ]
# List of file service discovery configurations.
file_sd_configs:
[ - <file_sd_config> ... ]
# List of GCE service discovery configurations.
gce_sd_configs:
[ - <gce_sd_config> ... ]
# List of Kubernetes service discovery configurations.
kubernetes_sd_configs:
[ - <kubernetes_sd_config> ... ]
# List of Marathon service discovery configurations.
marathon_sd_configs:
[ - <marathon_sd_config> ... ]
# List of AirBnB's Nerve service discovery configurations.
nerve_sd_configs:
[ - <nerve_sd_config> ... ]
# List of Zookeeper Serverset service discovery configurations.
serverset_sd_configs:
[ - <serverset_sd_config> ... ]
# List of Triton service discovery configurations.
triton_sd_configs:
[ - <triton_sd_config> ... ]
# List of labeled statically configured Alertmanagers.
static_configs:
[ - <static_config> ... ]
# List of Alertmanager relabel configurations.
relabel_configs:
[ - <relabel_config> ... ]
```
### `<remote_write>`
`write_relabel_configs` is relabeling applied to samples before sending them
to the remote endpoint. Write relabeling is applied after external labels. This
could be used to limit which samples are sent.
There is a [small demo](/documentation/examples/remote_storage) of how to use
this functionality.
```yaml
# The URL of the endpoint to send samples to.
url: <string>
# Timeout for requests to the remote write endpoint.
[ remote_timeout: <duration> | default = 30s ]
# List of remote write relabel configurations.
write_relabel_configs:
[ - <relabel_config> ... ]
# Sets the `Authorization` header on every remote write request with the
# configured username and password.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
basic_auth:
[ username: <string> ]
[ password: <string> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Sets the `Authorization` header on every remote write request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every remote write request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the remote write request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
document remote write queue parameters (#4126)
2018-04-30 11:08:45 -07:00
# Configures the queue used to write to remote storage.
queue_config:
# Number of samples to buffer per shard before we start dropping them.
Update remote_write queue docs to present defaults. (#4715) Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-10 10:51:27 -07:00
[ capacity: <int> | default = 10000 ]
document remote write queue parameters (#4126)
2018-04-30 11:08:45 -07:00
# Maximum number of shards, i.e. amount of concurrency.
[ max_shards: <int> | default = 1000 ]
# Maximum number of samples per send.
[ max_samples_per_send: <int> | default = 100]
# Maximum time a sample will wait in buffer.
[ batch_send_deadline: <duration> | default = 5s ]
# Maximum number of times to retry a batch on recoverable errors.
Update remote_write queue docs to present defaults. (#4715) Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-10 10:51:27 -07:00
[ max_retries: <int> | default = 3 ]
document remote write queue parameters (#4126)
2018-04-30 11:08:45 -07:00
# Initial retry delay. Gets doubled for every retry.
[ min_backoff: <duration> | default = 30ms ]
# Maximum retry delay.
[ max_backoff: <duration> | default = 100ms ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
```
There is a list of
Add prefix to relative 3 URLs (#3551)
2017-12-06 13:16:53 -08:00
[integrations](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
with this feature.
### `<remote_read>`
```yaml
# The URL of the endpoint to query from.
url: <string>
Add remote read filter option For special remote read endpoints which have only data for specific queries, it is desired to limit the number of queries sent to the configured remote read endpoint to reduce latency and performance overhead.
2017-11-11 17:23:20 -08:00
# An optional list of equality matchers which have to be
# present in a selector to query the remote read endpoint.
required_matchers:
[ <labelname>: <labelvalue> ... ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Timeout for requests to the remote read endpoint.
docs: Fix remote_read/remote_timeout default (#3829)
2018-02-12 04:52:33 -08:00
[ remote_timeout: <duration> | default = 1m ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
Fix default of read_recent to be false. (#3617) This is what is documented in the migration guide, and the default settings should make sense for a true long term storage. Document the setting.
2017-12-23 09:21:38 -08:00
# Whether reads should be made for queries for time ranges that
# the local storage should have complete data for.
[ read_recent: <boolean> | default = false ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Sets the `Authorization` header on every remote read request with the
# configured username and password.
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
# password and password_file are mutually exclusive.
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
basic_auth:
[ username: <string> ]
[ password: <string> ]
support reading basic_auth password_file for HTTP basic auth (#4077) Issue: https://github.com/prometheus/prometheus/issues/4076 Signed-off-by: Adam Shannon <adamkshannon@gmail.com>
2018-04-25 10:19:06 -07:00
[ password_file: <string> ]
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
# Sets the `Authorization` header on every remote read request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]
# Sets the `Authorization` header on every remote read request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]
# Configures the remote read request's TLS settings.
tls_config:
[ <tls_config> ]
# Optional proxy URL.
[ proxy_url: <string> ]
```
There is a list of
Add prefix to relative 3 URLs (#3551)
2017-12-06 13:16:53 -08:00
[integrations](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
Import first batch of Prometheus documentation In order to provide documentation for each individual version, this commit starts moving Prometheus server specific documentation into the repository itself.
2017-10-10 05:58:52 -07:00
with this feature.
Reference in a new issue Copy permalink