Commit graph

317 commits

Author SHA1 Message Date
Witek Bedyk 14986e52cf
Fix Uyuni SD initialization (#9924)
* Fix Uyuni SD initialization

The change prevents null pointer exception during SD initialization.

Signed-off-by: Witek Bedyk <witold.bedyk@suse.com>
2021-12-04 21:14:47 +01:00
Bryan Boreham 1ed94142fc
remote-write: slow down retries to avoid DDOS (#9634)
* remote-write: slow down retries to avoid DDOS

Increase the default max retry time from 100ms to 5 seconds.

Remote write calls are retried after a recoverable error such as the
back-end returning 500. Prometheus waits the minimum time and retries,
then doubles the wait on each subsequent retry until the maximum is
reached.

If some data is still getting through, remote-write will also increase
shards, and the default maximum is 200. 200 shards sending every 100ms
is 20 calls per second, to a back-end that is already in trouble.

5 seconds was chosen to match the default BatchSendDeadline: if we can
afford to wait that long for no response, then we can wait the same time
to retry. We will reach 5 seconds after 9 successive failures.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Update config doc for max_backoff change

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-11-09 14:08:24 -08:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Mateusz Gozdek 1a6c2283a3 Format Go source files using 'gofumpt -w -s -extra'
Part of #9557

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Arthur Silva Sens be2599c853
config: Make remote-write required for Agent mode (#9618)
* config: Make remote-write required for Agent mode

Signed-off-by: ArthurSens <arthursens2005@gmail.com>
2021-10-30 01:41:40 +02:00
Julien Pivotto 77f411b2ec
Enable tls_config in oauth2 (#9550)
* Enable tls_config in oauth2

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-10-20 23:10:18 +02:00
Levi Harrison 89a6ebd799
Add common HTTP client to Azure SD (#9267)
* Add `proxy_url` option to Azure SD

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-10-20 23:09:58 +02:00
Witek Bedyk cda2dbbef6
Add Uyuni service discovery (#8190)
* Add Uyuni service discovery

Signed-off-by: Witek Bedyk <witold.bedyk@suse.com>

Co-authored-by: Joao Cavalheiro <jcavalheiro@suse.de>
Co-authored-by: Marcelo Chiaradia <mchiaradia@suse.com>
Co-authored-by: Stefano Torresi <stefano@torresi.io>
Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
2021-10-19 01:00:44 +02:00
Julien Pivotto 9d65017798 config: fix puppetdb tests
This PR fixes the tests in main. The last merge introduced a failing
test in the config package.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-09-16 18:31:28 +02:00
Julien Pivotto 8920024323 Add PuppetDB service discovery
We have been Puppet user for 10 years and we are users of
https://github.com/camptocamp/prometheus-puppetdb-sd

However, that file_sd implementation contains business logic and
assumptions around e.g. the modules which you are using.

This pull request adds a simple PuppetDB service discovery, which will
enable more use cases than the upstream sd.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-09-16 16:54:26 +02:00
DrAuYueng e8be1d0a5c
Check relabel action at yaml unmarshal stage (#9224)
Signed-off-by: DrAuYueng <ouyang1204@gmail.com>
2021-08-31 17:52:57 +02:00
SuperQ e167a45c65
Add new Go build tags.
Add new go:build comments based on 1.17 formatting[0].

[0]: https://golang.org/doc/go1.17#gofmt

Signed-off-by: SuperQ <superq@gmail.com>
2021-08-27 10:24:14 +02:00
Levi Harrison bd57cd395e
Switch to common/sigv4
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-08-26 09:37:19 -04:00
Levi Harrison c1b1b826ce HostNetworkHost -> HostNetworkingHost
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-08-03 05:58:49 -06:00
Levi Harrison 89f154d643 Added tests
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-07-27 23:33:28 -04:00
austin ce bbc951f50b
Add config tests for kuma SD
Signed-off-by: austin ce <austin.cawley@gmail.com>
2021-07-21 12:55:02 -04:00
Martin Disibio 1bcd13d6b5
Exemplar resize (#8974)
* Create experimental circular buffer resize method, benchmarks

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Optimize exemplar resize to only replay as many exemplars as needed

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* More comments, benchmark AddExemplar

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* optimizations

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* comment

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Slight refactor of resize benchmark + make use of resize via runtime
reloadable storage config.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Some more config related changes.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Refactor to remove usage of noopExemplarStorage and avoid race condition
when resizing from Head code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix or add comments to clarify some of the new behaviour.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* fix potential panics related to negative exemplar buffer lengths

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
2021-07-20 10:22:57 +05:30
Ben Kochie ecc5960150
Merge pull request #8991 from roidelapluie/yamllint
Fix yaml indent to make CI happy
2021-06-25 10:04:41 +02:00
Julien Pivotto 17700e5600 Fix yaml indent to make CI happy
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-06-25 00:53:22 +02:00
Levi Harrison d5c3c567d3
Remote Write: Add max samples per metadata send (#8959)
* Added MaxSamplesPerSend

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added tests

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed order of require

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added docs

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* writes -> writesReceived

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Improved send loop

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-24 15:39:50 -07:00
Ben Kochie 7cb55d5732
Merge pull request #8802 from mwasilew2/yaml-linting
Adds yamllinting to Makefile.common
2021-06-24 15:59:35 +02:00
3Xpl0it3r a0bac4b488
add kubeconfig support in discovery module (#8811)
Signed-off-by: 3Xpl0it3r <shouc.wang@hotmail.com>
2021-06-17 12:41:50 +02:00
Michal Wasilewski 3f686cad8b
fixes yamllint errors
Signed-off-by: Michal Wasilewski <mwasilewski@gmx.com>
2021-06-12 12:47:47 +02:00
Levi Harrison faed8df31d
Enable reading consul token from file (#8926)
* Adopted common http client

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-12 00:06:59 +02:00
Julien Pivotto c0c22ed042
Merge pull request #8927 from LeviHarrison/move-to-go-kit/log
Migrate From `go-kit/kit/log` to `go-kit/log`
2021-06-11 21:15:56 +02:00
Levi Harrison b5f6f8fb36 Switched to go-kit/log
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-11 12:28:36 -04:00
Julien Pivotto 9444698ae2
http_sd (#8839)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-06-11 18:04:45 +02:00
Julien Pivotto 20c6739adc
Merge pull request #8833 from hanjm/feature/add-scape-read-body-limit
Add body_size_limit to prevent bad targets response large body cause Prometheus server OOM (#8827)
2021-06-02 09:24:59 +02:00
TJ Hoplock dc22c65349
Add Linode Service Discovery (#8846)
* Add Linode Service Discovery

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2021-06-01 20:32:36 +02:00
hanjm 1df05bfd49 Add body_size_limit to prevent bad targets response large body cause Prometheus server OOM (#8827)
Signed-off-by: hanjm <hanjinming@outlook.com>
2021-05-29 07:05:42 +08:00
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
Damien Grisonnet b50f9c1c84
Add label scrape limits (#8777)
* scrape: add label limits per scrape

Add three new limits to the scrape configuration to provide some
mechanism to defend against unbound number of labels and excessive
label lengths. If any of these limits are broken by a sample from a
scrape, the whole scrape will fail. For all of these configuration
options, a zero value means no limit.

The `label_limit` configuration will provide a mechanism to bound the
number of labels per-scrape of a certain sample to a user defined limit.
This limit will be tested against the sample labels plus the discovery
labels, but it will exclude the __name__ from the count since it is a
mandatory Prometheus label to which applying constraints isn't
meaningful.

The `label_name_length_limit` and `label_value_length_limit` will
prevent having labels of excessive lengths. These limits also skip the
__name__ label for the same reasons as the `label_limit` option and will
also make the scrape fail if any sample has a label name/value length
that exceed the predefined limits.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

* scrape: add metrics and alert to label limits

Add three gauge, one for each label limit to easily access the
limit set by a certain scrape target.
Also add a counter to count the number of targets that exceeded the
label limits and thus were dropped. This is useful for the
`PrometheusLabelLimitHit` alert that will notify the users that scraping
some targets failed because they had samples exceeding the label limits
defined in the scrape configuration.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

* scrape: apply label limits to __name__ label

Apply limits to the __name__ label that was previously skipped and
truncate the label names and values in the error messages as they can be
very very long.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>

* scrape: remove label limits gauges and refactor

Remove `prometheus_target_scrape_pool_label_limit`,
`prometheus_target_scrape_pool_label_name_length_limit`, and
`prometheus_target_scrape_pool_label_value_length_limit` as they are not
really useful since we don't have the information on the labels in it.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2021-05-06 09:56:21 +01:00
Julien Pivotto f3b2d2a998
Fix config tests in main branch (#8767)
The merge of 8761 did not catch that the secrets were off by one
because it was not rebased on top of 8693.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-04-29 00:00:30 +02:00
Levi Harrison fa184a5fc3
Add OAuth 2.0 Config (#8761)
* Introduced oauth2 config into the codebase

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-04-28 14:47:52 +02:00
n888 7c028d59c2
Add lightsail service discovery (#8693)
Signed-off-by: N888 <drifto@gmail.com>
2021-04-28 11:29:12 +02:00
Julien Pivotto 5bce801a09
Rename discovery/dockerswarm to discovery/moby (#8691)
This makes it clear that the dockerswarm package does more than docker
swarm, but does also docker.

I have picked moby as it is the upstream name: https://mobyproject.org/

There is no user-facing change, except in the case of a bad
configuration. Previously, a user who would have a bad docker sd config
would see an error like:

> field xx not found in type dockerswarm.plain

Now that error would be turned into:

> field xx not found in type moby.plain

While not perfect, it should at not be confusing between docker and
dockerswarm.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-04-13 09:33:54 +02:00
Julien Pivotto e635ca834b Add environment variable expansion in external label values
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-30 01:36:28 +02:00
Robert Jacob b253056163
Implement Docker discovery (#8629)
* Implement Docker discovery

Signed-off-by: Robert Jacob <xperimental@solidproject.de>
2021-03-29 22:30:23 +02:00
Julien Pivotto 5a6d244b00 Scaleway SD: Add the ability to read token from file
Prometheus adds the ability to read secrets from files. This add
this feature for the scaleway service discovery.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-25 00:52:33 +01:00
Julien Pivotto 49016994ac Switch to alertmanager api v2
According to the 2.25 release notes, 2.26 should switch to alertmanager
api v2 by default.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-20 01:01:10 +01:00
Rémy Léone f690b811c5
add support for scaleway service discovery (#8555)
Co-authored-by: Patrik <patrik@ptrk.io>
Co-authored-by: Julien Pivotto <roidelapluie@inuits.eu>

Signed-off-by: Rémy Léone <rleone@scaleway.com>
2021-03-10 15:10:17 +01:00
Robert Fratto 5b78aa0649
Contribute grafana/agent sigv4 code (#8509)
* Contribute grafana/agent sigv4 code
* address review feedback
  - move validation logic for RemoteWrite into unmarshal
  - copy configuration fields from ec2 SD config
  - remove enabled field, use pointer for enabling sigv4
* Update config/config.go
* Don't provide credentials if secret key / access key left blank
* Add SigV4 headers to the list of unchangeable headers.
* sigv4: don't include all headers in signature
* only test for equality in the authorization header, not the signed date
* address review feedback
  1. s/httpClientConfigEnabled/httpClientConfigAuthEnabled
  2. bearer_token tuples to "authorization"
  3. Un-export NewSigV4RoundTripper
* add x-amz-content-sha256 to list of unchangeable headers
* Document sigv4 configuration
* add suggestion for using default AWS SDK credentials

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
2021-03-08 12:20:09 -07:00
Julien Pivotto 93c6139bc1 Support follow_redirect
This PR introduces support for follow_redirect, to enable users to
disable following HTTP redirects.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-02-26 22:50:56 +01:00
Harkishen-Singh 79ba53a6c4 Custom headers on remote-read and refactor implementation to roundtripper.
Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2021-02-26 17:20:29 +05:30
Julien Pivotto 8787f0aed7 Update common to support credentials type
Most of the backwards compat tests is done in common.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-02-18 23:28:22 +01:00
Harkishen-Singh 77c20fd2f8 Adds support to configure retry on Rate-Limiting from remote-write config.
Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2021-02-16 14:52:49 +05:30
Nándor István Krácser 509000269a
remote_write: allow passing along custom HTTP headers (#8416)
* remote_write: allow passing along custom HTTP headers

Signed-off-by: Nandor Kracser <bonifaido@gmail.com>

* add warning

Signed-off-by: Nandor Kracser <bonifaido@gmail.com>

* remote_write: add header valadtion

Signed-off-by: Nandor Kracser <bonifaido@gmail.com>

* extend tests for bad remote write headers

Signed-off-by: Nandor Kracser <bonifaido@gmail.com>

* remote_write: add note about the authorization header

Signed-off-by: Nandor Kracser <bonifaido@gmail.com>
2021-02-04 14:18:13 -07:00
Alexey Shumkin 73ddf603af
discovery/kubernetes: Fix valid label selector causing config error
Label selector can be
"set-based"(https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement)
but such a selector causes Prometheus start failure with the "unexpected
error: parsing YAML file ...: invalid selector: 'foo in (bar,baz)';
can't understand 'baz)'"-like error.

This is caused by the `fields.ParseSelector(string)` function that
simply splits an expression as a CSV-list, so a comma confuses such a
parsing method and lead to the error.

Use `labels.Parse(string)` to use a valid lexer to parse a selector
expression.

Closes #8284.

Signed-off-by: Alexey Shumkin <Alex.Crezoff@gmail.com>
2020-12-16 10:56:01 +03:00
gotjosh 4eca4dffb8
Allow metric metadata to be propagated via Remote Write. (#6815)
* Introduce a metadata watcher

Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage.

Signed-off-by: gotjosh <josue@grafana.com>

* Additional fixes after rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Rework samples/metadata metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use more descriptive variable names in MetadataWatcher collect.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix issues caused during rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix missing metric add and unneeded config code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix metrics and docs

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Replace assert with require

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Bring back max_samples_per_send metric

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-11-19 20:53:03 +05:30
Julien Pivotto 3509647462
Docker swarm: add filtering of services (#8074)
* Docker swarm: add filtering of services

Add filters on all docker swarm roles (nodes, tasks and services).

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-11-09 12:41:02 +01:00