Commit graph

67 commits

Author SHA1 Message Date
machine424 94d28cd6cf chore(notifier): add a reproducer for https://github.com/prometheus/prometheus/issues/13676
to show "targets groups update" starvation when the notifications queue is full and an Alertmanager
is down.

The existing `TestHangingNotifier` that was added in https://github.com/prometheus/prometheus/pull/10948 doesn't really reflect the reality as the SD changes are manually fed into `syncCh` in a continuous way, whereas in reality, updates are only resent every `updatert`.

The test added here sets up an SD manager and links it to the notifier. The SD changes will be triggered by that manager as it's done in reality.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>

Co-authored-by: Ethan Hunter <ehunter@hudson-trading.com>
2024-06-19 09:43:52 +02:00
David Ashpole bbfc72b4e2
support unregistering discovery manager metrics (#13896)
Signed-off-by: David Ashpole <dashpole@google.com>
2024-04-05 16:19:07 +02:00
Paulin Todev 78411d5e8b
SD Managers taking over responsibility for registration of debug metrics (#13375)
SD Managers take over responsibility for SD metrics registration

---------

Signed-off-by: Paulin Todev <paulin.todev@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-01-23 16:53:55 +01:00
Paulin Todev 6a5306a53c
Use const labels for Discovery Manager metrics.
Signed-off-by: Paulin Todev <paulin.todev@gmail.com>
2023-12-11 11:14:27 +00:00
Paulin Todev 6de80d7fb0
Allow non-default registry to be used for metrics of SD components
Signed-off-by: Paulin Todev <paulin.todev@gmail.com>
2023-12-11 11:14:26 +00:00
Oleksandr Redko fa90ca46e5 ci(lint): enable godot; append dot at the end of comments
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 19:53:38 +02:00
Julien Pivotto 4b735f02a6
Merge pull request #10569 from zzJinux/discovery-manager-run
Fix discovery managers to be properly cancelled
2023-09-29 12:07:55 +02:00
Julien Pivotto 009017a3fb Revert "Remove deleted target from discovery manager"
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-08-14 23:29:39 +02:00
haleyao c5a37ddad5 Remove deleted target from discovery manager
Signed-off-by: haleyao <haleyao@tencent.com>
2023-07-10 00:09:25 +08:00
Bryan Boreham 2f58be840d service discovery: add config name to log messages
This makes it easier to connect a log message with the config it relates
to.

Each SD config has a name, either the scrape job name or something like
"config-0" for Alertmanager config.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-12 11:30:00 +00:00
Sebastian Poxhofer 3f9a9d1e62
chore(discoveryManager): expose Discoverer refresh function (#10531)
Signed-off-by: secustor <sebastian@poxhofer.at>
2022-06-13 21:06:15 +02:00
Jinwook Jeong c7c7847b6f Fix discovery managers to be properly cancelled
Signed-off-by: Jinwook Jeong <vustthat@gmail.com>
2022-04-09 01:12:46 +09:00
Robert Fratto 44a5e705be
discovery: Expose custom HTTP client options to discoverers (#10462)
* discovery: expose HTTP client options to discoverers

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* discovery/http: use HTTP client options for created client

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* scrape: use a list of HTTP client options instead of just dial context

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* discovery: rephrase comment

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
2022-03-24 18:16:59 -04:00
Julien Pivotto 9621c2c0cc
Fix race with targets update during ApplyConfig (#9656)
I ended up extending the lock so refTargets remains valid for the
duration of the update.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-05 01:13:04 +01:00
Vladimir Kononov 1043d2b594
Discovery: abstain from restarting providers if possible (#9321) (#9349)
* Abstain from restarting discovery providers if possible (#9321)

Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>
2021-10-20 10:16:20 +02:00
Julien Pivotto 432005826d
Add a feature flag to enable the new discovery manager (#9537)
* Add a feature flag to enable the new manager

This PR creates a copy of the legacy manager and uses it by default.

It is a companion PR to #9349. With this PR, users can enable the new
discovery manager and provide us with any feedback / side effects that
the new behaviour might have.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-10-20 10:15:54 +02:00
Levi Harrison b5f6f8fb36 Switched to go-kit/log
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-11 12:28:36 -04:00
Julien Pivotto e1774b6f83 Fix the computation of prometheus_sd_discovered_targets
prometheus_sd_discovered_targets is wrongly calculated when there are
multiple SD configurations in place. One discovery manager can have
multiple groups coming from multiple service discoveries.

When multiple service discovery configs are used, we do not compute the
metric correctly, and instead just set the metric to one of the service
discoveries.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-05-14 22:38:37 +02:00
Andy Bursavich 4e6a94a27d
Invert service discovery dependencies (#7701)
This also fixes a bug in query_log_file, which now is relative to the config file like all other paths.

Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2020-08-20 13:48:26 +01:00
Julien Pivotto 59de58d380
Docker Swarm service discovery (#7420)
* Docker Swarm service discovery

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-06-26 12:25:58 +02:00
Julien Pivotto c61141ce51
Add DigitalOcean service discovery (#7407)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-06-18 17:04:41 +02:00
Marek Slabicki 8224ddec23
Capitalizing first letter of all log lines (#7043)
Signed-off-by: Marek Slabicki <thaniri@gmail.com>
2020-04-11 09:22:18 +01:00
Julien Pivotto c67f81937c
discovery: updateGroup should not create targets[poolKey] in the loop (#6903)
We can assume that not all target groups are nil in normal scernarios,
so we can create targets[poolKey] outside the loop.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-03-02 07:35:02 +00:00
johncming 17683d074c discovery: fix bug that use rlock for read. (#5928)
Signed-off-by: johncming <johncming@yahoo.com>
2020-01-22 09:57:37 +00:00
Nevill 55661ab004 Set failedConfigs only once right after registerProviders finished
Signed-off-by: Nevill <nevill.dutt@gmail.com>
2019-09-24 09:15:40 +08:00
Nevill 048f81218d Change prometheus_sd_configs_failed_total to Gauge
Signed-off-by: Nevill <nevill.dutt@gmail.com>
2019-09-16 10:38:43 +08:00
Harkishen Singh d98d4a9bf0 remove resetting of manager properties and init manager props under locking (#5979)
Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2019-09-06 12:46:24 +02:00
Matt Layher 302148fd69 *: apply gofmt -s
Signed-off-by: Matt Layher <mdlayher@gmail.com>
2019-01-16 17:28:14 -05:00
Ilya Gladyshev 922c17e119 added name label to all discovery metrics (#5002)
Signed-off-by: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>
2018-12-20 14:47:29 +00:00
Simon Pasquier 8b91d39c43
discovery: send empty group on empty SD config (#4819)
* discovery: send empty group on blank SD config

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Update comments

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Add another comment

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-11-30 17:59:57 +01:00
Simon Pasquier a30348f1a4 discovery: add config label to discovered targets metric (#4753)
* discovery: add labels to discovered targets metric

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-10-18 16:46:59 +01:00
Goutham Veeramachaneni ffb7f829ec
Merge pull request #4730 from prometheus/release-2.4
Release 2.4
2018-10-12 14:15:42 -07:00
Simon Pasquier 657199af22 Address Krasi comments
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-28 12:29:24 +02:00
Simon Pasquier 5df757fdd4 zookeeper: fix panic
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-28 11:39:40 +02:00
Simon Pasquier 365931ea83 discovery: add metrics + send updates from one goroutine only
The added metrics are:

* prometheus_sd_discovered_targets
* prometheus_sd_received_updates_total
* prometheus_sd_updates_delayed_total
* prometheus_sd_updates_total

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-27 15:59:42 +02:00
Simon Pasquier 48989d8996 discovery: add more tests
Co-authored-by: Camille Janicki <camille.janicki@gmail.com>
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-12 16:13:15 +02:00
Krasi Georgiev ba7eb733e8 tidy up the discovery logs,updating loops and selects (#4556)
* tidy up the discovery logs,updating loops and selects

few objects renamings

removed a very noise debug log on the k8s discovery. It would be usefull
to show some summary rather than every update as this is impossible to
follow.

added most comments as debug logs so each block becomes self
explanatory.

when the discovery receiving channel is full will retry again on the
next cycle.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* add noop logger for the SD manager tests.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* spelling nits

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-05 17:02:47 +05:30
Simon Pasquier 674c76adb8 discovery: coalesce identical SD configurations (#3912)
* discovery: coalesce identical SD configurations

Instead of creating as many SD providers as declared in the
configuration, the discovery manager merges identical configurations
into the same provider and keeps track of the subscribers. When
the manager receives target updates from a SD provider, it will
broadcast the updates to all interested subscribers.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-01 08:51:31 +01:00
Krasi Georgiev 53691ae261 Simplify SD update throttling (#4523)
* simplfied SD updates throtling

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* add default to catch cases when we don't have new updates.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-08-27 17:12:11 +02:00
Paul Gier d24d2acd11 config: set target group source index during unmarshalling (#4245)
* config: set target group source index during unmarshalling

Fixes issue #4214 where the scrape pool is unnecessarily reloaded for a
config reload where the config hasn't changed.  Previously, the discovery
manager changed the static config after loading which caused the in-memory
config to differ from a freshly reloaded config.

Signed-off-by: Paul Gier <pgier@redhat.com>

* [issue #4214] Test that static targets are not modified by discovery manager

Signed-off-by: Paul Gier <pgier@redhat.com>
2018-06-13 16:34:59 +01:00
Callum Styan 03578d5df8 add example usage of SD adapter for converting unsupported SD type to filesd (#3720)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-05-30 13:14:34 +01:00
ferhat elmas ffa673f7d8 General simplifications (#3887)
Another try as in #1516
2018-02-26 07:58:10 +00:00
Krasi Georgiev acc4197098 remove dicovery race for the context field 2018-01-29 15:18:07 +00:00
Frederic Branczyk 73e829137b
discovery: Cleanup ticker 2018-01-29 13:51:04 +01:00
Frederic Branczyk cfa0253ed8
discovery: Schedule updates to throttle 2018-01-26 16:24:44 +01:00
Krasi Georgiev ec26751fd2 use mutexes for the discovery manager instead of a loop as this was a stupid idea 2018-01-17 18:12:58 +00:00
Krasi Georgiev d12e6f29fc discovery manager ApplyConfig now takes a direct ServiceDiscoveryConfig so that it can be used for the notify manager
reimplement the service discovery for the notify manager

Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>
2018-01-15 13:39:44 +00:00
Krasi Georgiev 790cf30fcb remove uneeded check 2018-01-15 11:52:20 +00:00
Krasi Georgiev 38938ba493 comment nits 2018-01-15 11:47:36 +00:00
Krasi Georgiev febebcd49a more comments for the future ME, and reverted the Discovery manager execution changes as these were correct in the first place 2018-01-12 22:07:21 +00:00