442 commits

Author	SHA1	Message	Date
Bartlomiej Plotka	6f1ad64e27	Update scrape/scrape.go ... Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	2025-01-17 09:09:28 +00:00
bwplotka	3119567d5b	scrape: Add metadata for automatic metrics. ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-01-16 19:49:12 +00:00
bwplotka	af928a1ba4	Addressed Krajo's comment. ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-01-16 13:15:56 +00:00
bwplotka	9385f31147	scrape: Fix metadata in WAL not working for histograms and summaries. ... The was a bug (due to confusion?) on the local metadata cache that is cached by metric family not the series metric name. The fix is to NOT use that local cache at all (it's still needed for current metadata API implementation, added TODO on how we can get rid of it). I went ahead and also rename Metric field in metadata structs to MetricFamily to make clear it's not always __name__. Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-01-15 20:12:38 +00:00
Bartlomiej Plotka	e5e8f90119	Merge pull request #15720 from prometheus/scrapebench ... scrape: Improved scrape loop benchmark	2025-01-14 15:15:37 +01:00
Arve Knudsen	f030894c2c	Fix issues raised by staticcheck (#15722) ... Fix issues raised by staticcheck We are not enabling staticcheck explicitly, though, because it has too many false positives. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-01-09 17:51:26 +01:00
bwplotka	281306765e	scrape: Unified scrape loop benchmark. ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2024-12-29 15:19:06 +00:00
Bryan Boreham	bc9210e393	[TESTS] Scrape: make caching work in benchmark ... Returning 0 from Append means 'unknown', so the series is never cached. Return arbitrary numbers instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-29 15:13:47 +00:00
Bryan Boreham	b4ef38cfc8	Scraping: Add benchmark for protobuf format ... Extract helper function textToProto(). Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-29 15:13:38 +00:00
Bryan Boreham	8f4557b0b1	Scraping benchmark: more realistic test ... Don't repeat type and help text. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-29 15:11:39 +00:00
Bryan Boreham	7b03796d0f	Scraping: stop storing discovered labels (#15261) ... Instead of storing discovered labels on every target, recompute them if required. The `Target` struct now needs to hold some more data required to recompute them, such as ScrapeConfig. This moves the load from every Prometheus all of the time, to just when someone views Service Discovery in the UI. The way `PopulateLabels` is used changes; you are no longer expected to call it with a part-populated `labels.Builder`. The signature of `Target.Labels` changes to take a `labels.Builder` instead of a `ScratchBuilder`, for consistency with `DiscoveredLabels`. This will save a lot of work when many targets are filtered out in relabeling. Combine with `keep_dropped_targets` to avoid ever computing most labels for dropped targets. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-21 13:33:08 +00:00
Joel Beckmeyer	39f5a07236	fix TestOOOHeadChunkReader_Chunk on 32-bit ... Signed-off-by: Joel Beckmeyer <joel@beckmeyer.us>	2024-12-16 10:45:07 -05:00
David Ashpole	953a873342	update links to openmetrics to reference the v1.0.0 release ... Signed-off-by: David Ashpole <dashpole@google.com>	2024-12-13 21:32:27 +00:00
bwplotka	f44bba31b3	config: Remove validation GetScrapeConfigs; require using config.Load. ... We should never modify (or even shallow copy) Config after config.Load; added comments and modified GetScrapeConfigs to do so. For GetScrapeConfigs the validation (even repeated) was likely doing writes (because global fields was 0). We GetScrapeConfigs concurrently in tests and ApplyConfig causing test races. In prod there were races but likelyt only to replace 0 with 0, so not too severe. I removed validation since I don't see anyone using our config.Config without Load. I had to refactor one test that was doing it, all others use yaml config. Fixes #15538 Previous attempt: https://github.com/prometheus/prometheus/pull/15634 Signed-off-by: bwplotka <bwplotka@gmail.com>	2024-12-11 13:31:01 +00:00
Bryan Boreham	c69aeabf1d	Merge pull request #15563 from GiedriusS/fix_setting_field ... scrape: fix nil panic after scrape loop reload	2024-12-10 15:33:56 +00:00
Giedrius Statkevičius	5d76510bd6	scrape: update test name ... Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-09 17:36:36 +02:00
Giedrius Statkevičius	5479fb3cfd	scrape: update test after review ... Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-09 08:18:06 +02:00
Björn Rabenstein	b7947b9155	Merge pull request #15529 from machine424/sunsgne ... test: add TestTargetsFromGroupWithLabelKeepDrop to reinforce a relabelling behaviour	2024-12-05 01:26:01 +01:00
machine424	7719ed18d7	test: add TestTargetsFromGroupWithLabelKeepDrop to reinforce a relabelling behaviour. ... fixes https://github.com/prometheus/prometheus/issues/12355 Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-12-04 17:17:10 +01:00
Björn Rabenstein	d8efb7169d	Merge pull request #15489 from yeya24/fix-always-scrape-classic-hist-reload ... hot reload always_scrape_classic_histograms and convert_classic_histo…	2024-12-04 14:03:21 +01:00
Bryan Boreham	48287b15d4	Merge pull request #13914 from bboreham/scrape-reload-metric ... [REFACTOR] Scraping: rename variables for clarity	2024-12-02 17:09:25 +00:00
Giedrius Statkevičius	90e832c861	scrape: fix nil panic after scrape loop reload ... Don't forget to set `metrics` field as otherwise scraping will lead to a nil panic in case the body size limit is reached. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-02 16:08:37 +02:00
Ben Ye	e10bbf0a84	hot reload always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs properly ... Signed-off-by: Ben Ye <benye@amazon.com>	2024-11-30 22:26:09 -08:00
TJ Hoplock	aa8e067f13	Merge release-3.0 into main	2024-11-27 17:51:51 -05:00
TJ Hoplock	3e24e84172	fix!: stop unbounded memory usage from query log ... Resolves: #15433 When I converted prometheus to use slog in #14906, I update both the `QueryLogger` interface, as well as how the log calls to the `QueryLogger` were built up in `promql.Engine.exec()`. The backing logger for the `QueryLogger` in the engine is a `util/logging.JSONFileLogger`, and it's implementation of the `With()` method updates the logger the logger in place with the new keyvals added onto the underlying slog.Logger, which means they get inherited onto everything after. All subsequent calls to `With()`, even in later queries, would continue to then append on more and more keyvals for the various params and fields built up in the logger. In turn, this causes unbounded growth of the logger, leading to increased memory usage, and in at least one report was the likely cause of an OOM kill. More information can be found in the issue and the linked slack thread. This commit does a few things: - It was referenced in feedback in #14906 that it would've been better to not change the `QueryLogger` interface if possible, this PR proposes changes that bring it closer to alignment with the pre-3.0 `QueryLogger` interface contract - reverts `promql.Engine.exec()`'s usage of the query logger to the pattern of building up an array of args to pass at once to the end log call. Avoiding the repetitious calls to `.With()` are what resolve the issue with the logger growth/memory usage. - updates the scrape failure logger to use the update `QueryLogger` methods in the contract. - updates tests accordingly - cleans up unused methods Builds and passes tests successfully. Tested locally and confirmed I could no longer reproduce the issue/it resolved the issue. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-11-23 14:20:37 -05:00
Arve Knudsen	89bbb885e5	Upgrade to golangci-lint v1.62.0 (#15424) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-11-20 17:22:20 +01:00
Bryan Boreham	26886d6d95	Merge branch 'release-3.0' into merge-release-into-main	2024-11-18 14:43:56 +00:00
Bryan Boreham	6380229c71	[REFACTOR] Scraping: rename variables for clarity ... Instead of shadowing the outer variables, use different names. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-07 17:29:11 +00:00
Bryan Boreham	3f6a133f26	[Test] Scraping: check reload metrics ... Need to extend `newTestScrapeMetrics`` to get at the Registry. `gatherLabels` function could go upstream to `client_golang`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-07 17:29:04 +00:00
alexgreenbank	6b09f094e1	scrape: stop erroring on empty scrapes ... Signed-off-by: alexgreenbank <alex.greenbank@grafana.com>	2024-11-07 11:30:03 +00:00
Arthur Silva Sens	96c2fe04e7	Merge pull request #15328 from prometheus/deadcode ... chore: Remove dead code	2024-11-06 13:20:53 -03:00
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-11-06 16:50:36 +01:00
Simon Pasquier	37f3f3f2db	Fix scrape failure logs ... Before this change, logs would show like: ``` {...,"target":"http://localhost:8080/metrics","!BADKEY":"Get ..."} ``` After this change ``` {...,"msg":"Get ...","job_name":...,"target":...} ``` Signed-off-by: Simon Pasquier <spasquie@redhat.com>	2024-11-06 15:38:47 +01:00
Arthur Silva Sens	88818c9cb3	chore: Remove dead code ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-11-04 13:00:22 -03:00
Vanshika	cccbe72514	TSDB: Fix some edge cases when OOO is enabled (#14710) ... Fix some edge cases when OOO is enabled Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	2024-10-23 17:34:28 +02:00
George Krajcsovits	aa81210c8b	NHCB scrape: refactor state handling and speed up scrape test (#15193) ... * NHCB: scrape use state field and not booleans From comment https://github.com/prometheus/prometheus/pull/14978#discussion_r1800898724 Also make compareLabels read only and move storeLabels to the first processed classic histogram series. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Speed up TestConvertClassicHistogramsToNHCB 3x Reduce the startup time and timeouts Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * lint fix Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-22 17:49:25 +01:00
George Krajcsovits	877fd2a60e	Update scrape/scrape.go ... Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2024-10-21 16:01:34 +02:00
György Krajcsovits	4283ae73dc	Rename convert_classic_histograms to convert_classic_histograms_to_nhcb ... On reviewer request. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-21 13:22:58 +02:00
György Krajcsovits	a23aed5634	More followup to #15164 ... Scrape test for NHCB modified. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-21 11:10:50 +02:00
György Krajcsovits	70742a64aa	Follow up #15178 ... Renaming Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-21 11:03:47 +02:00
György Krajcsovits	8c1b5a6251	Merge branch 'main' into nhcb-scrape-impl	2024-10-21 11:00:41 +02:00
Alex Greenbank	421a3c22ea	scrape: provide a fallback format (#15136) ... scrape: Remove implicit fallback to the Prometheus text format Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. --------- Signed-off-by: alexgreenbank <alex.greenbank@grafana.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com>	2024-10-18 17:12:31 +02:00
Bartlomiej Plotka	efc43d0714	s/scrape_classic_histograms/always_scrape_classic_histograms (3.0 breaking change) (#15178) ... This is for readability, especially when we can converting to nhcb option. See discussion https://cloud-native.slack.com/archives/C077Z4V13AM/p1729155873397889 Signed-off-by: bwplotka <bwplotka@gmail.com>	2024-10-18 08:32:15 +01:00
György Krajcsovits	5ee698de2c	Apply review comments ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-17 12:55:45 +02:00
György Krajcsovits	04b827dd77	Merge branch 'main' into nhcb-scrape-impl	2024-10-17 12:34:44 +02:00
Yi	2cabd1b707	config: remove expand-external-labels flag in release 3.0 (#14657) ... remove expand-external-labels feature flag and enabled env arg expansion for external labels by default. Signed-off-by: jyz0309 <45495947@qq.com>	2024-10-17 10:25:05 +02:00
György Krajcsovits	c13585665d	Merge branch 'main' into nhcb-scrape-impl ... # Conflicts: # promql/promqltest/test.go # util/convertnhcb/convertnhcb.go	2024-10-14 14:26:11 +02:00
Björn Rabenstein	8b545bab2f	Merge pull request #15026 from huochexizhan/main ... fix: fix slice init length	2024-10-10 12:55:28 +02:00
huochexizhan	ff16fa1827	fix: fix slice init length ... Signed-off-by: huochexizhan <huochexizhan@outlook.com>	2024-10-10 16:14:40 +08:00
György Krajcsovits	8dfa733596	Fix labels handling with dedupelabels tag ... Use scratch builder. Use hash compare instead of compare by label. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-09 12:29:59 +02:00