1487 commits

Author	SHA1	Message	Date
Björn Rabenstein	5e987e7c7d	Merge pull request #15092 from jan--f/clarify-storage-contract ... storage: require selectors to always return matching results	2024-10-09 13:07:59 +02:00
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log ... For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-10-07 15:58:50 -04:00
Matthieu MOREL	ab64966e9d	fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094) ... * fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-10-06 16:35:29 +00:00
Jan Fajerski	563bfdd384	storage: require selectors to always return matching results ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-10-04 12:10:16 +02:00
György Krajcsovits	44ebbb8458	Fix missing histogram copy in sampleRing ... The specialized version of sample add to the ring: func addH(s hSample, buf []hSample, r *sampleRing) []hSample func addFH(s fhSample, buf []fhSample, r *sampleRing) []fhSample already correctly copy histogram samples from the reused hReader, fhReader buffers, but the generic version does not. This means that the data is overwritten on the next read if the sample ring has seen histogram and float samples at the same time and switched to generic mode. The `genericAdd` function (which was commented anyway) is by now quite different from the specialized functions so that this commit deletes it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-02 13:57:28 +02:00
György Krajcsovits	b5479831b8	Unit test for regression in rate vs float and histogram mixup ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-02 13:56:33 +02:00
Ayoub Mrini	105ab2e95a	fix(test): adjust defer invocations (#14996) ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-27 17:13:51 +01:00
Arthur Silva Sens	6bd9b1a7cc	Histogram CT Zero ingestion ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-09-26 11:29:22 -03:00
Callum Styan	005bd33fe2	support v2 proto for BenchmarkSampleSend (#14935) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	2024-09-19 15:38:26 -07:00
Callum Styan	c328d5fc88	fix rwv2 build write request benchmark, also change how the memory usage (#14925) ... is reported for these benchmarks to more accurately reflect what's actually allocated Signed-off-by: Callum Styan <callumstyan@gmail.com>	2024-09-18 07:04:10 +01:00
Antoine Pultier	d90d0976b5	fix(bstream/writeByte): ensure it appends only one byte (#14854) ... fix(bstream/writeByte): ensure it appends only one byte Signed-off-by: Antoine Pultier <antoine.pultier@sintef.no>	2024-09-17 16:28:33 +02:00
Carrie Edwards	14e3c05ce8	tsdb: Add support for ingestion of out-of-order native histogram samples (#14546) ... Add support for ingesting OOO native histograms * Add flag for enabling and disabling OOO native histogram ingestion * Update OOO querying tests to include native histogram samples * Add OOO head tests * Add test for OOO native histogram counter reset headers Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored by: Carrie Edwards <edwrdscarrie@gmail.com> Co-authored by: Jeanette Tan <jeanette.tan@grafana.com> Co-authored by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored by: Fiona Liao <fiona.liao@grafana.com>	2024-09-17 11:19:06 +02:00
Harry John	919dc0cbc6	storage: Update LabelQuerier interface to return sorted label values (#14849) ... * Change LabelQuerier.LabelValues() to return sorted values --------- Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>	2024-09-17 08:55:02 +02:00
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868) ... * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2024-09-10 22:32:03 +02:00
Arve Knudsen	4fc562f9e7	OTLP: Support context cancellation/timeout during translation (#14612) ... * OTLP: Support context cancellation/timeout during translation --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-09-08 17:13:40 +02:00
György Krajcsovits	a693dd19f2	Fix: chunkenc.MockSeriesIterator ... Starts its index from 0 , but users call Next() before first sample so it needs to start from -1 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-30 16:44:36 +02:00
Arve Knudsen	bc6c2c5d35	OTLP Receiver: Add tests (#14764) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-30 11:30:57 +02:00
Bryan Boreham	f90c7a11d1	[REFACTOR] OTLP translator: simplify time conversion ... We don't need multiple levels of abstraction to convert nanoseconds to milliseconds. We do benefit from tests, however. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-29 12:00:01 +01:00
Owen Williams	9da75328ea	fix(utf8): ensure correct validation when legacy mode turned on (#14736) ... fix(utf8): ensure correct validation when legacy mode turned on This depends on the included update of the prometheus/common dependency. --------- Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-08-28 17:15:42 +02:00
Justin Lei	3a82cd5a7e	Add streaming remote read to ReadClient (#11379) ... * Add streaming remote read to ReadClient Signed-off-by: Justin Lei <justin.lei@grafana.com> * Apply suggestions from code review Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Justin Lei <justin.lei@grafana.com> * Remote read instrumentation tweaks Signed-off-by: Justin Lei <lei.justin@gmail.com> * Minor cleanups Signed-off-by: Justin Lei <lei.justin@gmail.com> * In-line handleChunkedResponse Signed-off-by: Justin Lei <lei.justin@gmail.com> * Fix lints Signed-off-by: Justin Lei <lei.justin@gmail.com> * Explicitly call cancel() when needed Signed-off-by: Justin Lei <lei.justin@gmail.com> * Update chunkedSeries, chunkedSeriesIterator for new interfaces Signed-off-by: Justin Lei <lei.justin@gmail.com> * Adapt remote.chunkedSeries to use prompb.ChunkedSeries Signed-off-by: Justin Lei <lei.justin@gmail.com> * Fix lint Signed-off-by: Justin Lei <lei.justin@gmail.com> --------- Signed-off-by: Justin Lei <justin.lei@grafana.com> Signed-off-by: Justin Lei <lei.justin@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2024-08-28 07:23:54 +01:00
Devin Trejo	d4994e5bc4	fix: Remote-write-reciever returns 4xx when request contains a time series with duplicate labels. (#14716) ... Signed-off-by: Devin Trejo <dtrejo@palantir.com>	2024-08-23 15:15:27 -07:00
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments ... Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2024-08-22 17:36:11 +02:00
Bryan Boreham	95b3c49836	[TESTS] remote-write: fix slice init length (#14702) ... fix: fix slice init length	2024-08-22 12:53:45 +01:00
Arve Knudsen	b50c5d42fe	OTLP receiver: Warn when encountering invalid exponential histograms ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-21 16:43:16 +02:00
cuishuang	2110661121	fix: fix slice init length ... Signed-off-by: cuishuang <imcusg@gmail.com>	2024-08-21 18:15:25 +08:00
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-18 12:13:25 +02:00
Björn Rabenstein	1daf7cdd62	Merge pull request #14626 from cuiweiyuan/main ... chore: fix some function names	2024-08-15 11:46:21 +02:00
cuiweiyuan	1800af54f0	chore: fix some function names ... Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2024-08-15 13:57:21 +08:00
Arve Knudsen	5fd66ba855	Merge pull request #14510 from aknuds1/arve/add-normalize-name-tests ... otlptranslator: Add tests for BuildCompliantName	2024-08-09 18:21:16 +02:00
Oleg Zaytsev	0833d2a230	Fix appendable: check whether last val was a histogram (#14613) ... * Fix appendable: check whether last val was a histogram When appending a float, we were checking whether lastValue was equal to current value, but we didn't check whether last value was a float value. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-08-07 15:02:59 +02:00
George Krajcsovits	00ab05c3b9	Native histograms: fix spurios counter reset when merging recoded chunk to normal chunk (#14513) ... * chunkenc: allow missing empty buckets on histogram append Allow appending to chunks when the histogram to be added is missing some buckets, but the missing buckets are empty in the chunk. For example bucket at index 5 is present in the chunk, but its value is 0 and the new histogram doesn't have a bucket at index 5. This fixes an issue of merging chunks where one chunk was recoded to retroactively have some empty buckets in all the histograms and we are merging in a histogram that doesn't have the empty bucket (because it was not recoded yet). The operation alters the histogram that is being added, however this has already been the case when appending gauge histograms. Thus the test TestHistogramSeriesToChunks in storage package is changed to explicitly test what happened to the appended histogram - Compact(0) call is removed. The new expandIntSpansAndBuckets and expandFloatSpansAndBuckets functions are a merge of expandSpansForward and counterResetInAnyBucket and counterResetInAnyFloatBucket. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-01 09:22:32 +02:00
Callum Styan	1561815732	remote write: increase time threshold for resharding (#14450) ... Don't reshard if we haven't successfully sent a sample in the last shardUpdateDuration seconds. Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: kushagra Shukla <kushalshukla110@gmail.com>	2024-07-30 14:08:28 -07:00
Max Amin	84b819a69f	feat: add Google cloud roundtripper for remote write (#14346) ... * feat: Google Auth for remote write Signed-off-by: Max Amin <maxamin@google.com> --------- Signed-off-by: Max Amin <maxamin@google.com>	2024-07-30 16:25:19 +01:00
Arve Knudsen	4fb2183437	Test a couple more cases without suffix gen ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-26 11:21:58 +02:00
Bryan Boreham	6e89250a5d	Revert "Chunked remote read: close the querier earlier" ... Believed to trigger segmentation faults due to memory-mapped block data still being accessed by iterators after the querier is closed. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-26 09:56:38 +01:00
Arve Knudsen	c04924bc41	otlptranslator: Add tests for BuildCompliantName ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-24 18:17:56 +02:00
Bryan Boreham	c037a3df84	lint ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 16:34:42 +01:00
Bryan Boreham	12bd92a25c	[ENHANCEMENT] Storage: Short-circuit merge of single querier with no-op queriers ... Filter before checking whether there is only one. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 15:35:09 +01:00
Bryan Boreham	90d793e8c5	[BUGFIX] Storage: Single secondary querier errors should be warnings. ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 15:33:07 +01:00
Bryan Boreham	0ae881739b	[TEST] Storage: check MergeQuerier with nil primary ... This test fails on current code. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 15:02:52 +01:00
Bryan Boreham	677cdcdcec	[TEST] Storage: ignore difference between nil and empty ... We need this for subsequent changes. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 15:01:00 +01:00
Bryan Boreham	23307b02c5	[TESTS] Storage: Improve MergeQuerier tests ... `TestMergeQuerierWithSecondaries_ErrorHandling` now tests `NewMergeQuerier` rather than creating the data structure directly. This means we now test short-circuiting when only a single querier is required. Merge `mockGenericQuerier` into `mockQuerier`. Replace `unwrapMockGenericQuerier` with a visitor pattern. No change in functionality intended. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-22 14:37:20 +01:00
Goutham Veeramachaneni	e2ef0dc381	Merge branch 'main' into feat/promote-attributes ... Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	2024-07-21 13:12:17 +02:00
Bartlomiej Plotka	a60e5ce362	[PRW 2.0] Added Sender and RW Handler support for Response Stats. (#14444) ... * [PRW 2.0] Added Sender support for Response Stats. Chained on top of https://github.com/prometheus/prometheus/pull/14427 Fixes https://github.com/prometheus/prometheus/issues/14359 Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * move write stats to it's own file Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up header usage Signed-off-by: Callum Styan <callumstyan@gmail.com> * add missing license to new stats file Signed-off-by: Callum Styan <callumstyan@gmail.com> * Addressed all comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2024-07-19 18:53:40 +01:00
Arve Knudsen	1d21867d8b	Add otlptranslator tests ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-16 15:31:52 +02:00
Arve Knudsen	a25b626792	prometheusremotewrite: Support resource attribute promotion ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-16 11:51:10 +02:00
Charles Korn	2ca4d16236	storage: add AtT method to MemoizedSeriesIterator ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-07-15 14:09:02 +10:00
Bryan Boreham	d116bf7b9f	Merge pull request #14109 from harry671003/pass_limit_to_querier ... storage: pass limit param as hint in querier	2024-07-12 10:27:52 +01:00
Bartlomiej Plotka	0c87643abd	[PRW 2.0] write_handler: Support for rc.2 spec & improved error handling for v2. (#14427) ... * [PRW 2.0] write_handler: Support for rc.2 spec & improved error handling for v2. Fixes: https://github.com/prometheus/prometheus/issues/14359 Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed Callum comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * Added missing lock on flush. Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed lint. Signed-off-by: bwplotka <bwplotka@gmail.com> * Added tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed Callum's comments & updated re spec. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update storage/remote/write_handler_test.go Co-authored-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2024-07-12 08:11:38 +01:00
machine424	caa71fb3c7	chore(storage/remote): collect maxTimestamp when value is 0 as well. ... This change enables the PrometheusRemoteWriteBehind alert’s expression to be evaluated even when the remote endpoint has never been reached. As a result, PrometheusRemoteWriteBehind will fire to easily detect configuration mistakes (such as incorrect endpoint URLs) or unrecoverable connectivity issues. See https://github.com/prometheus/prometheus/issues/14350 for details. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-07-11 16:43:58 +02:00