425 commits

Author	SHA1	Message	Date
Bryan Boreham	485523eed2	Merge pull request #14816 from bboreham/improve-promql-tracing ... Improve promql tracing	2024-09-04 14:32:22 +01:00
Bryan Boreham	abb0502685	[ENHANCEMENT] PromQL: Add detail to tracing spans ... For aggregates, operators, calls, show what operation is performed. Also add an event when series are expanded, typically time spent accessing TSDB. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-03 14:15:58 +01:00
Bryan Boreham	8742077498	[BUGFIX] PromQL: pass Context so spans parent correctly ... Assigning to `evaluator.ctx` in `eval()` broke the parent-child relationship. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-03 14:15:29 +01:00
Arve Knudsen	5dfbcc390e	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-09-02 16:26:59 +02:00
Jorge Creixell	e9e3d64b7c	PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477) ... PromQL engine: Delay deletion of __name__ label to the end of the query evaluation - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error. - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation. - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If `__name__` is used as target label, it won't be dropped at the end of the query evaluation. - Fixes https://github.com/prometheus/prometheus/issues/11397 - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`). - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags. - Note: a feature flag `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792) Example (this always fails, as `__name__` is being dropped by `count_over_time`): ``` count_over_time({__name__!=""}[1m]) => Error executing query: vector cannot contain metrics with the same labelset ``` Before: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => Error executing query: vector cannot contain metrics with the same labelset ``` After: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 ... ``` Signed-off-by: Jorge Creixell <jcreixell@gmail.com> --------- Signed-off-by: Jorge Creixell <jcreixell@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in>	2024-08-29 15:50:39 +02:00
Arve Knudsen	c9a460d570	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-26 12:17:10 +02:00
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments ... Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2024-08-22 17:36:11 +02:00
Björn Rabenstein	1daf7cdd62	Merge pull request #14626 from cuiweiyuan/main ... chore: fix some function names	2024-08-15 11:46:21 +02:00
cuiweiyuan	1800af54f0	chore: fix some function names ... Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2024-08-15 13:57:21 +08:00
Charles Korn	52818a97e2	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms ... # Conflicts: # promql/promqltest/testdata/native_histograms.test	2024-08-14 07:52:08 +10:00
Björn Rabenstein	c2bc6cfe97	Merge pull request #14621 from charleskorn/panic-message ... promql: clarify error message logged when panic occurs during query evaluation	2024-08-13 23:02:43 +02:00
Arve Knudsen	0503d4f372	PromQL: Fix comment regarding non-nil histogram pointer ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-13 08:55:24 +02:00
Charles Korn	f992f81bd0	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms ... Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>	2024-08-09 13:58:54 +10:00
Charles Korn	82bb35fabb	Address PR feedback: fix typo and rename variable ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-09 13:51:31 +10:00
Charles Korn	f91009aa2e	promql: clarify error message when panic occurs during query evaluation ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-08 09:11:38 +10:00
Björn Rabenstein	27579c9148	Merge pull request #14605 from krajorama/fix-staleness-pool-corrupt ... Fix histogram pool poisoning bug chunkenc.Iterator	2024-08-07 21:02:08 +02:00
George Krajcsovits	17b0b788da	Update promql/engine.go ... Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2024-08-07 20:15:46 +02:00
Charles Korn	0f4bc87b4f	Make linter happy ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:35:06 +10:00
Charles Korn	f07b3ae67b	Fix issue where avg over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results or panics ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:32:35 +10:00
Charles Korn	5ee94f49a2	Fix issue where sum over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:30:01 +10:00
Björn Rabenstein	ee5bba07c0	Merge pull request #14413 from prometheus/beorn7/promql ... promql: more Kahan summation (avg) and less incremental mean calculation (avg, avg_over_time)	2024-08-06 19:56:32 +02:00
György Krajcsovits	37c8c9257b	Fix histogram pool poisoning bu chunkenc.Iterator ... chunkenc.Iterator.AtFloatHistogram may do a shallow copy if it receives nil as input pointer. This can in turn share the span slice with multiple histograms in the matrixSelectorHPool, leading to unexpected errors. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 19:40:14 +02:00
Arve Knudsen	fec6adadcd	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-07-14 13:19:11 +02:00
beorn7	c46074f4dd	promql: make avg aggregation more precise and less expensive ... The basic idea here is that the previous code was always doing incremental calculation of the mean value, which is more costly and can be less precise. It protects against overflows, but in most cases, an overflow doesn't happen anyway. The other idea applied here is to expand on #14074, where Kahan summation was applied to sum(). With this commit, the average is calculated in a conventional way (adding everything up and divide in the end) as long as the sum isn't overflowing float64. This is combined with Kahan summation so that the avg aggregation, in most cases, is really equivalent to the sum aggregation with a following division (which is the user's expectation as avg is supposed to be syntactic sugar for sum with a following divison). If the sum hits ±Inf, the calculation reverts to incremental calculation of the mean value. Kahan summation is also applied here, although it cannot fully compensate for the numerical errors introduced by the incremental mean calculation. (The tests added in this commit would fail if incremental mean calculation was always used.) Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-10 19:20:24 +02:00
Filip Petkovski	acb6c1ae4b	Fix decoding buckets for native histograms in binops ... The optimizer which detects cases where histogram buckets can be skipped does not take into account binary expressions. This can lead to buckets not being decoded if a metric is used with both histogram_fraction/quantile and histogram_sum/count in the same expression. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-10 11:55:29 +02:00
beorn7	44d8c1d182	nit: add period at end of sentence ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-04 17:31:39 +02:00
beorn7	9a837b7f3c	promql: Make groupedAggregation.groupCount a float64 ... It's always used as such. Let's avoid the countless conversions. Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-04 17:31:34 +02:00
JuanJo Ciarlante	c94c5b64c3	feat: add limitk() and limit_ratio() operators (#12503) ... * rebase 2024-07-01, picks previous renaming to `limitk()` and `limit_ratio()` Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * gofumpt -d -extra Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * more lint fixes Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * more lint fixes+ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * put limitk() and limit_ratio() behind --enable-feature=promql-experimental-functions Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * EnableExperimentalFunctions for TestConcurrentRangeQueries() also Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * use testutil.RequireEqual to fix tests, WIP equivalent thingie for require.Contains Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * lint fix Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * moar linting Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * rebase 2024-06-19 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * re-add limit(2, metric) testing for N=2 common series subset Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move `ratio = param` to default switch case, for better readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * gofumpt -d -extra util/testutil/cmp.go Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * early break when reaching k elems in limitk(), should have always been so (!) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * small typo fix Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change small break-loop rearrange for readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove IsNan(ratio) condition in switch-case, already handled as input validation Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change adding some comments Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change simplify fullMatrix() helper functions used for tests Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add `limitk(-1, metric)` testcase, which is handled as any k < 1 case Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * engine_test.go: no-change create `requireCommonSeries() helper func (moving code into it) for readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * rebase 2024-06-21 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * engine_test.go: HAPPY NOW about its code -> reorg, create and use simpleRangeQuery() function, less lines and more readable ftW \o/ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move limitk(), limit_ratio() testing to promql/promqltest/testdata/limit.test Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove stale leftover after moving tests from engine_test.go to testdata/ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix flaky `limit_ratio(0.5, ...)` test case Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix AddRatioSample() implementation to use a single conditional (instead of switch/case + fallback return) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * docs/querying/operators.md: document r < 0 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add negative limit_ratio() example to docs/querying/examples.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move more extensive docu examples to docs/querying/operators.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * typo Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * small docu fix for poor-mans-normality-check, add it to limit.test ;) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * limit.test: expand "Poor man's normality check" to whole eval range Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * restore mistakenly removed existing small comment Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * expand poors-man-normality-check case(s) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Revert "expand poors-man-normality-check case(s)" This reverts commit f69e1603b2ebe69c0a100197cfbcf6f81644b564, indeed too flaky 0:) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove humor from docs/querying/operators.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix signoff Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add web/ui missing changes Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * expand limit_ratio test cases, cross-fingering they'll not be flaky Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove flaky test Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add missing warnings.Merge(ws) in instant-query return shortcut Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add missing LimitK||LimitRatio case to codemirror-promql/src/parser/parser.ts Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix ui-lint Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * actually fix returned warnings :] Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> --------- Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> Co-authored-by: Julius Volz <julius.volz@gmail.com>	2024-07-03 22:18:57 +02:00
Arve Knudsen	e8ae8cf012	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-01 10:47:21 +02:00
Björn Rabenstein	2e58d46522	Merge pull request #13662 from prometheus/nhcb ... Native histograms custom buckets storage	2024-06-27 21:44:20 +02:00
Bryan Boreham	b6aba4ff14	Merge pull request #14074 from bboreham/kahan-sum-sum ... [ENHANCEMENT] PromQL: use Kahan summation for sum()	2024-06-24 11:13:26 +01:00
Arve Knudsen	b7320ef636	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-06-14 10:51:35 +02:00
Jeanette Tan	14f8dded39	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 19:17:14 +08:00
Filip Petkovski	6e68046c25	Implement histogram statistics decoder (#14097) ... Implement histogram statistics decoder This commit speeds up histogram_count and histogram_sum functions on native histograms. The idea is to have separate decoders which can be used by the engine to only read count/sum values from histogram objects. This should help with reducing allocations when decoding histograms, as well as with speeding up aggregations like sum since they will be done on floats and not on histogram objects. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>	2024-06-06 17:17:13 +02:00
Arve Knudsen	e57aac8084	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-06-05 11:37:44 +02:00
Arve Knudsen	0cc99e677a	promql.Engine: Add Close method ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-28 12:01:47 +02:00
Jeanette Tan	f028496133	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-05-14 16:20:15 +08:00
Charles Korn	0e934dba8e	Capture timing information while sorting ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-05-13 19:47:18 +10:00
Charles Korn	036c87223c	Ensure series in matrix values returned for instant queries are always sorted ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-05-13 11:03:15 +10:00
Bryan Boreham	ea82b49c33	[ENHANCEMENT] PromQL: use Kahan summation for sum() ... This can give a more precise result, by keeping a separate running compensation value to accumulate small errors. See https://en.wikipedia.org/wiki/Kahan_summation_algorithm Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-09 14:29:38 +01:00
Bryan Boreham	3fd24d1cd7	Merge pull request #13999 from bboreham/extract-promqltest ... [Test] Extract most PromQL test code into separate packages	2024-05-09 13:23:11 +01:00
Bryan Boreham	e7c77f7b40	promql: export NewTestQuery ... So that tests can call it from another package. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:08:04 +01:00
Jeanette Tan	796b1bbfde	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-05-08 19:11:39 +08:00
Arve Knudsen	a25160e6a4	[REFACTOR] PromQL: simplify rangeEvalTimestampFunctionOverVectorSelector (#14021) ... The function `rangeEvalTimestampFunctionOverVectorSelector` appeared to be checking histogram size, however the value it used was always 0 due to subtle variable shadowing. However we don't need to pass sample values to the `timestamp` function, since the latter only cares about timestamps. This also affects peak sample count in statistics, since we are no longer copying histogram samples. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-08 10:39:44 +01:00
György Krajcsovits	bcafa5f1f9	Merge remote-tracking branch 'upstream/main' into update-nhcb	2024-04-24 11:06:59 +02:00
Bryan Boreham	12961c6a37	promql: refactor: eliminate one 'else' ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	0ac927515b	promql: move group-seen into group struct ... Save allocating an auxilliary array. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	7499d90913	promql: remove pointer to aggregation groups ... Just allocate in one slice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	cfbeb6681b	promql: re-use one heap for topk and bottomk ... Slightly ugly casting saves memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	5e3914a27c	promql: remove histogramMean from groupedAggregation ... Re-use histogramValue since we don't need them separately. Tidy up initialization. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00