1216 commits

Author	SHA1	Message	Date
cuiweiyuan	1800af54f0	chore: fix some function names ... Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2024-08-15 13:57:21 +08:00
Charles Korn	52818a97e2	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms ... # Conflicts: # promql/promqltest/testdata/native_histograms.test	2024-08-14 07:52:08 +10:00
Björn Rabenstein	c2bc6cfe97	Merge pull request #14621 from charleskorn/panic-message ... promql: clarify error message logged when panic occurs during query evaluation	2024-08-13 23:02:43 +02:00
George Krajcsovits	fe5354df88	Merge pull request #14658 from krajorama/zero-rate-nh-quantile ... Native histograms: define behavior when rate is null.	2024-08-13 16:05:43 +02:00
György Krajcsovits	386fc8b9f6	Update from review comments. ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-13 15:26:07 +02:00
Arve Knudsen	0503d4f372	PromQL: Fix comment regarding non-nil histogram pointer ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-13 08:55:24 +02:00
György Krajcsovits	6aee5b4b38	fix typo ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-12 12:04:45 +02:00
György Krajcsovits	06a8886b94	Native histograms: define behavior when rate is null. ... Histogram quantile returns NaN in this case, which might be surprising, so add a unit test that clarifies that this is intentional. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-12 10:40:21 +02:00
suntala	fd2f44af7f	Fall back to comparing by label set when sorting by label desc ... Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com> Signed-off-by: suntala <arati.rana@grafana.com>	2024-08-11 21:44:03 +02:00
suntala	94ad489328	Fall back to comparing by label set when sorting by label ... Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com> Signed-off-by: suntala <arati.rana@grafana.com>	2024-08-11 21:44:03 +02:00
Charles Korn	f992f81bd0	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms ... Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>	2024-08-09 13:58:54 +10:00
Charles Korn	5cfdde327c	Address PR feedback: add extra test case ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-09 13:57:37 +10:00
Charles Korn	82bb35fabb	Address PR feedback: fix typo and rename variable ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-09 13:51:31 +10:00
Björn Rabenstein	41f8a0223c	Merge pull request #14609 from charleskorn/rate-over-mixed-custom-exponential-histograms ... promql: fix "cannot reduce resolution to custom buckets schema" panic in `rate` over native histograms with mix of custom and exponential buckets	2024-08-08 14:03:41 +02:00
Manik Rana	02c465bf58	textparse: Implement CreatedTimestamp() in openmetricsparse.go (#14356) ... * feat: initial implement of createedTimestamp() with tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: return ct after finding it Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove unneeded test Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: multiple changes - implement changes from pair programming session - use newParse.val() - advance parser p if ct is found Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: check if err from p.Next() Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: advance parser and parse histograms + summary Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: restore previous tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: retore failing tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove unneeded comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: return nil when mtype doesn't match Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: update go fmt version Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: cleanup Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: document deepcopyparser Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: cover edgecase of `gauge_created` in CreatedTimestamp() Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: readability updates Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: dedeuplicate labeldiff checks Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add tests for new label functions Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: document CreatedTimestamp func Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: optimize `CreatedTimestamp()` - Use refactored CreatedTimestamp function with bug fixes - Remove unused code in labels.go - Improve code documentation Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add tests and lint fixes Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove mName Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add tests for CT parse failures and deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: edit expectCT struct Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add new label in deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: use p.builder in deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: add NewMetricsParserWithOpts Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * chore: comments Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * chore: rename var Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: add condition for OM fuzzing Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: build tags Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: default skipCT to false Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: rename skipCT to skipCTSeries Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: formatting Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments and readability updates Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * refac: remove NewOpenMetricsParserWithOpts Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: extract skipCTSeries logic from parseMetricSuffix Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: inline create a NewOpenMetricsParser Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * refac: improve error handling Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: return error instead of nil Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: remove skipCT check from tBraceOpen Signed-off-by: Manik Rana <manikrana54@gmail.com> * Pair programming with Manik, Arthur and Daniel. Signed-off-by: bwplotka <bwplotka@gmail.com> * chore: comments and use helper funcs Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> --------- Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2024-08-08 12:35:35 +01:00
Charles Korn	f91009aa2e	promql: clarify error message when panic occurs during query evaluation ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-08 09:11:38 +10:00
Björn Rabenstein	27579c9148	Merge pull request #14605 from krajorama/fix-staleness-pool-corrupt ... Fix histogram pool poisoning bug chunkenc.Iterator	2024-08-07 21:02:08 +02:00
George Krajcsovits	17b0b788da	Update promql/engine.go ... Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2024-08-07 20:15:46 +02:00
Charles Korn	0f4bc87b4f	Make linter happy ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:35:06 +10:00
Charles Korn	f07b3ae67b	Fix issue where avg over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results or panics ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:32:35 +10:00
Charles Korn	5ee94f49a2	Fix issue where sum over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 15:30:01 +10:00
Charles Korn	424cefcf5e	Fix "cannot reduce resolution to custom buckets schema" panic in rate over native histograms with mix of custom and exponential buckets ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 14:45:02 +10:00
Charles Korn	f6e4b775e2	Check for errors first ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-07 14:25:46 +10:00
György Krajcsovits	b91acc61b0	Add unit test ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 20:50:51 +02:00
Björn Rabenstein	ee5bba07c0	Merge pull request #14413 from prometheus/beorn7/promql ... promql: more Kahan summation (avg) and less incremental mean calculation (avg, avg_over_time)	2024-08-06 19:56:32 +02:00
Björn Rabenstein	71ba554293	Merge pull request #14482 from ywwg/owilliams/group-quote ... promql: support quoting in grouping label lists	2024-08-06 19:43:38 +02:00
György Krajcsovits	37c8c9257b	Fix histogram pool poisoning bu chunkenc.Iterator ... chunkenc.Iterator.AtFloatHistogram may do a shallow copy if it receives nil as input pointer. This can in turn share the span slice with multiple histograms in the matrixSelectorHPool, leading to unexpected errors. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 19:40:14 +02:00
Charles Korn	aadec25faf	promql: Fix issue where some native histogram-related annotations are not emitted by rate (#14575) ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-08-06 09:10:40 +01:00
Owen Williams	d90c5a71d7	support quoting in grouping label lists ... Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-08-01 10:07:12 -04:00
Jan Fajerski	8a96a75ba8	Merge branch 'main' into 3.0-main-sync-24-08-01 ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-08-01 15:44:47 +02:00
Filip Petkovski	3e6155e596	Use CopyTo when resetting histogram in stats iterator ... The histogram stats iterator does not fully clear the histogram object and is not resilient to new fields being added to the histogram type. To resolve the issue, the commit uses the CopyTo methods which should be future proof to new fields being added. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-31 11:18:47 +02:00
Charles Korn	7fab72a280	promqltest: add support for setting counter reset hint on histogram samples (#14537) ... * promqltest: add support for setting counter reset hint on histogram samples Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-07-31 09:53:05 +02:00
Max Amin	84b819a69f	feat: add Google cloud roundtripper for remote write (#14346) ... * feat: Google Auth for remote write Signed-off-by: Max Amin <maxamin@google.com> --------- Signed-off-by: Max Amin <maxamin@google.com>	2024-07-30 16:25:19 +01:00
Filip Petkovski	2cd97c61e0	Add more test cases ... Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-29 14:53:32 +02:00
Arve Knudsen	7c873004c7	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-07-26 11:48:33 +02:00
Filip Petkovski	be7a4c9b83	Ignore stale histograms for counter reset detection ... The histogram stats decoder keeps track of the last seen histogram sample in order to properly detect counter resets. We are seeing an issue where a histogram with UnknownResetHint gets treated as a counter reset when it follows a stale histogram sample. I believe that this is incorrect since stale samples should be completely ignored in PromQL. As a result, they should not be stored in the histogram stats iterator and the counter reset detection needs to be done against the last non-stale sample. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-26 10:08:31 +02:00
Jan Fajerski	adf5d6bce1	Merge branch 'main' into 3.0-main-sync-24-07-18 ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Conflicts: VERSION pick 3.0.0 promql/promqltest/testdata/histograms.test pick changes from c39776c5b5, but adjust 5m range selectors to 10m to account for https://github.com/prometheus/prometheus/pull/13904. Fixes: promql/promqltest/testdata/functions.test promql/promqltest/testdata/staleness.test Tests added in https://github.com/prometheus/prometheus/pull/9138 need to be adjusted to account for https://github.com/prometheus/prometheus/pull/13904.	2024-07-18 15:56:40 +02:00
Björn Rabenstein	ca7062cf49	Merge pull request #14464 from prometheus/beorn7/histogram ... promql: Add NHCB tests	2024-07-17 14:57:46 +02:00
beorn7	c39776c5b5	promql: Add NHCB tests ... This adds equivalent NHCB tests to the existing classic histogram tests. Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-16 12:20:43 +02:00
Bryan Boreham	bb241d7960	Merge pull request #14463 from fpetkovski/customize-acceptance-tests ... Allow built-in tests to be customized	2024-07-15 10:04:41 +01:00
Arve Knudsen	fbc9eddfaf	Refactor engine creation in tests ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-14 13:58:51 +02:00
Arve Knudsen	fec6adadcd	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-07-14 13:19:11 +02:00
Bryan Boreham	d116bf7b9f	Merge pull request #14109 from harry671003/pass_limit_to_querier ... storage: pass limit param as hint in querier	2024-07-12 10:27:52 +01:00
Filip Petkovski	cb8b6d7504	Allow built-in tests to be customized ... The RunBuiltinTests function accepts a concrete type which makes it hard to exclude certain tests from the suite. It would be great if we could skip tests which might not be critical in order to unblock updates. By accepting an interface instead, we can inject a custom implementation which would skips select test cases. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-11 14:19:09 +02:00
beorn7	cff0429b1a	promql: make avg_over_time faster and more precise ... Same idea as for the avg aggregator before: Most of the time, there is no overflow, so we don't have to revert to the more expensive and less precise incremental calculation of the mean value. Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-10 19:20:24 +02:00
beorn7	3a908d8e08	promql: Improve Kahan usage in avg_over_time ... The calculation of the mean value in avg_over_time is performed in an incremental fashion. This introduces additional numerical errors that even Kahan summation cannot compensate, but at least we can use the Kahan-corrected mean value when we use the intermediate mean value in the calculation. Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-10 19:20:24 +02:00
beorn7	c46074f4dd	promql: make avg aggregation more precise and less expensive ... The basic idea here is that the previous code was always doing incremental calculation of the mean value, which is more costly and can be less precise. It protects against overflows, but in most cases, an overflow doesn't happen anyway. The other idea applied here is to expand on #14074, where Kahan summation was applied to sum(). With this commit, the average is calculated in a conventional way (adding everything up and divide in the end) as long as the sum isn't overflowing float64. This is combined with Kahan summation so that the avg aggregation, in most cases, is really equivalent to the sum aggregation with a following division (which is the user's expectation as avg is supposed to be syntactic sugar for sum with a following divison). If the sum hits ±Inf, the calculation reverts to incremental calculation of the mean value. Kahan summation is also applied here, although it cannot fully compensate for the numerical errors introduced by the incremental mean calculation. (The tests added in this commit would fail if incremental mean calculation was always used.) Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-10 19:20:24 +02:00
darshanime	bd4ea118e9	Allow durations for number rule ... Signed-off-by: darshanime <deathbullet@gmail.com>	2024-07-10 15:50:22 +02:00
darshanime	cfad8ff3b2	Deprecate duration token ... Signed-off-by: darshanime <deathbullet@gmail.com>	2024-07-10 15:49:58 +02:00
darshanime	8c8860d2d6	Allow number literals as duration ... Signed-off-by: darshanime <deathbullet@gmail.com>	2024-07-10 15:49:30 +02:00
Filip Petkovski	acb6c1ae4b	Fix decoding buckets for native histograms in binops ... The optimizer which detects cases where histogram buckets can be skipped does not take into account binary expressions. This can lead to buckets not being decoded if a metric is used with both histogram_fraction/quantile and histogram_sum/count in the same expression. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-10 11:55:29 +02:00
Jan Fajerski	c0d67fd845	Merge branch 'main' into 3.0-main-sync-24-07-09	2024-07-09 16:51:07 +02:00
beorn7	b5b04ddbe3	promql: add avg aggregation benchmark ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-04 18:59:30 +02:00
beorn7	44d8c1d182	nit: add period at end of sentence ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-04 17:31:39 +02:00
beorn7	9a837b7f3c	promql: Make groupedAggregation.groupCount a float64 ... It's always used as such. Let's avoid the countless conversions. Signed-off-by: beorn7 <beorn@grafana.com>	2024-07-04 17:31:34 +02:00
JuanJo Ciarlante	c94c5b64c3	feat: add limitk() and limit_ratio() operators (#12503) ... * rebase 2024-07-01, picks previous renaming to `limitk()` and `limit_ratio()` Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * gofumpt -d -extra Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * more lint fixes Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * more lint fixes+ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * put limitk() and limit_ratio() behind --enable-feature=promql-experimental-functions Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * EnableExperimentalFunctions for TestConcurrentRangeQueries() also Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * use testutil.RequireEqual to fix tests, WIP equivalent thingie for require.Contains Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * lint fix Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * moar linting Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * rebase 2024-06-19 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * re-add limit(2, metric) testing for N=2 common series subset Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move `ratio = param` to default switch case, for better readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * gofumpt -d -extra util/testutil/cmp.go Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * early break when reaching k elems in limitk(), should have always been so (!) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * small typo fix Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change small break-loop rearrange for readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove IsNan(ratio) condition in switch-case, already handled as input validation Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change adding some comments Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * no-change simplify fullMatrix() helper functions used for tests Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add `limitk(-1, metric)` testcase, which is handled as any k < 1 case Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * engine_test.go: no-change create `requireCommonSeries() helper func (moving code into it) for readability Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * rebase 2024-06-21 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * engine_test.go: HAPPY NOW about its code -> reorg, create and use simpleRangeQuery() function, less lines and more readable ftW \o/ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move limitk(), limit_ratio() testing to promql/promqltest/testdata/limit.test Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove stale leftover after moving tests from engine_test.go to testdata/ Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix flaky `limit_ratio(0.5, ...)` test case Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Update promql/engine.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix AddRatioSample() implementation to use a single conditional (instead of switch/case + fallback return) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * docs/querying/operators.md: document r < 0 Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add negative limit_ratio() example to docs/querying/examples.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * move more extensive docu examples to docs/querying/operators.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * typo Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * small docu fix for poor-mans-normality-check, add it to limit.test ;) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * limit.test: expand "Poor man's normality check" to whole eval range Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * restore mistakenly removed existing small comment Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * expand poors-man-normality-check case(s) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * Revert "expand poors-man-normality-check case(s)" This reverts commit f69e1603b2ebe69c0a100197cfbcf6f81644b564, indeed too flaky 0:) Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove humor from docs/querying/operators.md Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix signoff Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add web/ui missing changes Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * expand limit_ratio test cases, cross-fingering they'll not be flaky Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * remove flaky test Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add missing warnings.Merge(ws) in instant-query return shortcut Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * add missing LimitK||LimitRatio case to codemirror-promql/src/parser/parser.ts Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * fix ui-lint Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> * actually fix returned warnings :] Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> --------- Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com> Co-authored-by: Julius Volz <julius.volz@gmail.com>	2024-07-03 22:18:57 +02:00
Arve Knudsen	493e196f10	Fix bad merge ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-03 15:12:40 +02:00
Arve Knudsen	020b0f200a	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-07-03 15:12:12 +02:00
Jan Fajerski	b4152309a4	Merge branch 'main' into 3.0-main-sync ... Conflicts: promql/engine_test.go Resolved by picking main changes but adjusting total_samples for query "max_over_time(metricWith1HistogramEvery10Seconds[60s])[20s:5s]" to 312. Via https://github.com/prometheus/prometheus/pull/13662 this histogram now stores 13 values per timestamp, but via https://github.com/prometheus/prometheus/pull/13904 the range query is now left-open. promql/promqltest/testdata/functions.test Resolved by picking changes in main. See also https://github.com/prometheus/prometheus/pull/13662, but adjust some range selectors (`s/1m/2m/`) to account for https://github.com/prometheus/prometheus/pull/13904. promql/promqltest/testdata/histograms.test Resolved by picking changes in main. See also https://github.com/prometheus/prometheus/pull/13662, but adjust some range selectors (`s/5m/10m/`) to account for https://github.com/prometheus/prometheus/pull/13904. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-07-03 09:45:25 +02:00
Ganesh Vernekar	3d54bcc018	Merge pull request #14362 from charleskorn/charleskorn/sum-infinity	2024-07-03 01:05:03 -04:00
Arve Knudsen	e8ae8cf012	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-07-01 10:47:21 +02:00
Charles Korn	fd6bdf5230	Fix issue where summation of +/- infinity returns NaN instead of infinity ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-06-28 11:26:54 +10:00
Björn Rabenstein	2e58d46522	Merge pull request #13662 from prometheus/nhcb ... Native histograms custom buckets storage	2024-06-27 21:44:20 +02:00
Björn Rabenstein	19da579799	Merge pull request #12821 from rexagod/12769 ... parser: support underscores	2024-06-27 17:04:18 +02:00
Pranshu Srivastava	3c9e3ee552	parser: support underscores ... Support underscores in numbers, namely, decimals, hexadecimals, and exponentials. Fixes #12769 Signed-off-by: Pranshu Srivastava <rexagod@gmail.com> Signed-off-by: Pranshu Srivastava <rexagod@gmail.com>	2024-06-27 19:20:52 +05:30
Bryan Boreham	b6aba4ff14	Merge pull request #14074 from bboreham/kahan-sum-sum ... [ENHANCEMENT] PromQL: use Kahan summation for sum()	2024-06-24 11:13:26 +01:00
🌲 Harry 🌊 John 🏔	d5f6887294	Pass limit param as hint to storage.Querier ... Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>	2024-06-20 09:47:38 -07:00
Jeanette Tan	dda5f48c9e	Merge branch 'main' into nhcb-review-2	2024-06-20 22:50:00 +08:00
Jeanette Tan	fc9dc72028	remove eval_with_nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-20 22:49:00 +08:00
Jeanette Tan	a6d788b1be	update missed suggested change from code review ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-20 22:49:00 +08:00
Zhang Zhanpeng	debbdb8608	make matrix selection and lookback left-open and right-closed ... Signed-off-by: Zhang Zhanpeng <zhangzhanpeng.zzp@alibaba-inc.com> Signed-off-by: beorn7 <beorn@grafana.com> Co-authored-by: beorn7 <beorn@grafana.com>	2024-06-20 22:05:40 +08:00
Bryan Boreham	84602bbace	Merge branch 'main' into fix-matcher-string-with-empty-label-name	2024-06-19 05:56:25 -04:00
Charles Korn	aeec30f082	Convert TestTimestampFunction_StepsMoreOftenThanSamples ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-06-17 16:56:56 +10:00
Charles Korn	987fa5c6a2	Convert range query test cases to test scripting language ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-06-17 16:43:01 +10:00
Arve Knudsen	b7320ef636	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	2024-06-14 10:51:35 +02:00
Oleg Zaytsev	03cf6141d4	Fix Matcher.String() with empty label name ... When the label name is empty, which can happen now with quoted label name, it should be quoted when printed as a string again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-06-13 18:46:35 +02:00
Björn Rabenstein	08621bebe9	Merge pull request #14269 from prometheus/beorn7/histogram-test ... promql: Add tests for histogram counter reset only in bucket	2024-06-08 16:59:47 +02:00
Jeanette Tan	b8cb12b989	rename hist to histogram according to code review ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 20:26:41 +08:00
Jeanette Tan	4c2aa872d4	update readme for testing framework ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 20:21:05 +08:00
Jeanette Tan	14f8dded39	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 19:17:14 +08:00
Jeanette Tan	b0e320425c	refine test rewriting regex and add validation ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 18:50:59 +08:00
Jeanette Tan	9adc1699c3	fix according to code review ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-06-07 18:50:59 +08:00
Charles Korn	1f988f77ff	promql: extend test scripting language to support asserting on expected error message (#14038) ... Add ability to assert that a query fails with a particular error message This also adds documentation for the test scripting language in general, including the new feature. Signed-off-by: Charles Korn <charles.korn@grafana.com> --------- Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-06-06 17:56:25 +02:00
beorn7	c7fdfe8004	promql: Add tests for histogram counter reset only in bucket ... This also exercises the "fast path" (only decoding count and sum), i.e. where the counter reset isn't visible at all in the decoded data. Signed-off-by: beorn7 <beorn@grafana.com>	2024-06-06 17:47:38 +02:00
Filip Petkovski	6e68046c25	Implement histogram statistics decoder (#14097) ... Implement histogram statistics decoder This commit speeds up histogram_count and histogram_sum functions on native histograms. The idea is to have separate decoders which can be used by the engine to only read count/sum values from histogram objects. This should help with reducing allocations when decoding histograms, as well as with speeding up aggregations like sum since they will be done on floats and not on histogram objects. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>	2024-06-06 17:17:13 +02:00
Arve Knudsen	e57aac8084	Merge remote-tracking branch 'prometheus/main' into arve/close-engine ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-06-05 11:37:44 +02:00
Charles Korn	24ef000025	Merge branch 'main' into sort-matrix-series ... # Conflicts: # promql/engine_test.go	2024-05-31 12:44:27 +10:00
Arve Knudsen	0cc99e677a	promql.Engine: Add Close method ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-28 12:01:47 +02:00
Arve Knudsen	f3b8750339	Join errors ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-27 17:14:17 +02:00
Arve Knudsen	7b56353090	Merge remote-tracking branch 'prometheus/main' into arve/query-logger-munmap	2024-05-27 17:08:33 +02:00
Björn Rabenstein	114dc5c393	Merge pull request #13638 from NeerajGartia21/promql-test ... Converts existing native histogram unit tests to the PromQL testing framework	2024-05-19 19:35:36 +02:00
Arve Knudsen	0f01d4b336	Fix flaky test ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-15 21:58:56 +02:00
Arve Knudsen	bf8d88f326	Merge remote-tracking branch 'origin/main' into arve/query-logger-munmap	2024-05-15 21:02:03 +02:00
Oleksandr Redko	f10c3454e9	Enable perfsprint linter and fix up code ... Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>	2024-05-15 17:51:05 +03:00
Björn Rabenstein	e6be4240be	Merge pull request #14068 from colega/quote-label-name-in-matchers-when-needed ... Bugfix: quote label name in matchers when needed	2024-05-14 17:18:58 +02:00
Jeanette Tan	f028496133	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-05-14 16:20:15 +08:00
Neeraj Gartia	661856cb65	removes the added tests from engine_test.go ... Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2024-05-13 22:58:25 +05:30
Neeraj Gartia	6119124d0e	some nits ... Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2024-05-13 22:55:38 +05:30
Neeraj Gartia	adf5a36c1e	adds test for sum, count, stddev, stdvar, quantile and fraction func to promql testing framework ... Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2024-05-13 22:55:38 +05:30
Neeraj Gartia	8b838a05d9	adds test for native histogram rate func in promql testing framework ... Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2024-05-13 22:55:38 +05:30
Neeraj Gartia	548bd9d6fb	adds TestNativeHistogramRate func to promql test framework ... Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2024-05-13 22:55:38 +05:30
Charles Korn	0e934dba8e	Capture timing information while sorting ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-05-13 19:47:18 +10:00
Charles Korn	036c87223c	Ensure series in matrix values returned for instant queries are always sorted ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-05-13 11:03:15 +10:00
Charles Korn	a4381608a0	Add failing test case ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-05-13 11:03:14 +10:00
Bryan Boreham	ea82b49c33	[ENHANCEMENT] PromQL: use Kahan summation for sum() ... This can give a more precise result, by keeping a separate running compensation value to accumulate small errors. See https://en.wikipedia.org/wiki/Kahan_summation_algorithm Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-09 14:29:38 +01:00
Arve Knudsen	aabe4d6e4a	promql.ActiveQueryTracker: Unmap mmapped file when done ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-09 14:27:29 +02:00
Bryan Boreham	3fd24d1cd7	Merge pull request #13999 from bboreham/extract-promqltest ... [Test] Extract most PromQL test code into separate packages	2024-05-09 13:23:11 +01:00
Bryan Boreham	786e0e706c	test: PromQL: stop using internal fields of engine ... * set enablePerStepStats and lookback duration via `NewTestEngine` parameters. * check maxSamples by recreating query engine * check lookback without modifying internals Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 19:58:41 +01:00
Bryan Boreham	0dbfd20b69	test: move most PromQL tests into separate test package ... So that they can import promqltest which imports promql. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:28:56 +01:00
Bryan Boreham	4a72607c4a	refactor: extract some PromQL Engine tests which use unexported structs ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:27:17 +01:00
Bryan Boreham	e7c77f7b40	promql: export NewTestQuery ... So that tests can call it from another package. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:08:04 +01:00
Bryan Boreham	babfcfdd91	refactor: Move NewTestEngine into promqltest ... And export `DefaultMaxSamplesPerQuery` so callers can replicate previous behaviour. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:07:28 +01:00
Oleg Zaytsev	dabd789fd5	Quote label name in matchers when needed ... When the label name of a matcher contains non-standard characters, like a dot, or starts with a digit, it should be quoted. If it's not quoted, then `VectorSelector.String()` isn't a valid PromQL. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-05-08 16:58:51 +02:00
Bryan Boreham	b3c15d2246	test: clean up promqltest package references ... So it nearly compiles. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	9aa321d672	test: make field initializers explicit ... Lint started complaining after I moved the file. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	ccf73266f0	test: move promqltest tests together with the implementation ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	11b27d5d22	test: move test files into new promqltest package ... So that promql package does not bring in test-only dependencies. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	a6e427660a	test: check for @-modifier without using engine internals ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	2b0c87b1b6	test: turn TestKahanSum into scripted test ... This saves having a function solely to call kahanSumInc. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:55 +01:00
Bryan Boreham	a1af3c27d4	refactor: extract almost.Equal() to new package ... To avoid a circular reference between promql and promqltest. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 13:42:10 +01:00
zenador	545a7e48f9	[nhcb branch] Enhance unit test framework for nhcb and warnings (#14009) ... * process custom values in histogram unit test framework * check for warnings when evaluating in unit test framework * add test cases for custom buckets in test framework Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-05-08 13:58:24 +02:00
Jeanette Tan	796b1bbfde	Merge branch 'main' into nhcb ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-05-08 19:11:39 +08:00
Arve Knudsen	a25160e6a4	[REFACTOR] PromQL: simplify rangeEvalTimestampFunctionOverVectorSelector (#14021) ... The function `rangeEvalTimestampFunctionOverVectorSelector` appeared to be checking histogram size, however the value it used was always 0 due to subtle variable shadowing. However we don't need to pass sample values to the `timestamp` function, since the latter only cares about timestamps. This also affects peak sample count in statistics, since we are no longer copying histogram samples. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-05-08 10:39:44 +01:00
Oleg Zaytsev	4b7a44c7a0	Fix parser.VectorSelector.String() with empty name matcher (#14015) ... The check fell into "this matcher equals vector selector's name" case when vector selector doesn't have a name and the matcher is an explicit matcher for an empty __name__ label. To provide some context about why this is important: some downstream projects use the promql.Parse(expr.String()) to clone an expression's AST, and with this bug that matcher disappears in the cloning. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-05-06 11:51:08 +02:00
guangwu	9fda9443d4	fix(promql/query_logger): close file in error handling (#13948) ... Signed-off-by: guoguangwu <guoguangwug@gmail.com>	2024-04-30 10:47:10 +02:00
György Krajcsovits	bcafa5f1f9	Merge remote-tracking branch 'upstream/main' into update-nhcb	2024-04-24 11:06:59 +02:00
zenador	81862aabd7	[nhcb branch] Add basic unit tests for native histograms with custom buckets converted from classic histograms (#13794) ... * modify unit test framework to automatically generate native histograms with custom buckets from classic histogram series * add very basic tests for classic histogram converted into native histogram with custom bounds * fix histogram_quantile for native histograms with custom buckets * make loading with nhcb explicit * evaluate native histograms with custom buckets on queries with explicit keyword * use regex replacer * use temp histogram struct for automatically loading converted nhcb Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2024-04-24 09:36:05 +02:00
Owen Williams	4a6f8704ef	parser: remake generated_parser output (#13923) ... In a previous PR, the generated parser was created using an old version of goyacc. Also adds -l to disable line directives, which fixes debug processing and reduces diffs at the expense of making it more difficult to reason about the generated output. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-04-13 12:59:54 +02:00
Neeraj Gartia	612de026da	Adds Inf and NaN as Numbers to Histogram in Promql Testing Framework (#13916) ... includes Inf and NaN as numbers to histogram --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2024-04-11 12:53:28 +02:00
Matthieu MOREL	6f595c6762	golangci-lint: enable whitespace linter (#13905) ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-04-11 09:27:54 +01:00
Bryan Boreham	12961c6a37	promql: refactor: eliminate one 'else' ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	0ac927515b	promql: move group-seen into group struct ... Save allocating an auxilliary array. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	7499d90913	promql: remove pointer to aggregation groups ... Just allocate in one slice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	cfbeb6681b	promql: re-use one heap for topk and bottomk ... Slightly ugly casting saves memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	5e3914a27c	promql: remove histogramMean from groupedAggregation ... Re-use histogramValue since we don't need them separately. Tidy up initialization. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	2cf3c9de8f	promql: store labels per-group only for count_values ... This saves memory in other kinds of aggregation. We don't need `orderedResult` in `aggregationCountValues`; the ordering is not guaranteed. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	185290a0d2	promql: pull checking of q and k out of loop ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	4584f67e17	promql: inline nextSample function ... Move Sample out of loop to reduce allocations, otherwise it escapes to the heap. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	526ce4ee7a	promql: simplify data collection in aggregations ... We don't need a Sample, just the float and histogram values. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	2f03acbafc	promql: refactor: split topk/bottomk from sum/avg/etc ... They aggregate results in different ways. topk/bottomk don't consider histograms so can simplify data collection. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	74eed67ef6	promql: refactor: pull fetching input data out of rangeEvalAgg ... This is a cleaner split of responsibilities. We now check the sample count after calling rangeEvalAgg. Changed re-use of samples to use `Clone` and `defer`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	602eb69edf	promql: refactor: extract function nextSample ... With sub-function nextValues which we shall use shortly. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	eb41e770b7	promql: refactor: extract function addToSeries ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	53a3138eeb	promql aggregations: pre-generate mapping from inputs to outputs ... So we don't have to re-create it on every time step. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	cb6c4b3092	promql: simplify k/q parameter to topk/bottomk/quantile ... Pass it as a float64 not as interface{}. Make k a simple int, since that is the parameter to make(). Pull invalid quantile warning out of the loop. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	b3bda7df4b	promql: aggregations: skip copying input to a Vector ... We can work directly from the inputMatrix on each timestep. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	c9b6c4c55a	promql: aggregations: output directly to matrix for instant queries ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	3851b74db1	promql: aggregations: skip result vector in range queries ... Adjust test to match the lower count, since samples in the vector are no longer counted. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	59548b8a0b	promql: refactor: move collection of results into aggregation() ... We don't need to check for duplicates as aggregation cannot generate them. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	bd9bdccb22	promql: refactor: simplify internal data structures ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	5f10d17cef	promql: refactor: split out aggregations over range ... The new function `rangeEvalAgg` is mostly a copy of `rangeEval`, but without `initSeries` which we don't need and inlining the callback to `aggregation()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	e5f667537c	promql: refactor: initialize aggregation before storing in map ... This seems more consistent to me. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	29244fb841	promql: refactor: extract count_values implementation ... The existing aggregation function is very long and covers very different cases. `aggregationCountValues` is just for `count_values`, which differs from other aggregations in that it outputs as many series per group as there are values in the input. Remove the top-level switch on string parameter type; use the same `Op` check there as elswehere. Pull checking parameters out to caller, where it is only executed once. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	8e04ab6dd4	promql: refactor: extract generateGroupingLabels function ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-05 15:47:54 +01:00
Bryan Boreham	7c28521451	[TESTS] Truncate some long test names, for readability ... The strings produced by these tests can run to thousands of characters, which makes test logs difficult to read. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-04-03 10:10:39 +01:00
Charles Korn	cd72ebb05f	promql: include more details in error message when creating test query fails or an unexpected series is returned (#13847) ... * promql: include more details in error message when creating test query fails Signed-off-by: Charles Korn <charles.korn@grafana.com> * Include more details when an unexpected metric is returned Signed-off-by: Charles Korn <charles.korn@grafana.com> --------- Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-04-03 10:57:08 +02:00
Julius Volz	9b7de47787	Remove unused Dmn field on EvalNodeHelper (#13877) ... https://github.com/prometheus/prometheus/pull/13446 removed the last usage of this field, but didn't remove the field. Signed-off-by: Julius Volz <julius.volz@gmail.com>	2024-04-02 18:45:46 +02:00
beorn7	65b4696b88	promql: Remove leftover debug output ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-03-27 19:02:27 +01:00
beorn7	2c1f9558b2	promql: Fix histogram comparison in test framework ... The definition of histograms in the test framework may create histograms in a non-compact form. Since histogram comparison relies on exact equality of the bucket layout, we have to compact the histograms created by the test framework language before comparing them to histograms returned from the PromQL engine. Signed-off-by: beorn7 <beorn@grafana.com>	2024-03-27 19:00:16 +01:00
György Krajcsovits	2a4aa085d2	Merge branch 'main' into nhcb	2024-03-27 18:42:10 +01:00
George Krajcsovits	dc7b282d39	engine_test: adjust and comment histogram sample counts (#13841) ... The size of histogram points are now bigger by 24 bytes due to the custom values slice. When histograms are loaded into partial results in vector selectors we use HPoint type where the size is calculated as (size of histogram + 8 for timestamp)/16. a3d1a46eda/promql/value.go (L176) When histograms are put into Sample type in range evaluations, the Sample has more overhead and the size is calculated differently: (size of histogram / 16) + 1 for time stamp. a3d1a46eda/promql/engine.go (L1928) When the size of the histogram is 16k, then the first calculation gives k but the second gives k+1 for the sample count. If the histogram size is 16k+8, then both would give k+1. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-03-27 18:19:14 +01:00
Björn Rabenstein	b9a2a4e329	Merge pull request #13852 from prometheus/fix-hist-std-dev-var-negative ... Fix hist std dev var negative	2024-03-27 17:58:03 +01:00
Jeanette Tan	4f2df329bd	improve handling of empty buckets with infinite bounds in histogram std dev/var ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-03-27 17:06:12 +01:00
Jeanette Tan	22d0f4f114	improve handling of negative bounds in histogram std dev/var ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-03-27 17:06:12 +01:00
Domantas	435f330d0b	[BUGFIX] labels: don't modify original labels in DropMetricName (#13845) ... Restrict the capacity of first argument to `append()` to force an allocation. This is for the slice implementation only. Signed-off-by: Domantas Jadenkus <djadenkus@gmail.com>	2024-03-27 10:35:17 +00:00
Charles Korn	5cc97a1820	[tests]: extend test scripting language to support range queries (#13825) ... * Extract method to make it easier to test. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Remove superfluous interface definition. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Add test cases for existing instant query functionality. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Add support for testing range queries Signed-off-by: Charles Korn <charles.korn@grafana.com> * Expand test coverage for instant queries and clarify error when a float is returned but a histogram is expected (or vice versa) Signed-off-by: Charles Korn <charles.korn@grafana.com> * Improve error message formatting Signed-off-by: Charles Korn <charles.korn@grafana.com> * Add test case for instant query command with invalid timestamp Signed-off-by: Charles Korn <charles.korn@grafana.com> * Fix linting warning. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Remove superfluous print statement and expected result Signed-off-by: Charles Korn <charles.korn@grafana.com> * Fix linting warning. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Add note about ordered range eval commands. Signed-off-by: Charles Korn <charles.korn@grafana.com> * Check that matrix results are always sorted by labels. Signed-off-by: Charles Korn <charles.korn@grafana.com> --------- Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-03-26 11:22:22 +00:00
Bryan Boreham	080d440bf8	Merge remote-tracking branch 'origin/main' into pr/13461	2024-03-25 12:14:26 +00:00
Bryan Boreham	773170f372	Merge pull request #13822 from dgl/promtool-test-errors ... promtool: Avoid using testify for user rule tests	2024-03-23 09:42:34 +01:00
György Krajcsovits	a3d1a46eda	Merge branch 'main' into nhcb	2024-03-22 14:51:48 +01:00
Jeanette Tan	9d32754bc0	add unit tests with all negative values for histogram_stddev and var ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-03-22 03:42:50 +08:00
David Leadbeater	7ec4a11472	promtool: Avoid using testify for user rule tests ... Using testify outside of unit tests results in panics rather than a useful error for the user. Fixes #13703 Signed-off-by: David Leadbeater <dgl@dgl.cx>	2024-03-21 22:08:10 +11:00
tdakkota	f6834c347a	promql: validate label_join destination label ... Signed-off-by: tdakkota <tanc13@yandex.ru>	2024-03-20 22:02:10 +03:00
Bartlomiej Plotka	312e3fd728	Merge pull request #13713 from charleskorn/query-engine-interface ... rules: allow using alternative PromQL engines for rule evaluation by callers using Prometheus as a lib.	2024-03-13 14:45:42 +01:00
Björn Rabenstein	af3618fd35	Merge pull request #13667 from prometheus/beorn7/promql ... Improve TestQueryStatistics and fix bugs exposed by it	2024-03-12 16:17:11 +01:00
Charles Korn	26262a1eb7	Remove unnecessary SetQueryLogger method on QueryEngine interface ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-03-12 22:01:01 +11:00
carrychair	856f6e49c8	fix function and struct name ... Signed-off-by: carrychair <linghuchong404@gmail.com>	2024-03-09 17:53:17 +08:00
beorn7	7f912db15a	promql: Fix limiting of extrapolation to negative values ... This is a bit tough to explain, but I'll try: `rate` & friends have a sophisticated extrapolation algorithm. Usually, we extrapolate the result to the total interval specified in the range selector. However, if the first sample within the range is too far away from the beginning of the interval, or if the last sample within the range is too far away from the end of the interval, we assume the series has just started half a sampling interval before the first sample or after the last sample, respectively, and shorten the extrapolation interval correspondingly. We calculate the sampling interval by looking at the average time between samples within the range, and we define "too far away" as "more than 110% of that sampling interval". However, if this algorithm leads to an extrapolated starting value that is negative, we limit the start of the extrapolation interval to the point where the extrapolated starting value is zero. At least that was the intention. What we actually implemented is the following: If extrapolating all the way to the beginning of the total interval would lead to an extrapolated negative value, we would only extrapolate to the zero point as above, even if the algorithm above would have selected a starting point that is just half a sampling interval before the first sample and that starting point would not have an extrapolated negative value. In other word: What was meant as a _limitation_ of the extrapolation interval yielded a _longer_ extrapolation interval in this case. There is an exception to the case just described: If the increase of the extrapolation interval is more than 110% of the sampling interval, we suddenly drop back to only extrapolate to half a sampling interval. This behavior can be nicely seen in the testcounter_zero_cutoff test, where the rate goes up all the way to 0.7 and then jumps back to 0.6. This commit changes the behavior to what was (presumably) intended from the beginning: The extension of the extrapolation interval is only limited if actually needed to prevent extrapolation to negative values, but the "limitation" never leads to _more_ extrapolation anymore. The difference is subtle, and probably it never bothered anyone. However, if you calculate a rate of a classic histograms, the old behavior might create non-monotonic histograms as a result (because of the jumps you can see nicely in the old version of the testcounter_zero_cutoff test). With this fix, that doesn't happen anymore. Signed-off-by: beorn7 <beorn@grafana.com>	2024-03-07 01:20:33 +01:00
Charles Korn	4e77e8e5ef	Allow using alternative PromQL engines for rule evaluation ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-03-06 14:54:33 +11:00
beorn7	f48c7a5503	promql: Add histograms to TestQueryStatistics ... Also, fix the bugs exposed by the tests. Signed-off-by: beorn7 <beorn@grafana.com>	2024-02-29 19:02:40 +01:00
beorn7	f46dd34982	promql: Add code comment ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-02-29 19:02:40 +01:00
beorn7	7d364c0451	promql: remove redundant line ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-02-29 19:02:40 +01:00
Björn Rabenstein	d5f0a240fa	Merge pull request #13674 from prometheus/owilliams/dupeok ... fix: restore ability to match __name__ multiple times in selector	2024-02-29 17:49:15 +01:00
Björn Rabenstein	9187bcbdd5	Merge pull request #13536 from bboreham/faster-label-replace ... promql: faster range-query of label_replace and label_join	2024-02-29 17:03:00 +01:00
Owen Williams	e01e7d36e2	fix: restore ability to match __name__ multiple times in selector ... Add tests to cover this case. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-02-29 10:46:31 -05:00
machine424	f477e0539a	Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 ... Prevent adding back golang.org/x/exp/slices. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-02-28 14:54:53 +01:00
György Krajcsovits	5d0a0a7542	Add custom buckets to native histogram model (#13592) ... * add custom buckets to native histogram model * simple copy for custom bounds * return errors for unsupported add/sub operations * add test cases for string and update appendhistogram in scrape to account for new schema * check fields which are supposed to be unused but may affect results in equals * allow appending custom buckets histograms regardless of max schema Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-02-28 14:06:43 +01:00
Björn Rabenstein	11932cd345	Merge pull request #13637 from bboreham/agg-warning ... PromQL: improve warning for mixed values in aggregations	2024-02-28 13:52:30 +01:00
Owen Williams	ac51a8024c	tests(utf8): confirm that other quote marks are handled correctly in promql ... Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-02-27 15:47:47 -05:00
Bryan Boreham	0347148628	promql: fuzz test needs symbol table for parser ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-26 11:45:25 +00:00
Bryan Boreham	22890b1eb3	PromQL: improve warning for mixed values in aggregations ... Aggregations discard the metric name, so don't try to include it in the error message. Add a test that generates this warning. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-25 19:57:06 +00:00
Owen Williams	a28d7865ad	UTF-8: Add support for parsing UTF8 metric and label names ... This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats. This grammar will also be valid for non-UTF-8 names. UTF-8 names will not be considered valid unless model.NameValidationScheme is changed. This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/. Part of https://github.com/prometheus/prometheus/issues/13095 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-02-15 14:34:37 -05:00
Bryan Boreham	ff6c83269c	Merge pull request #13452 from bboreham/go-cmp ... Tests: Use DeepEqual replacement using go-cmp, which is more flexible	2024-02-12 15:40:08 +01:00
Bryan Boreham	17f48f2b3b	Tests: use replacement DeepEquals in more places ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-08 19:32:33 +00:00
Bryan Boreham	39af788dbd	Tests: use replacement DeepEquals using go-cmp ... Use DeepEqual replacement using go-cmp, which is more flexible. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-08 19:30:20 +00:00
beorn7	86d7618d84	promql: Fix wrongly scoped range vectors ... Fixes #11708. If a range vector is fixen in time with the @ modifier, it gets still moved around for different steps in a range query. Since no additional points are retrieved from the TSDB, this leads to steadily emptying the range, leading to the weird behavior described in isse #11708. This only happens for functions listed in `AtModifierUnsafeFunctions`, and the only of those that takes a range vector is `predict_linear`, which is the reason why we see it only for this particular function. Signed-off-by: beorn7 <beorn@grafana.com>	2024-02-07 18:07:51 +01:00
beorn7	384ab025e0	promql: Expose issue #11708 ... The shorter range in the test makes early points drop out of the range in range queries, exposing the issue. Signed-off-by: beorn7 <beorn@grafana.com>	2024-02-07 18:06:05 +01:00
Bryan Boreham	98c4889029	Merge pull request #9298 from Creatone/creatone/use-testify ... tests: Move from t.Errorf and others.	2024-02-04 16:27:57 +01:00
Bryan Boreham	d3c1f0d8e0	promql: can now remove regex field from EvalNodeHelper ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-04 11:32:26 +01:00
Bryan Boreham	fdd5b85e06	promql: faster range-query of label_replace and label_join ... These functions act on the labels only, so don't need to go step by step over the samples in a range query. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-04 11:12:55 +01:00
Faustas Butkus	6feffeb92e	promql: add histogram_avg function (#13467) ... Add histogram_avg function --------- Signed-off-by: Faustas Butkus <faustas.butkus@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2024-02-01 18:28:42 +01:00