Commit graph

852 commits

Author SHA1 Message Date
Ayoub Mrini ace9c8a3da
promtool: allow setting multiple matchers to "promtool tsdb dump" command. (#13296)
Conditions are ANDed inside the same matcher but matchers are ORed

Including unit tests for "promtool tsdb dump".

Refactor some matchers scraping utils.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-01-15 10:29:53 +00:00
Bryan Boreham 252031c86f Revert "Adding small test update for temp dir using t.TempDir (#13293)"
This reverts commit 2ddb3596ef.

Various tests are failing in CI after this change; reverting to free up
other work.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-30 19:17:30 +00:00
Mile Druzijanic 2ddb3596ef
Adding small test update for temp dir using t.TempDir (#13293)
* Adding small test update for temp dir using t.TempDir

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

* removing not required cleanup

Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

---------

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>
2023-12-28 21:49:57 +01:00
Filip Petkovski 0e1ae1d1ca
Add comment
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-25 11:41:07 +01:00
Filip Petkovski 35f9620cd1
Expand benchmark
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-25 11:30:29 +01:00
Filip Petkovski 5df3820c7a
Copy last histogram point
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-25 11:20:51 +01:00
Filip Petkovski 1f69dcfa6b
Fix reusing float histograms
In https://github.com/prometheus/prometheus/pull/13276 we started reusing float histogram objects to reduce allocations in PromQL.
That PR introduces a bug where histogram pointers gets copied to the beginning of the histograms slice,
but are still kept in the end of the slice. When a new histogram is read into the last element,
it can overwrite a previous element because the pointer is the same.

This commit fixes the issue by moving outdated points to the end of the slice
so that we don't end up with duplicate pointers in the same buffer. In other words,
the slice gets rotated so that old objects can get reused.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-14 11:53:58 +01:00
Filip Petkovski bb8363dbb3
Add comment on SampleRingIterator
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-13 08:30:02 +01:00
Filip Petkovski e2a9f8ac0f
Reuse float histogram objects
This commit reduces the memory needed to query native histogram objects
by reusing existing HPoint instances.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-11 08:24:58 +01:00
Björn Rabenstein db915b07cb
Merge pull request #13215 from fpetkovski/float-histogram-reuse
Enable reusing memory when converting between histogram types
2023-12-09 22:44:46 +01:00
Bartlomiej Plotka 91a383f52c
Merge pull request #13059 from zenador/add-mad-function
Add mad_over_time function
2023-12-08 11:53:22 +00:00
Filip Petkovski 10a82f87fd
Enable reusing memory when converting between histogram types
The 'ToFloat' method on integer histograms currently allocates new memory
each time it is called.

This commit adds an optional *FloatHistogram parameter that can be used
to reuse span and bucket slices. It is up to the caller to make sure the
input float histogram is not used anymore after the call.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:22:59 +01:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Jeanette Tan 2910b48180 Make mad_over_time experimental and move tests
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-12-01 01:56:07 +08:00
Jeanette Tan 9bf4cc993e Add mad_over_time function
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-12-01 01:22:58 +08:00
Björn Rabenstein 5dbbadf598
Merge pull request #13216 from prometheus/beorn7/doc
Update “conventional histogram” → “classic histogram”
2023-11-30 10:35:27 +01:00
Oleksandr Redko 2a75604f8e
Enable default revive rules (#13068)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-11-29 17:23:34 +00:00
beorn7 0eb0ca42c5 Update “conventional histogram” → “classic histogram”
Signed-off-by: beorn7 <beorn@grafana.com>
2023-11-29 15:22:58 +01:00
Julien Pivotto c1ec6ae851 sort_by_label: Switch to feature flag
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-11-28 15:10:12 +01:00
Alexander Trost 5051a993ab promql: add sort_by_label and sort_by_label_desc functions
This adds functions to sort a vector by its label value.

Based on https://github.com/prometheus/prometheus/pull/1533

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2023-11-28 14:40:07 +01:00
zenador ccfe14d7e7
PromQL: ignore small errors for bucketQuantile (#13153)
promql: Improve histogram_quantile calculation for classic buckets

Tiny differences between classic buckets are most likely caused by floating point precision issues. With this commit, relative changes below a certain threshold are ignored. This makes the result of histogram_quantile more meaningful, and also avoids triggering the _input to histogram_quantile needed to be fixed for monotonicity_ annotations in unactionable cases.

This commit also adds explanation of the new adjustment and of the monotonicity annotation to the documentation of `histogram_quantile`.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-25 00:05:38 +01:00
Filip Petkovski 35a15e8f04
Add benchmark for native histograms (#13160)
* Add benchmark for native histograms

This commit adds a PromQL benchmark for queries on native histograms.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-11-23 14:09:17 +00:00
Julien Pivotto c92fbf3fdf Add feature flag for PromQL experimental functions.
This PR adds an Experimental flag to the functions.

This can be used by https://github.com/prometheus/prometheus/pull/13059
but also xrate and other future functions.

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-11-14 17:48:58 +01:00
Linas Medziunas 1cd6c1cde5 ValidateHistogram: strict Count check in absence of NaNs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
Dimitar Dimitrov 9e3df532d8
Export promql.Engine.FindMinMaxTime
This function is useful to analyze promQL queries. We want to use this in Mimir to record the time range which the query touches.

I also chose to remove the `Engine` receiver because it was unnecessary, and it makes it easier to use, but happy to refactor that if you disagree.

The function is untested on its own. If you prefer to have unit tests now that its exported, I can look into adding some.

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-11-02 13:17:35 +01:00
Björn Rabenstein a43669e611
Merge pull request #12928 from alexandear/ci-enable-godot
ci(lint): enable godot; append dot at the end of comments
2023-11-01 17:15:41 +01:00
Julien Pivotto f568221610
Merge pull request #13057 from prometheus/release-2.48
Merge release-2.48 back into main
2023-10-31 15:24:39 -04:00
Oleksandr Redko fa90ca46e5 ci(lint): enable godot; append dot at the end of comments
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 19:53:38 +02:00
Oleksandr Redko 8e5f0387a2
ci(lint): enable nolintlint and remove redundant comments (#12926)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 12:35:13 +01:00
Bryan Boreham 49c5e7afe1 PromQL: reduce garbage in range-query evaluation
The temporary variable was allocated on the heap, and it is unnecessary.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-29 19:45:06 +00:00
zenador 80e977aae6
Remove NewPossibleNonCounterInfo and minimise creating empty annotations (#13012)
* Remove NewPossibleNonCounterInfo until it can be made more efficient, and avoid creating empty annotations as much as possible

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-24 17:36:07 +01:00
Marc Tuduri af7c31ee10
PR feedback
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:53:50 +02:00
Marc Tuduri 8fededf6ad
promql(histograms): Change sample total calculation for histograms
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:51:11 +02:00
Jeanette Tan 9a8bd8eac6 Fix possible non-counter warning for empty names and native histograms
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-16 15:52:10 +08:00
Julius Volz 191c24a0ed Fix: Exempt "_bucket" suffix from PossibleNonCounterInfo warning (#12982)
Related to PR #12152

Signed-off-by: Julius Volz <julius.volz@gmail.com>
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2023-10-15 13:47:42 -04:00
Jeanette Tan 0cbf0c1c68 Revise according to code review
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-06 19:09:32 +08:00
Jeanette Tan feaa93da77 Add warning when monotonicity is forced in the input to histogram_quantile
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-04 18:53:55 +08:00
Alan Protasio a15e884e7a
Prevent Prometheus from overallocating memory on subquery with large amount of steps. (#12734)
* change initial points slice size

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* refactor on the steps calculation and moving the getXPoint/putXPoint method to the evaluator

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* prevent potential panic

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update promql/engine.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update promql/engine.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update promql/engine.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update promql/engine.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Allocating slice with maximum size of 5k

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* adding comments

Signed-off-by: Alan Protasio <alanprot@gmail.com>

---------

Signed-off-by: Alan Protasio <alanprot@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2023-09-25 20:15:41 +01:00
Goutham Veeramachaneni 86729d4d7b
Update exp package (#12650) 2023-09-21 22:53:51 +02:00
Bryan Boreham 91054875d6
Merge pull request #12732 from bboreham/simplify-rangeeval
promql: simplify inner loop of rangeEval
2023-09-20 20:22:05 +00:00
zenador 69edd8709b
Add warnings (and annotations) to PromQL query results (#12152)
Return annotations (warnings and infos) from PromQL queries

This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations".

Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future.

The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms).

The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later.

Another TODO is to take annotations into account when evaluating recording rules.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-09-14 18:57:31 +02:00
Arve Knudsen 156222cc50
Add context argument to LabelQuerier.LabelValues (#12665)
Add context argument to LabelQuerier.LabelValues and
LabelQuerier.SortedLabelValues.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 16:02:04 +02:00
Arve Knudsen a964349e97
Add context argument to LabelQuerier.LabelNames (#12666)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 10:39:51 +02:00
Arve Knudsen 4451ba10b4
Add context argument to IndexReader.Postings (#12667)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-13 17:45:06 +02:00
Arve Knudsen 6daee89e5f
Add context argument to Querier.Select (#12660)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-12 12:37:38 +02:00
Julien Pivotto 284ba3426b
Merge pull request #12758 from bboreham/trim-rangequery-benchmarks
PromQL: reduce numbers of benchmarks
2023-09-08 14:06:21 +02:00
Bryan Boreham e4dd3469ac lint
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-28 10:39:16 +01:00
Bryan Boreham 5ce990cabc promql: simplify rangeEval a bit more
We can't have both a float and a histogram at the same timestep.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-28 10:28:09 +01:00
Bryan Boreham c5671c6d97
Merge pull request #12755 from bboreham/rangequery-benchmark-mmap
promql: force mmap of head chunks in BenchmarkRangeQuery
2023-08-26 15:56:52 +01:00
Bryan Boreham 1ea57a3f8c PromQL: reduce numbers of benchmarks
Make it more likely that contributors will run the benchmark suite.

count_values needs more than 2GB at 1,000 steps, so just run it for 100.

And remove 10-step variant because it doesn't add much to 100 and
1000-step benchmarks.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-26 14:12:28 +00:00