Commit graph

379 commits

Author SHA1 Message Date
Arve Knudsen a25160e6a4
[REFACTOR] PromQL: simplify rangeEvalTimestampFunctionOverVectorSelector (#14021)
The function `rangeEvalTimestampFunctionOverVectorSelector` appeared to be checking histogram size, however the value it used was always 0 due to subtle variable shadowing.
However we don't need to pass sample values to the `timestamp` function, since the latter only cares about timestamps. This also affects peak sample count in statistics, since we are no longer copying histogram samples.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-05-08 10:39:44 +01:00
Bryan Boreham 12961c6a37 promql: refactor: eliminate one 'else'
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 0ac927515b promql: move group-seen into group struct
Save allocating an auxilliary array.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 7499d90913 promql: remove pointer to aggregation groups
Just allocate in one slice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham cfbeb6681b promql: re-use one heap for topk and bottomk
Slightly ugly casting saves memory.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 5e3914a27c promql: remove histogramMean from groupedAggregation
Re-use histogramValue since we don't need them separately.

Tidy up initialization.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 2cf3c9de8f promql: store labels per-group only for count_values
This saves memory in other kinds of aggregation.

We don't need `orderedResult` in `aggregationCountValues`; the ordering
is not guaranteed.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 185290a0d2 promql: pull checking of q and k out of loop
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 4584f67e17 promql: inline nextSample function
Move Sample out of loop to reduce allocations, otherwise it escapes to
the heap.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 526ce4ee7a promql: simplify data collection in aggregations
We don't need a Sample, just the float and histogram values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 2f03acbafc promql: refactor: split topk/bottomk from sum/avg/etc
They aggregate results in different ways.
topk/bottomk don't consider histograms so can simplify data collection.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 74eed67ef6 promql: refactor: pull fetching input data out of rangeEvalAgg
This is a cleaner split of responsibilities.
We now check the sample count after calling rangeEvalAgg.
Changed re-use of samples to use `Clone` and `defer`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 602eb69edf promql: refactor: extract function nextSample
With sub-function nextValues which we shall use shortly.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham eb41e770b7 promql: refactor: extract function addToSeries
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 53a3138eeb promql aggregations: pre-generate mapping from inputs to outputs
So we don't have to re-create it on every time step.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham cb6c4b3092 promql: simplify k/q parameter to topk/bottomk/quantile
Pass it as a float64 not as interface{}.
Make k a simple int, since that is the parameter to make().
Pull invalid quantile warning out of the loop.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham b3bda7df4b promql: aggregations: skip copying input to a Vector
We can work directly from the inputMatrix on each timestep.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham c9b6c4c55a promql: aggregations: output directly to matrix for instant queries
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 3851b74db1 promql: aggregations: skip result vector in range queries
Adjust test to match the lower count, since samples in the vector
are no longer counted.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 59548b8a0b promql: refactor: move collection of results into aggregation()
We don't need to check for duplicates as aggregation cannot generate them.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham bd9bdccb22 promql: refactor: simplify internal data structures
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 5f10d17cef promql: refactor: split out aggregations over range
The new function `rangeEvalAgg` is mostly a copy of `rangeEval`, but
without `initSeries` which we don't need and inlining the callback to
`aggregation()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham e5f667537c promql: refactor: initialize aggregation before storing in map
This seems more consistent to me.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 29244fb841 promql: refactor: extract count_values implementation
The existing aggregation function is very long and covers very different
cases.

`aggregationCountValues` is just for `count_values`, which differs from
other aggregations in that it outputs as many series per group as there
are values in the input.

Remove the top-level switch on string parameter type; use the same `Op`
check there as elswehere.

Pull checking parameters out to caller, where it is only executed once.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Bryan Boreham 8e04ab6dd4 promql: refactor: extract generateGroupingLabels function
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-04-05 15:47:54 +01:00
Julius Volz 9b7de47787
Remove unused Dmn field on EvalNodeHelper (#13877)
https://github.com/prometheus/prometheus/pull/13446 removed the last usage of
this field, but didn't remove the field.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2024-04-02 18:45:46 +02:00
Bartlomiej Plotka 312e3fd728
Merge pull request #13713 from charleskorn/query-engine-interface
rules: allow using alternative PromQL engines for rule evaluation by callers using Prometheus as a lib.
2024-03-13 14:45:42 +01:00
Björn Rabenstein af3618fd35
Merge pull request #13667 from prometheus/beorn7/promql
Improve TestQueryStatistics and fix bugs exposed by it
2024-03-12 16:17:11 +01:00
Charles Korn 26262a1eb7
Remove unnecessary SetQueryLogger method on QueryEngine interface
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-03-12 22:01:01 +11:00
carrychair 856f6e49c8 fix function and struct name
Signed-off-by: carrychair <linghuchong404@gmail.com>
2024-03-09 17:53:17 +08:00
Charles Korn 4e77e8e5ef
Allow using alternative PromQL engines for rule evaluation
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-03-06 14:54:33 +11:00
beorn7 f48c7a5503 promql: Add histograms to TestQueryStatistics
Also, fix the bugs exposed by the tests.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
beorn7 f46dd34982 promql: Add code comment
Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
beorn7 7d364c0451 promql: remove redundant line
Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
Björn Rabenstein 9187bcbdd5
Merge pull request #13536 from bboreham/faster-label-replace
promql: faster range-query of label_replace and label_join
2024-02-29 17:03:00 +01:00
machine424 f477e0539a
Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21
Prevent adding back golang.org/x/exp/slices.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-02-28 14:54:53 +01:00
Bryan Boreham 22890b1eb3 PromQL: improve warning for mixed values in aggregations
Aggregations discard the metric name, so don't try to
include it in the error message.

Add a test that generates this warning.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-25 19:57:06 +00:00
beorn7 86d7618d84 promql: Fix wrongly scoped range vectors
Fixes #11708.

If a range vector is fixen in time with the @ modifier, it gets still
moved around for different steps in a range query. Since no additional
points are retrieved from the TSDB, this leads to steadily emptying
the range, leading to the weird behavior described in isse #11708.

This only happens for functions listed in `AtModifierUnsafeFunctions`,
and the only of those that takes a range vector is `predict_linear`,
which is the reason why we see it only for this particular function.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-07 18:07:51 +01:00
Bryan Boreham d3c1f0d8e0 promql: can now remove regex field from EvalNodeHelper
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-04 11:32:26 +01:00
Bryan Boreham fdd5b85e06 promql: faster range-query of label_replace and label_join
These functions act on the labels only, so don't need to go step by step
over the samples in a range query.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-04 11:12:55 +01:00
Alan Protasio c006c57efc
Proposal to improve FPointSlice and HPointSlice allocation. (#13448)
* Reusing points slice from previous series when the slice is under utilized
* Adding comments on the bench test

Signed-off-by: Alan Protasio <alanprot@gmail.com>
2024-02-01 16:22:38 +00:00
Bryan Boreham 74b73d1e2c
Labels: Add DropMetricName function, used in PromQL (#13446)
This function is called very frequently when executing PromQL functions,
and we can do it much more efficiently inside Labels.

In the common case that `__name__` comes first in the labels, we simply
re-point to start at the next label, which is nearly free.

`DropMetricName` is now so cheap I removed the cache - benchmarks show
everything still goes faster.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:48:49 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
Ben Ye d778591fd3 add more context cancellation check at evaluation time
Signed-off-by: Ben Ye <benye@amazon.com>
2024-01-21 14:19:39 -08:00
Björn Rabenstein bfbb13cf36
Merge pull request #13267 from linasm/simplify-native-histogram-math
promql: simplify Native Histogram arithmetics
2024-01-18 13:50:59 +01:00
zenador 72a8f1084b
Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022)
Restore more efficient version of NewPossibleNonCounterInfo annotation

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-01-16 09:54:16 +01:00
Filip Petkovski 0e1ae1d1ca
Add comment
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-25 11:41:07 +01:00
Filip Petkovski 5df3820c7a
Copy last histogram point
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-25 11:20:51 +01:00
Filip Petkovski 1f69dcfa6b
Fix reusing float histograms
In https://github.com/prometheus/prometheus/pull/13276 we started reusing float histogram objects to reduce allocations in PromQL.
That PR introduces a bug where histogram pointers gets copied to the beginning of the histograms slice,
but are still kept in the end of the slice. When a new histogram is read into the last element,
it can overwrite a previous element because the pointer is the same.

This commit fixes the issue by moving outdated points to the end of the slice
so that we don't end up with duplicate pointers in the same buffer. In other words,
the slice gets rotated so that old objects can get reused.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-14 11:53:58 +01:00
Filip Petkovski bb8363dbb3
Add comment on SampleRingIterator
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-13 08:30:02 +01:00