Commit graph

344 commits

Author SHA1 Message Date
Callum Styan 9bca041285 WIP: keep track of samples per query, set a max # of samples (#4513)
* keep track of samples per query, set a max # of samples that can be in
memory at once

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-10-02 12:59:19 +01:00
Thomas Jackson abf6fe0a98 Change max/min over_time to handle NaNs properly (#4386)
We only want to return a NaN if the NaN is the only value

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4385
2018-09-26 08:58:16 +01:00
Tom Wilkie 4c52400708
Limit concurrent remote reads. (#4656)
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-09-25 20:07:34 +01:00
Harsh Agarwal 18a9a390b5 Add duplicate-labelset check for range/instant vectors (#4589)
Signed-off-by: Harsh Agarwal <cs15btech11019@iith.ac.in>
2018-09-18 10:46:13 +01:00
Ganesh Vernekar 576ee4d309 Label name check for 'count_values' (#4585)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-13 15:27:36 +05:30
Ganesh Vernekar 73db8b8cea [bugfix] Parse negative value in PromQL (#4564)
* Parse negative value in PromQL
* Enforce space between values

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-13 09:08:01 +01:00
Dan Cech 9f4cb06a37 use Welford/Knuth method to compute standard deviation and variance (#4533)
* use Welford/Knuth method to compute standard deviation and variance, avoids float precision issues
* use better method for calculating avg and avg_over_time

Signed-off-by: Dan Cech <dcech@grafana.com>
2018-08-26 10:28:47 +01:00
Julius Volz 8fbe1b5133
Handle a bunch of unchecked errors (#4461)
There are many more (mostly finalizers like Close/Stop/etc.), but most of
the others seemed like one couldn't do much about them anyway.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-08-17 17:24:35 +02:00
Goutham Veeramachaneni 71855a22a4
Add tracing spans to promql (#4436)
* Add spans to promql

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Simplify timer and span tracking.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-08-16 13:11:34 +05:30
Benjamin Raskin 9353696d77 Fix spelling and holt-winters check (#4424)
Signed-off-by: Benjamin Raskin <braskin@uber.com>
2018-07-27 18:17:43 +01:00
Thomas Jackson 56daa1f28a Only add LookbackDelta to vector selectors (#4399)
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Related to #4226
2018-07-19 06:16:05 +01:00
Alin Sinpalean 372e7652b7 Reuse (copy) overlapping matrix samples between range evaluation steps (#4315)
* Reuse (copy) overlapping matrix samples between range evaluation steps.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-07-18 11:14:02 +01:00
Tony Lee bcdaf8e2d2 add unused pointslices to the pool (#4363)
Signed-off-by: Tony Lee <tl@hudson-trading.com>
2018-07-18 05:29:21 +01:00
Alin Sinpalean e3b775b78b Simplify BufferedSeriesIterator usage (#4294)
* Allow for BufferedSeriesIterator instances to be created without an underlying iterator, to simplify their usage.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-07-18 05:10:28 +01:00
Julius Volz 219e477272 Fix some (valid) lint errors (#4287)
Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-07-18 05:07:33 +01:00
Thomas Jackson 92c6f0c92e Add offset to selectParams (#4226)
* Add Start/End to SelectParams
* Make remote read use the new selectParams for start/end

This commit will continue sending the start/end time of the remote read
query as the overarching promql time and the specific range of data that
the query is intersted in receiving a response to is now part of the
ReadHints (upstream discussion in #4226).

* Remove unused vendored code

The genproto.sh script was updated, but the code wasn't regenerated.
This simply removes the vendored deps that are no longer part of the
codegen output.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2018-07-18 04:58:00 +01:00
Alin Sinpalean 96fb0b2155 Optimize PromQL aggregations (#4248)
* Compute hash of label subsets without creating a LabelSet first.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-07-18 04:56:27 +01:00
Tom Wilkie 3228814456 Don't forget to register query_duration_seconds{slice="queue_time"} (#4381)
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-07-15 12:24:37 +01:00
Thomas Jackson a6dace8829 Check for timeout in each iteration of matrixSelector (#4300)
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4288
2018-06-21 22:43:31 +01:00
Thomas Jackson 630f42fcf1 Timeout if populating iterators takes too long (#4291)
Right now promql won't time out a request if populating the iterators
takes a long time.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4289
2018-06-21 08:14:51 +01:00
Alin Sinpalean 91ce63a140 Log the line when failing a PromQL test. (#4272)
Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-06-14 15:18:16 +01:00
Thomas Jackson 404abe0f1c Bubble up errors to promql from populating iterators (#4136)
This changes the Walk/Inspect API inside the promql package to bubble
up errors. This is done by having the inspector return an error (instead
of a bool) and then bubbling that up in the Walk. This way if any error
is encountered in the Walk() the walk will stop and return the error.
This avoids issues where errors from the Querier where being ignored
(causing incorrect promql evaluation).

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4136
2018-06-07 17:27:34 +01:00
Mario Trangoni 0e2aa35771 promql: fix unconvert issues (#4040)
See,
$ gometalinter --vendor --disable-all --enable=unconvert --deadline 6m ./...
promql/engine.go:1396:26⚠️ unnecessary conversion (unconvert)
promql/engine.go:1396:40⚠️ unnecessary conversion (unconvert)
promql/engine.go:1398:26⚠️ unnecessary conversion (unconvert)
promql/engine.go:1398:40⚠️ unnecessary conversion (unconvert)
promql/engine.go:1427:26⚠️ unnecessary conversion (unconvert)
promql/engine.go:1427:40⚠️ unnecessary conversion (unconvert)
promql/engine.go:1429:26⚠️ unnecessary conversion (unconvert)
promql/engine.go:1429:40⚠️ unnecessary conversion (unconvert)
promql/engine.go:1505:50⚠️ unnecessary conversion (unconvert)
promql/engine.go:1573:46⚠️ unnecessary conversion (unconvert)
promql/engine.go:1578:46⚠️ unnecessary conversion (unconvert)
promql/engine.go:1591:80⚠️ unnecessary conversion (unconvert)
promql/engine.go:1602:94⚠️ unnecessary conversion (unconvert)
promql/engine.go:1630:18⚠️ unnecessary conversion (unconvert)
promql/engine.go:1631:24⚠️ unnecessary conversion (unconvert)
promql/engine.go:1634:18⚠️ unnecessary conversion (unconvert)
promql/engine.go:1635:34⚠️ unnecessary conversion (unconvert)
promql/functions.go:302:42⚠️ unnecessary conversion (unconvert)
promql/functions.go:315:42⚠️ unnecessary conversion (unconvert)
promql/functions.go:334:26⚠️ unnecessary conversion (unconvert)
promql/functions.go:395:31⚠️ unnecessary conversion (unconvert)
promql/functions.go:406:31⚠️ unnecessary conversion (unconvert)
promql/functions.go:454:27⚠️ unnecessary conversion (unconvert)
promql/functions.go:701:46⚠️ unnecessary conversion (unconvert)
promql/functions.go:701:78⚠️ unnecessary conversion (unconvert)
promql/functions.go:730:43⚠️ unnecessary conversion (unconvert)
promql/functions.go:1220:23⚠️ unnecessary conversion (unconvert)
promql/functions.go:1249:23⚠️ unnecessary conversion (unconvert)
promql/quantile.go:107:54⚠️ unnecessary conversion (unconvert)
promql/quantile.go:182:16⚠️ unnecessary conversion (unconvert)
promql/quantile.go:182:64⚠️ unnecessary conversion (unconvert)

Signed-off-by: Mario Trangoni <mjtrangoni@gmail.com>
2018-06-06 18:20:38 +01:00
Brian Brazil dd6781add2 Optimise PromQL (#3966)
* Move range logic to 'eval'

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make aggregegate range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* PromQL is statically typed, so don't eval to find the type.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Extend rangewrapper to multiple exprs

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Start making function evaluation ranged

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make instant queries a special case of range queries

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Eliminate evalString

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Evaluate range vector functions one series at a time

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make unary operators range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make binops range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Pass time to range-aware functions.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make simple _over_time functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reduce allocs when working with matrix selectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add basic benchmark for range evaluation

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse objects for function arguments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Do dropmetricname and allocating output vector only once.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add range-aware support for range vector functions with params

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise holt_winters, cut cpu and allocs by ~25%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make rate&friends range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make more functions range aware. Document calling convention.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make date functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make simple math functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Convert more functions to be range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make more functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Specialcase timestamp() with vector selector arg for range awareness

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove transition code for functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove the rest of the engine transition code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove more obselete code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove the last uses of the eval* functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove engine finalizers to prevent corruption

The finalizers set by matrixSelector were being called
just before the value they were retruning to the pool
was then being provided to the caller. Thus a concurrent query
could corrupt the data that the user has just been returned.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add new benchmark suite for range functinos

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Migrate existing benchmarks to new system

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Expand promql benchmarks

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Simply test by removing unused range code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* When testing instant queries, check range queries too.

To protect against subsequent steps in a range query being
affected by the previous steps, add a test that evaluates
an instant query that we know works again as a range query
with the tiimestamp we care about not being the first step.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse ring for matrix iters. Put query results back in pool.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse buffer when iterating over matrix selectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Unary minus should remove metric name

Cut down benchmarks for faster runs.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reduce repetition in benchmark test cases

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Work series by series when doing normal vectorSelectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise benchmark setup, cuts time by 60%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Have rangeWrapper use an evalNodeHelper to cache across steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Use evalNodeHelper with functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Cache dropMetricName within a node evaluation.

This saves both the calculations and allocs done by dropMetricName
across steps.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse input vectors in rangewrapper

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse the point slices in the matrixes input/output by rangeWrapper

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make benchmark setup faster using AddFast

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Simplify benchmark code.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add caching in VectorBinop

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Use xor to have one-level resultMetric hash key

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add more benchmarks

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Call Query.Close in apiv1

This allows point slices allocated for the response data
to be reused by later queries, saving allocations.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise histogram_quantile

It's now 5-10% faster with 97% less garbage generated for 1k steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make the input collection in rangeVector linear rather than quadratic

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise label_replace, for 1k steps 15x fewer allocs and 3x faster

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise label_join, 1.8x faster and 11x less memory for 1k steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Expand benchmarks, cleanup comments, simplify numSteps logic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address Fabian's comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Comments from Alin.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address jrv's comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove dead code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address Simon's comments.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Rename populateIterators, pre-init some sizes

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Handle case where function has non-matrix args first

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Split rangeWrapper out to rangeEval function, improve comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Cleanup and make things more consistent

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make EvalNodeHelper public

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Fabian's comments.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-06-04 15:47:45 +02:00
Henri DF 986674a790 Make some lexing errors more informative (#4167)
Signed-off-by: Henri DF <henridf@gmail.com>
2018-05-16 16:18:15 +01:00
Elif T. Kuş 57dcdfb15f Rewrote tests with testutil for several test files (#4086)
* promql: Rewrote tests with testutil for functions_test

Signed-off-by: Elif T. Kuş <elifkus@gmail.com>

* pkg/relabel: Rewrote tests with testutil for relabel_test

Signed-off-by: Elif T. Kuş <elifkus@gmail.com>

* discovery/consul: Rewrote tests with testutil for consul_test

Signed-off-by: Elif T. Kuş <elifkus@gmail.com>

* scrape: Rewrote tests with testutil for manager_test

Signed-off-by: Elif T. Kuş <elifkus@gmail.com>
2018-04-27 13:11:16 +01:00
Karsten Weiss d79d573f71 Fix spelling mistakes found by codespell (#4065)
Signed-off-by: Karsten Weiss <knweiss@gmail.com>
2018-04-27 13:04:02 +01:00
David King 6286c10df0 Fix OOM when a large K is used in topk queries (#4087)
This attempts to close #3973.

Handles cases where the length of the input vector to an aggregate topk
/ bottomk function is less than the K paramater. The change updates
Prometheus to allocate a result vector the same length as the input
vector in these cases.

Previously Prometheus would out-of-memory panic for large K values. This
change makes that unlikely unless the size of the input vector is
equally large.

Signed-off-by: David King <dave@davbo.org>
2018-04-16 09:03:04 +01:00
Tony Lee 7cd56f56df add queue_time slice to query_duration_seconds (#4050) 2018-04-05 19:56:58 +01:00
Warren Fernandes d49a3df55b Parser test cleanup (#3977)
* parser test cleanup

- Test against the exported package functions instead of the private functions.

* Improves readability of TestParseSeries

- Moves package function closer to parser function
2018-03-20 14:30:52 +00:00
Anton Tereshchenkov 18bbec050c promql: propagate storage errors 2018-03-14 15:19:22 +01:00
Brian Brazil bf7d87aed2 Cleanup storage from all tests.
Fixed #3299
2018-03-09 07:53:35 +00:00
Brian Brazil c0ce35d2d3 Only show debug output on test failure 2018-03-09 07:53:35 +00:00
Brian Brazil e6ea146c81 Make benchmark tests pass
A new query object is needed for each evaulation,
as the iterators would otherwise be shared across evaluations.
2018-03-09 07:53:35 +00:00
Nikunj Aggarwal 998dfcbac6 Expose itemtype outside the package (#3933) 2018-03-08 16:52:44 +00:00
ferhat elmas ffa673f7d8 General simplifications (#3887)
Another try as in #1516
2018-02-26 07:58:10 +00:00
Fabian Reinartz 309c666426
Merge pull request #3671 from prometheus/queryparams
*: implement query params
2018-02-15 12:24:34 +01:00
Fabian Reinartz 7ccd4b39b8 *: implement query params
This adds a parameter to the storage selection interface which allows
query engine(s) to pass information about the operations surrounding a
data selection.
This can for example be used by remote storage backends to infer the
correct downsampling aggregates that need to be provided.
2018-02-13 12:17:22 +01:00
Krasi Georgiev a53d4ed197 drop metric name for bool modifier (#3821)
fixes #3820
2018-02-11 16:15:55 +00:00
Krasi Georgiev 4801573b64 time() return milliseconds (#3811) 2018-02-08 11:39:13 +00:00
Julius Volz 953af2c089
promql: Make printer formatting less vintage (#3721)
- lower-case modifiers
- reverse order of aggregation modifiers and aggregated expression
- remove spacing before modifier parentheses
2018-01-22 11:14:59 +01:00
Julius Volz 1e943fc10a
promql: Fix printing of empty without() (#3719)
* promql: Fix printing of empty without()

Fixes https://github.com/prometheus/prometheus/issues/3704

* Test cleanup fixup
2018-01-21 22:22:55 +01:00
Brian Brazil b418063d1a
Add tests for negative selectors. (#3616)
https://github.com/prometheus/prometheus/issues/3575
2017-12-23 14:06:37 +00:00
Fabian Reinartz f8fccc73d8 promql: remove global metrics 2017-11-24 07:57:54 +01:00
Fabian Reinartz 83cd270ea4 *: adapt to storage interface changes 2017-11-23 19:05:04 +01:00
David Kaltschmidt 87c46ea6c3 Renamed TotalEvalTime to EvalTotalTime
* TotalFoo suggested a comprehensive timing, but TotalEvalTime was part
of the Exec timings, together with Queue timings
* The other option was to rename ExecTotalTime to TotalExecTime, but
 there was already ExecQueueTime, suggesting Exec to be some sort of
group
2017-11-17 17:46:51 +01:00
David Kaltschmidt c93e54d240 Adds execution timer stats to the range query
API consumers should be able to get insight into the query run times.
The UI currently measures total roundtrip times. This PR allows for more
fine grained metrics to be exposed.

* adds new timer for total execution time (queue + eval)

* expose new timer, queue timer, and eval timer in stats field of the
 range query response:
```json
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [],
    "stats": {
      "execQueueTimeNs": 4683,
      "execTotalTimeNs": 2086587,
      "totalEvalTimeNs": 2077851
    }
  }
}
```

* stats field is optional, only set when query parameter `stats` is not
empty

Try it via
```sh
curl 'http://localhost:9090/api/v1/query_range?query=up&start=1486480279&end=1486483879&step=14000&stats=true'
```

Review feedback

* moved query stats json generation to query_stats.go
* use seconds for all query timers
* expose all timers available
* Changed ExecTotalTime string representation from Exec queue total time to Exec total time
2017-11-16 16:05:10 +01:00
Julius Volz 099df0c5f0 Migrate "golang.org/x/net/context" -> "context" (#3333)
In some places, where ctxhttp or gRPC are concerned, we still need to use the
old contexts.
2017-10-24 21:21:42 -07:00
Brian Brazil 7158675aa8 Add back continue.
Accidentally removed in
15a931dbdb
2017-10-09 19:44:03 +01:00
Brian Brazil 99905f82a6 Remove keep_common modifier.
See #3060
2017-10-05 13:27:48 +01:00