Commit graph

729 commits

Author SHA1 Message Date
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
Julien Pivotto e36e5fa833
Merge pull request #8731 from yeya24/update
Improve grouping label match logic
2021-04-27 00:55:22 +02:00
Fiona Liao 9b83d8330a
Fix memSafeIterator.Seek() (#8748)
* Add range query test cases

This includes a couple of failing ones that double count some points due
to the iterator seek bug.

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Add Seek() implementation for memSafeIterator

Previously, calling memSafeIterator.Seek() would call the Seek() method
on its embedded iterator. This was causing the embedded iterator and the
memSafeIterator to get out of sync because when the embedded Seek()
moved to the next element of the embedded iterator, memSafeIterator
didn't "know" about it. memSafeIterator has to "know" when the embedded
iterator has moved to be able to work out when it should be reading from
its buffer rather than the embedded iterator.

Used same logic as for xorIterator.Seek() (which in runtime is used as
the embedded iterator) - return false if the iterator has an error and
try to move to next element if the required time hasn't been reached, or
if no elements have been read yet. The memSafeIterator.Next() method is
being called so memSafeIterator.i is always accurate.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Add tsdb package test

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
2021-04-27 00:43:22 +02:00
yeya24 d698e062dc improve grouping label match logic
Signed-off-by: yeya24 <yb532204897@gmail.com>
2021-04-16 22:04:58 -04:00
Marco Pracucci 6719071a0f
Optimize aggregations in PromQL engine (#8594)
* Optimize aggregations in PromQL engine

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-19 17:52:29 +01:00
Callum Styan 289ba11b79
Add circular in-memory exemplars storage (#6635)
* Add circular in-memory exemplars storage

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>

* Fix some comments, clean up exemplar metrics struct and exemplar tests.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix exemplar query api null vs empty array issue.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-03-16 15:17:45 +05:30
Marco Pracucci 7bbab380b6
Added tracing span to evaluator.eval()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-15 15:05:47 +01:00
Marco Pracucci b92c03023d
Optimized vector selector
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-11 14:32:56 +01:00
Julien Pivotto ad5ed416ba
Merge pull request #8487 from pschou/dev_neg_offset
allow negative offset
2021-03-08 22:18:45 +01:00
Julien Pivotto 5742a18590 Fix subqueries with default resolution in promql unit tests
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-07 09:20:04 +01:00
schou 3bafe17870 add negative offset to printer
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 21:16:28 -05:00
pschou f80b52be69
Merge branch 'main' into dev_neg_offset 2021-02-23 20:52:57 -05:00
schou efbf3af05d reduce addOffst to 2 parameters
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:46:52 -05:00
schou 75d932a172 var init for bool
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:26:35 -05:00
schou 22bfc11738 aggregate booleans for ease of reading
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:26:35 -05:00
schou 22cd48868a adding feature flag, promql-negative-offset
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
schou 851acf86ee rebuild generated_parser.y.go
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
schou 0c577c547b add negative offset ability
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
songjiayang 776c2b8f42 Speed delta value without loop to calculate resultValue
Signed-off-by: songjiayang <songjiayang1@gmail.com>
2021-02-23 22:28:04 +08:00
pschou aff3c702ab
promql: Add sgn, clamp and last_over_time functions (#8457)
* Add sgn, clamp and last_over_time functions

Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-20 16:34:52 +01:00
Danny Kopping 42a0e0acad
Prevent lexer from seeking to next rune after lexing escape sequence. (#8517)
* Prevent lexer from seeking to next rune after lexing escape sequence.

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2021-02-19 07:38:05 +01:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489)
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
Mauro Stettler 7715fe3219
Add matchers to LabelValues() call (#8400)
* Accept matchers in querier LabelValues()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* create matcher to only select metrics which have searched label

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test case for merge querier with matchers

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test LabelValues with matchers on head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add test for LabelValues on block

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* formatting fix

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Add comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add missing lock release

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unused parameter

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Benchmarks for LabelValues() methods on block/head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Better comment

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update comment

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* minor refactor make code cleaner

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* better comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix expected errors in test

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Deleting parameter which can only be empty

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary lock

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lookup label value if label name was looked up

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Return error when there is one

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Call .Get() on decoder before checking errors

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lock head.symMtx when necessary

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary delete()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* re-use code instead of duplicating it

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Consistently return error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* move helper func from util.go to querier.go

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Fix test expectation

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* ensure result de-duplication and sorting works

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* return named error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-02-09 23:08:35 +05:30
Ganesh Vernekar 86c71856e8
Add start() and end() pre-processors for @ modifier (#8425)
* Add start() and end() pre-processors for @ modifier

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-09 21:33:16 +05:30
Marcelo E. Magallon 75d86c6747 Update golangci-lint to 1.36.0
In the previous version, 1.18.0, the "megacheck" linter paid attention
to the '//lint:ignore' comment, but that is no longer there.

Newer version pay attention to '//nolint:<linter>,<linter>,...'
comments, optionally followed by a "second" comment introduced by '//'.

Update the directives to use this style.

This is related to prometheus/blackbox_exporter#738 and
prometheus/blackbox_exporter#745.

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
2021-02-04 08:53:33 -06:00
Ganesh Vernekar ed8d31b059
Remove empty test
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-04 11:53:32 +05:30
Ganesh Vernekar b18fde996e
Fix timestamp() function for @ modifier
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-03 19:13:12 +05:30
Ganesh Vernekar b5dfa2414b
Fix flaky TestAtModifier (#8401)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-24 20:23:30 +05:30
Ganesh Vernekar 9199fcb8d1
'@ <timestamp>' modifier (#8121)
This commit adds `@ <timestamp>` modifier as per this design doc: https://docs.google.com/document/d/1uSbD3T2beM-iX4-Hp7V074bzBRiRNlqUdcWP6JTDQSs/edit.

An example query:

```
rate(process_cpu_seconds_total[1m]) 
  and
topk(7, rate(process_cpu_seconds_total[1h] @ 1234))
```

which ranks based on last 1h rate and w.r.t. unix timestamp 1234 but actually plots the 1m rate.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-20 16:27:39 +05:30
Julien Pivotto 34656bbf8e
PromQL parser: fastpath for non-empty matchers and metric name (#8345)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-01-07 11:38:38 +01:00
Ganesh Vernekar 063154eab7
Automate part of TestMaxQuerySamples (#8314)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-04 14:33:36 +05:30
Atibhi Agrawal 0dd6e0cb76
change string(model.metricnamelabel) to model.metricnamelabel (#8289)
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-12-15 20:17:31 +05:30
Brian Brazil bef9d4e182
Don't include rendered expression in parse errors. (#8177)
For a sufficiently complex expression, having to render
thousands of nodes for every subquery type error can take
a while - so don't do that.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-11-12 15:25:52 +01:00
Ganesh Vernekar d30da66d77
Fix timestamp() method for vector selector inside paren (#8164)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-11-09 18:21:50 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122)
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110)
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Harold Dost bf7c771f54
promql: Organize legacy tests into appropriate files (#7989)
* Separate tests into:
** Aggregators
** Functions
** Operators
** Selectors
* Remove simple files and place tests into other files.
* Eliminate some of the _over_time tests since there are already plenty of
  edge cases present in the functions.test file.

Relates to #740

Signed-off-by: Harold Dost <harolddost@gmail.com>
2020-10-07 10:09:20 +01:00
Brian Brazil 924ce978b7
Don't do literal regex matching optimisation when case insensitive. (#8013)
Fixes #7994

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-10-06 13:16:26 +01:00
Harkishen Singh fc8e769d71
Use ASSIGN when using = inside braces (#7911)
* Fix EQL when using = inside braces.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* EQL => EQLC and ASSIGN => EQL

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* Aligned yacc code.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2020-09-09 15:40:02 +05:30
Vijay Samuel 00ee73ef91
Export members of EvalNodeHelper to facilitate usage in external functions (#7860)
Signed-off-by: Vijay Samuel <vjsamuel@ebay.com>
2020-08-27 19:30:10 +01:00
johncming 267dd17fe7
promql/parser: use pos instead of origin. (#7853)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-26 15:26:26 +05:30
Julien Pivotto 6f9e7ff750
Drop metric name in bool comparison between two instant vectors (#7819)
* Drop metric name in bool comparison between two instant vectors

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-22 21:04:03 +02:00
Julien Pivotto 20ab94fedf
Hints: Separating out the range and offsets of PromQL subqueries (#7667)
Fix #7629

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-11 07:21:39 +01:00
Annanay Agarwal 118aeab02c
Make context key type public (#7748)
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-08-05 09:51:36 +01:00
Julien Pivotto d867491364
Human-friendly durations in PromQL (#7713)
* Add support for user-friendly durations

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-04 21:12:41 +02:00
johncming 31929b83d5
promql: use explicit type declare instead of string. (#7716)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-02 09:57:38 +01:00
johncming 1c1b394e5e
promql: Swap order of parseBrokenJSON. (#7718)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-02 09:48:57 +01:00
Bartlomiej Plotka e6d7cc5fa4
tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069)
* tsdb: Added ChunkQueryable implementations to db; unified compactor, querier and fanout block iterating.

Chained to https://github.com/prometheus/prometheus/pull/7059

* NewMerge(Chunk)Querier now takies multiple primaries allowing tsdb DB code to use it.
* Added single SeriesEntry / ChunkEntry for all series implementations.
* Unified all vertical, and non vertical for compact and querying to single
merge series / chunk sets by reusing VerticalSeriesMergeFunc for overlapping algorithm (same logic as before)
* Added block (Base/Chunk/)Querier for block querying. We then use populateAndTomb(Base/Chunk/) to iterate over chunks or samples.
* Refactored endpoint tests and querier tests to include subtests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments from Brian and Beorn.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed snapshot test and added chunk iterator support for DBReadOnly.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed race when iterating over Ats first.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed populate block tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed endpoints test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added test & fixed case of head open chunk.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed DBReadOnly tests and bug producing 1 sample chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added cases for partial block overlap for multiple full chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added extra tests for chunk meta after compaction.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed small vertical merge bug and added more tests for that.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-07-31 16:03:02 +01:00
Annanay 9bba8a6eae Merge branch 'master' into appender-context
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-30 16:43:18 +05:30
Julien Pivotto 22acb87e09
refactoring: make sure that query_duration_seconds metrics are the same (#7668)
* refactoring: make sure that query_duration_seconds are the same

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-25 11:55:59 +02:00
Owen Diehl 00b7bdb1b6
parser.{Expr,Statement} publicly implementable (#7639)
* parser.{Expr,Statement} publicly implementable

Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
2020-07-25 09:05:58 +01:00
Annanay 7f98a744e5 Add context to Appender interface
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-24 19:40:51 +05:30
Guangwen Feng 6b7ac2ac1b
Add unit test case to improve test coverage for matcher.go (#7658)
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-07-24 11:21:42 +01:00
Julien Pivotto 93e9c010f3
Add more Go leak tests (#7652)
* Implement go leak test for promql

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Implement go leak test for Consul SD

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Implement go leak test in discovery manager

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-24 10:10:20 +01:00
Bartlomiej Plotka 841b13641c
promql: Refactored subquery hint tests and added todos. (#7636)
* promql: Refactorer subquery hint tests and added todos.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* fmt.


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixes.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-07-23 23:05:43 +01:00
Bartlomiej Plotka a0df8a383a
promql: Removed global and add ability to have better interval for subqueries if not specified (#7628)
* promql: Removed global and add ability to have better interval for subqueries if not specified

## Changes
* Refactored tests for better hints testing
* Added various TODO in places to enhance.
* Moved DefaultEvalInterval global to opts with func(rangeMillis int64) int64 function instead

Motivation: At Thanos we would love to have better control over the subqueries step/interval.
This is important to choose proper resolution. I think having proper step also does not harm for
Prometheus and remote read users. Especially on stateless querier we do not know evaluation interval
and in fact putting global can be wrong to assume for Prometheus even.

I think ideally we could try to have at least 3 samples within the range, the same
way Prometheus UI and Grafana assumes.

Anyway this interfaces allows to decide on promQL user basis.

Open question: Is taking parent interval a smart move?

Motivation for removing global: I spent 1h fighting with:


=== RUN   TestEvaluations
    TestEvaluations: promql_test.go:31: unexpected error: error evaluating query "absent_over_time(rate(nonexistant[5m])[5m:])" (line 687): unexpected error: runtime error: integer divide by zero
--- FAIL: TestEvaluations (0.32s)
FAIL

At the end I found that this fails on most of the versions including this master if you run this test alone. If run together with many
other tests it passes. This is due to SetDefaultEvaluationInterval(1 * time.Minute)
in test that is ran before TestEvaluations. Thanks to globals (:

Let's fix it by dropping this global.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added issue links for TODOs.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed irrelevant changes.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-07-22 14:39:51 +01:00
Guangwen Feng b30654211c
Fix incorrect arguments order in TestExprString (#7602)
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-07-17 13:38:04 +01:00
Julien Pivotto d77b56e88e
Fix avg_over_time for nan and float64 overflows (#7346)
* Fix avg_over_time with Inf and NaN values

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-13 17:30:50 +02:00
Tobias Guggenmos 1f73073d73
Make without a valid metric identifier (#7533)
Discussion see #7532.

cc @juliusv

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
2020-07-08 12:58:12 +02:00
Julien Pivotto 72425d4e3d
Add group() aggregator (#7480)
* Add group() aggregator

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-06-30 16:51:18 +02:00
Guangwen Feng 9ab072b470
Fix golint issue caused by typo (#7475)
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-06-28 11:03:09 +01:00
Linas Medžiūnas 7eaffa7180
Fix off-by-one error in funcHistogramQuantile / ensureMonotonic (#7393)
* Fix off-by-one error in funcHistogramQuantile / ensureMonotonic
* Additional coverage for nonmonotonic histogram buckets

Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2020-06-15 11:32:10 +01:00
Kemal Akkoyun 66dfb951c4
*: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251)
* Add errors and Warnings to SeriesSet

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Change Querier interface and refactor accordingly

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactor promql/engine to propagate warnings at eval stage

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Make sure all the series from all Selects are pre-advanced

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Separate merge series sets

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Clean

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactor merge querier failure handling

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactored and simplified fanout with improvements from incoming chunk iterator PRs.

* Secondary logic is hidden, instead of weird failed series set logic we had.
* Fanout is well commented
* Fanout closing record all errors
* MergeQuerier improved API (clearer)
* deferredGenericMergeSeriesSet is not needed as we return no samples anyway for failed series sets (next = false).

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix formatting

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix CI issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Added final tests for error handling.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

* Moved hints in populate to be allocated only when needed.
* Used sync.Once in secondary Querier to achieve all-or-nothing partial response logic.
* Select after first Next is done will panic.

NOTE: in lazySeriesSet in theory we could just panic, I think however we can
totally just return error, it will panic in expand anyway.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Utilize errWithWarnings

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix recently introduced expansion issue

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Add tests for secondary querier error handling

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Implement lazy merge

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Add name to test cases

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Reorganize

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review comments

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review comments

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Remove redundant warnings

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix rebase mistake

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-06-09 17:57:31 +01:00
Julien Pivotto 4284dd1f1b
promql: cleanup: use errors.As (#7351)
This was TODO because circleci was not in go1.13 yet.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-06-05 19:51:25 +02:00
B++ d6374ae1b6
Return NaN for histogram_quantile when buckets have 0 observations (#7318)
Signed-off-by: jberny <f.bernardi89@gmail.com>
2020-06-01 09:40:39 +01:00
Julien Pivotto 58c445e6ef
Fuzz: limit input size (#7317)
We know that fuzzParseExpr and fuzzParseMetricSelector make use of heavy
things like regexes, which take a fairly big amount of memory.

OSS-Fuzz does not offer a proper way to increase the memory [1], therefore
we limit the input size [2].

[1] https://google.github.io/oss-fuzz/faq/#how-do-you-handle-timeouts-and-ooms
[2] https://google.github.io/oss-fuzz/getting-started/new-project-guide/#input-size

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-05-31 09:42:56 +02:00
Brian Brazil 3932a7149f
Correctly track points no longer used by matrixIterSlice's slice. (#7307)
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-05-28 13:36:30 +01:00
Callum Styan 5bb7f00d00
change labelset comparison in promql engine to avoid false positive during detection of duplicates (#7058)
* Use go1.14 new hash/maphash to hash both RHS and LHS instead of XOR'ing
which has been resulting in hash collisions.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Refactor engine labelset signature generation, just use labels.Labels
instead of hashes.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments; function comments + store result of
lhs.String+rhs.String as key.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Replace all signatureFunc usage with signatureFuncString.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Make optimizations to labels String function and generation of rhs+lhs
as string in resultMetric.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use separate string functions that don't use strconv just for engine
maps.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use a byte invalid separator instead of quoting and have a buffer
attached to EvalNodeHelper instead of using a global pool in the labels
package.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more review comments, labels has a function that now builds a
byte slice without turning it into a string.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use two different non-ascii hex codes as byte separators between labels
and between sets of labels when building bytes of a Labels struct.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We only need the 2nd byte invalid sep. at the beginning of a
labels.Bytes

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2020-05-12 14:03:15 -07:00
Ben Ye 1e4e37144d
Fixed wrongly handled not ready TSDB on web and API. (#7182)
* fix federate endpoint panic

Signed-off-by: yeya24 <yb532204897@gmail.com>

* Fixed all cases of not ready TSDB being wrongly handled.

* Fixed issue for federation.
* Ensured this will never happen again thanks to interfaces
* Fixes same issue for stats.
* Added tests for readiness.
* Fixed bug in stats. It was:
   status.MaxTime = db.Head().MaxTime()
   status.MinTime = db.Head().MaxTime()


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-04-29 17:16:14 +01:00
Vasily Sliouniaev 0393b188c9
Add Jaeger (#7148)
* Trace remote read

Signed-off-by: vas <vasily.sliouniaev@jet.com>

* Use jaeger

Signed-off-by: vas <vasily.sliouniaev@jet.com>
2020-04-23 02:05:55 +02:00
Julien Pivotto 1f6f8e60ee promql/parser: Cleanup generatedParserResult accross reuse
Reusing the same generatedParserResult ends up in strange panics:
See #7131 and #7127.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-04-16 01:51:08 +02:00
Marek Slabicki 8224ddec23
Capitalizing first letter of all log lines (#7043)
Signed-off-by: Marek Slabicki <thaniri@gmail.com>
2020-04-11 09:22:18 +01:00
Brian Brazil 7646cbca32
Use .UTC everywhere we use time.Unix (#7066)
time.Unix attaches the local timezone, which can then
leak out (e.g. in the alert json). While this is harmless,
we should be consistent.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-03-29 17:35:39 +01:00
Ben Kochie 269e7c8091
Fix golint issues.
Signed-off-by: Ben Kochie <superq@gmail.com>
2020-03-23 20:38:43 +01:00
johncming bdc45c2b9e
remove unused code. (#7019)
Signed-off-by: johncming <johncming@yahoo.com>
2020-03-21 07:15:51 +00:00
Björn Rabenstein 1da83305be
Merge pull request #7009 from prometheus/release-2.17
Merge release-2.17 into master
2020-03-19 13:46:28 +01:00
Tobias Guggenmos 012161d90d
PromQL: Fix lexer error handling (#6958)
* PromQL: Fix lexer error handling

This fixes bugs in the handling of lexer errors that are only noticeable for users of the language server and caused https://github.com/prometheus-community/promql-langserver/issues/104 .

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>

* Add test for error position ranges

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
2020-03-16 15:47:47 +01:00
Björn Rabenstein a28fa010ee
TSDB: Extract parts out of populateSeries (#6983)
This addresses fabxc's TODO.

More importantly, it now properly defers the
querier.Close(). Previously, if a panic happened after creation of the
querier within the populateSeries function, querier.Close() was never called.

The latter was responsible for #6977.

Signed-off-by: beorn7 <beorn@grafana.com>
2020-03-14 09:03:40 +01:00
Bartlomiej Plotka fe802f29c9 storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response.
This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on https://github.com/prometheus/prometheus/pull/5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-03-13 13:06:25 +00:00
Björn Rabenstein bc703b6456
Use struct{} as underlying type for context keys (#6965)
This is an alternative to #6963.

Signed-off-by: beorn7 <beorn@grafana.com>
2020-03-11 15:05:35 +01:00
Julien Pivotto 5ddd1dcf0f
Fix panic when parsing varags (#6940)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-03-08 13:09:24 +01:00
Tobias Guggenmos 1dbd799354
PromQL: Fix regression tests (#6935)
This PR fixes the regression tests for the issue fixed in #6931 .

The reason for that is that all of the invalid queries that triggered the regression have become more or less valid syntax in #6933 (they might still fail typechecking).

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
2020-03-06 08:17:01 +00:00
Brian Brazil 44ad28dd5e
PromQL: Allow more keywords as metric names (#6933)
* Allow more keywords as metric names
* Add documentation about forbidden keywords

Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
2020-03-05 13:20:53 +00:00
Brian Brazil 7164b58945
PromQL: Fix parser panic (#6931)
Signed-off-by: Tobias Guggenmos <tobias.guggenmos@uni-ulm.de>
2020-03-05 08:03:38 +00:00
李国忠 2bf4952049
remove Unused parameter 'sf' in calcTrendValue function (#6900)
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
2020-03-03 08:23:27 +00:00
LongKB 82f7ed208b
Remove some duplicated words (#6882)
Signed-off-by: Pham Duc Hanh <hanhpd@fujitsu.com>
2020-02-27 07:08:31 +01:00
Tobias Guggenmos 3d74fcfa6a Bartek's suggestions
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-25 13:57:30 +01:00
Tobias Guggenmos f9db320e5a Look up function call in all cases
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-24 13:45:03 +01:00
Tobias Guggenmos 9ebf6bd1e6 Remove superfluous blank lines
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-21 13:36:57 +01:00
Tobias Guggenmos 7143d64fc1
Julien's suggestion
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

Co-Authored-By: Julien Pivotto <roidelapluie@gmail.com>
2020-02-21 13:34:15 +01:00
Tobias Guggenmos 4124828c00 Add test to check that promql.FunctionCalls and parser.Functions contain the same functions
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-21 12:43:30 +01:00
Tobias Guggenmos 4835bbf376
Merge branch 'master' into split_parser 2020-02-19 15:18:13 +01:00
Bartlomiej Plotka 849faa407b Minor fixes.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka 2cf637fbf5 Addressed comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka 34426766d8 Unify Iterator interfaces. All point to storage now.
This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:54 +00:00
Tobias Guggenmos 2d77611f73 Unbump year number
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos bf8542a99e Add missing imports to fuzz tests
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos ec3df53f78 Add nolint comment
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos b93845d6bf Remove unused function
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos 43d206f0a1 Add missing file
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos 6c00f2ffcb Comment fixes
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos 273ea9093a Fix promql tests
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos 1360f9ff12 Fix all build errors in promql package
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos 2164e366ea Fix more identifiers
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Tobias Guggenmos 5caf7ed6db Fix more identifiers
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Tobias Guggenmos 9a1366775e Store function implementations independently of their signatures
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Tobias Guggenmos ff0ea1c1ac Fix more identifiers
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Tobias Guggenmos 6b1b323558 Export sequenceValue
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Tobias Guggenmos 228967a507 Fix usages of more things that have moved the package
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:27 +01:00
Tobias Guggenmos 4a4817a444 Fix usages of parser.Statement
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Tobias Guggenmos 2f1113479f Fix usages of ValueType
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Tobias Guggenmos 65f5b9827d Rename package
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Tobias Guggenmos 98a9b7c90c Remove engine specific types
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Tobias Guggenmos fab2373752 Add everything the parser needs to build
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Tobias Guggenmos e7c2716f55 Move files
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:05:21 +01:00
Zhou Hao e628fd7735
fix comments spelling (#6829)
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
2020-02-17 12:45:11 +01:00
Björn Rabenstein af04cb22c8
Merge pull request #6821 from prometheus/release-2.16
Release 2.16
2020-02-14 13:10:14 +01:00
Julien Pivotto ff0003e072
Make lookbackDelta a option of QueryEngine (#6746)
* Make lookbackDelta a option of QueryEngine

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* julius' suggestion

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* remove trivial getter

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Assume lookback delta is always > 0

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* add debug log

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* don't expose loopback delta

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Specify that lookack delta is also used in federation

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Fix federation test

While we have added some logic to the promql engine to keep it backwards
compatible and have a 5 minute loopback by default, the web/ package is
likely to really be internal to Prometheus and we should not add the
same kind of heuritstics here.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* loopback delta: Fix debug log

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-02-10 00:58:23 +01:00
Julien Pivotto cbd0eec9fc Avoid /1000
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-02-08 16:30:09 +01:00
Julien Pivotto 881dde505a promql: fix promql query log step unit
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-02-08 16:26:56 +01:00
Julien Pivotto 9adad8ad30 Remove MaxConcurrent from the PromQL engine opts (#6712)
Since we use ActiveQueryTracker to check for concurrency in
d992c36b3a it does not make sense to keep
the MaxConcurrent value as an option of the PromQL engine.

This pull request removes it from the PromQL engine options, sets the
max concurrent metric to -1 if there is no active query tracker, and use
the value of the active query tracker otherwise.

It removes dead code and also will inform people who import the promql
package that we made that change, as it breaks the EngineOpts struct.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-28 20:38:49 +00:00
Brian Brazil 61262159c4 Simplify benchmark given the new API
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 14:38:09 +00:00
Brian Brazil 38d32e0686 Don't sort postings if we only have one block.
Sorting the heads postings can be quite slow.
We only need sorted series when merging with another
querier, so only sort then.
This will make big queries that only touch the head faster,
though queries that touch both the head and a block will still
be the same speed. This probably won't help much with graphing
unless the range is under an hour, however it should make most
recording rules faster.

Add gaurantee that remote read streaming produces sorted series.

PromQL benchmarks for histograms show only 2-3% improvement, but
they're only over 1k series.

benchmark                                                old ns/op      new ns/op      delta
BenchmarkQuerierSelect/Head/1of1000000-4                 1375486282     507657736      -63.09%
BenchmarkQuerierSelect/Head/10of1000000-4                1387859004     507769850      -63.41%
BenchmarkQuerierSelect/Head/100of1000000-4               1387087935     506029110      -63.52%
BenchmarkQuerierSelect/Head/1000of1000000-4              1386869064     504521986      -63.62%
BenchmarkQuerierSelect/Head/10000of1000000-4             1386213685     505210422      -63.55%
BenchmarkQuerierSelect/Head/100000of1000000-4            1392754988     529842406      -61.96%
BenchmarkQuerierSelect/Head/1000000of1000000-4           1569414722     725059506      -53.80%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           1381019902     1370495863     -0.76%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          1375696209     1366789468     -0.65%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         1386009422     1364519297     -1.55%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        1377700532     1364486191     -0.96%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       1383539536     1369545314     -1.01%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      1410089163     1394731339     -1.09%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     1634744148     1581554956     -3.25%
BenchmarkQuerierSelect/Block/1of1000000-4                881741242      879839470      -0.22%
BenchmarkQuerierSelect/Block/10of1000000-4               880381562      882846038      +0.28%
BenchmarkQuerierSelect/Block/100of1000000-4              887519357      881016916      -0.73%
BenchmarkQuerierSelect/Block/1000of1000000-4             902194205      883433524      -2.08%
BenchmarkQuerierSelect/Block/10000of1000000-4            892321964      885130170      -0.81%
BenchmarkQuerierSelect/Block/100000of1000000-4           938604466      933527150      -0.54%
BenchmarkQuerierSelect/Block/1000000of1000000-4          1313510845     1295881124     -1.34%

benchmark                                                old allocs     new allocs     delta
BenchmarkQuerierSelect/Head/1of1000000-4                 4000056        4000018        -0.00%
BenchmarkQuerierSelect/Head/10of1000000-4                4000074        4000036        -0.00%
BenchmarkQuerierSelect/Head/100of1000000-4               4000254        4000216        -0.00%
BenchmarkQuerierSelect/Head/1000of1000000-4              4002054        4002016        -0.00%
BenchmarkQuerierSelect/Head/10000of1000000-4             4020054        4020016        -0.00%
BenchmarkQuerierSelect/Head/100000of1000000-4            4200054        4200016        -0.00%
BenchmarkQuerierSelect/Head/1000000of1000000-4           6000054        6000016        -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           4000071        4000071        +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          4000089        4000089        +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         4000269        4000269        +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        4002069        4002069        +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       4020069        4020069        +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      4200069        4200069        +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     6000069        6000069        +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4                6000023        6000022        -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4               6000059        6000058        -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4              6000419        6000418        -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4             6004019        6004018        -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4            6040019        6040018        -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4           6400019        6400018        -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4          10000020       10000019       -0.00%

benchmark                                                old bytes     new bytes     delta
BenchmarkQuerierSelect/Head/1of1000000-4                 229192200     176001176     -23.21%
BenchmarkQuerierSelect/Head/10of1000000-4                229193352     176002328     -23.21%
BenchmarkQuerierSelect/Head/100of1000000-4               229204872     176013848     -23.21%
BenchmarkQuerierSelect/Head/1000of1000000-4              229320072     176129048     -23.20%
BenchmarkQuerierSelect/Head/10000of1000000-4             230472072     177281048     -23.08%
BenchmarkQuerierSelect/Head/100000of1000000-4            241992072     188801048     -21.98%
BenchmarkQuerierSelect/Head/1000000of1000000-4           357192072     304001048     -14.89%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           229193928     229193928     +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          229195080     229195080     +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         229206600     229206600     +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        229321800     229321800     +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       230473800     230473800     +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      241993800     241993800     +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     357193800     357193800     +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4                227201516     227201500     -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4               227202924     227202908     -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4              227217036     227217020     -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4             227358156     227358140     -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4            228769356     228769340     -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4           242881356     242881340     -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4          384001616     384001600     -0.00%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
Julien Pivotto d992c36b3a promql: make active query tracker context-aware (#6701)
* promql: make query logger context-aware
* Remove gate
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-27 22:29:44 +00:00
Julien Pivotto cf42888e4d Fix order of testutil.Equals (#6695)
Equals takes the expected value as first parameter, and the actual value
as second parameter.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-27 12:21:59 +00:00
Julien Pivotto 5f27ac3583 Refactor query log fields (#6694)
* Refactor query log fields

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-27 09:53:10 +00:00
Julien Pivotto 3fbeee07fa Fix flaky windows test (#6677)
The windows clock is sometime off by 25ms, and as precise as 15ms.

Let's give it more time to avoid flaky tests.

Fix #6672

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-22 10:09:31 +00:00
Julien Pivotto 2b2eb79e8b Add windows tests for query logger (#6653)
* Add windows tests
* Do not rely on time.Time in timer

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-20 13:17:11 +00:00
Tobias Guggenmos 2aacd807b3 PromQL: Various small improvements in the parser (#6652)
* Move check for empty VectorSelector to typeChecking
* Move check for twice set metric name to typeChecking
* Make child of MatrixSelector a general Node
* rename checkType to checkAST
* Rename fail to addParseErr
* Remove trailing whitespace

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-17 15:16:58 +00:00
Tobias Guggenmos f5eed7ae0a PromQL: Fix panic in parser (#6650)
Fixes #6649.

The crash is fixed here, was caused because some  AST sanity checks were performed on the syntax tree while assembling it. In case of previous parsing errors this could lead to undefined behaviour.

The fix is to move the checks to the typechecking phase, which runs only when a syntax tree was assembled without there being parsing errors.

There are other places, where similiar checks are performed while assembling the syntax tree. It might be a good idea to move those to the typechecking phase, too. Should I do this in the same or a separate PR?

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-17 13:06:26 +00:00
Tobias Guggenmos 2ca25d1c96 PromQL: Don't panic on syntax errors (#6643)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-16 15:20:20 +00:00
Tobias Guggenmos d642570924 PromQL: Use PositionRange in parser errors (#6634)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-15 20:01:49 +00:00
Tobias Guggenmos 3a204be6b7 PromQL: Fix string and parentheses handling in engine (#6612)
* WIP: PromQL: Allow engine to return strings

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

* Add test suggested by @roidelapluie

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

* Fix typo in React UI

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

* Fix parenthesis handling for functions and aggregator params

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

* Add more tests

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>

* Fix React UI test

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-15 18:31:58 +01:00
Tobias Guggenmos 0c8e9ef09e PromQL: Add position metadata to the AST (#6615)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
Co-authored-by: Julius Volz <julius.volz@gmail.com>
2020-01-14 16:12:15 +00:00
Tobias Guggenmos 64194f7d45 PromQL: AST: Make VectorSelector Children of MatrixSelector (#6590)
Make Vector selectors children of Matrix Selectors

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-10 14:25:41 +00:00
Julien Pivotto 3885562587 Query Logging styling (#6594)
- Fix Json vs JSON in activequerylogger
- Fix SetQueryLogger always returns nil

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-09 21:11:39 +00:00
Tobias Guggenmos 6534ce843f PromQL: Use a sync.Pool for the generated parser structure (#6591)
* PromQL: Use a sync.Pool for the generated parser structure

The generated PromQL parser allocates a struct about 4kb in size on every run.

This puts a high load on the garbage collector.

To reduce that load, a sync.Pool is used to recycle these structures.

On small queries this makes parsing 2-3 times faster.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-09 17:36:13 +00:00
Guangwen Feng 835c27a76b Fix typo in comment for func Lex (#6588)
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-01-09 11:27:26 +00:00
Tobias Guggenmos b18b6cb332 PromQL: Avoid lexer item copies and allocations (#6584)
* PromQL: Avoid lexer item copies and allocations

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-09 11:26:58 +00:00
Tobias Guggenmos 4aef43a9f5 Cleanup: Remove parser switching logic (#6583)
During the PromQL parser rewrite there was some logic put in place that allowed switching between the non generated and the generated parser. Since the parser is now fully generated this is not needed anymore.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:59:25 +00:00
Tobias Guggenmos 2064abab40 Cleanup: Remove unused variable (#6581)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:58:52 +00:00
Tobias Guggenmos 7c7746257c Cleanup: remove function that does not do anything (#6580)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:58:21 +00:00
Tobias Guggenmos d8bc24349c PromQL: Fix unary operator precedence (#6579)
* PromQL: Fix unary operator precedence

Fixes #6575

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:07:10 +00:00
Julien Pivotto 9d9bc524e5 Add query log (#6520)
* Add query log, make stats logged in JSON like in the API

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-08 13:28:43 +00:00
Tobias Guggenmos 3d6cf1c289 PromQL: Make parser completely generated (#6548)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 11:04:47 +00:00
Julien Pivotto 577e738986 Cleanup PromQL functions (#6551)
* Cleanup PromQL functions

The engine ensures, for Matrix functions, that functions are called with exactly one series at the time.
Therefore a lot of code can be inlined and we can directly assume the first element of the arguments exists and contains all the samples needed.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-06 10:33:36 +00:00
Julien Pivotto e0afec906f add absent_over_time (#6490)
* Implement absent_over_time

Fixes #2882

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-03 15:26:12 +00:00
Josh Soref 91d76c8023 Spelling (#6517)
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Tobias Guggenmos 213a8fe89a PromQL: Parse Series descriptions using the generated parser (#6494)
* Use generated parser for series descriptions

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-21 08:44:12 +00:00
Julien Pivotto f363a71ef7 promql: function: fix typo (#6501)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 22:30:55 +00:00
Julien Pivotto 4b3bab890d Fix absent output where two label matchers are identical (#6493)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 14:33:33 +00:00
Tobias Guggenmos a48acbcb23 PromQL: Fix parser benchmark (#6495)
The parser benchmarks called the `ParseMetric` function instead of the `ParseExpr` function, which resulted in parsing failing every time.

This means only the case of PromQL parser failure was benchmarked.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:30:41 +00:00
Tobias Guggenmos 1e0cd28bf6 PromQL: Support trailing commas in grouping opts (#6480)
* PromQL: Support trailing commas in grouping opts

Fixes #6470

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:28:56 +00:00
Julien Pivotto f315c69129 promql: fix typo in value.go (#6491)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-19 22:30:32 +00:00
Tobias Guggenmos db1258f2a5 PromQL: Refactor error message generation (#6481)
* Add parser method to produce errors messages about unexpected items
* PromQL: use parser.unexpected in generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 17:36:43 +00:00
Tobias Guggenmos 9e34f08ac3 PromQL: Parse grouping opts with the generated parser (#6472)
* PromQL: Parse grouping opts with the generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 14:18:52 +00:00
Tobias Guggenmos 53615412b4 PromQL: Parse Metrics using generated parser (#6466)
* Parse Metrics with the generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 16:38:18 +00:00
Tobias Guggenmos b38b25e9e1 Add Benchmark for the PromQL Parser (#6355)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 14:04:56 +00:00
Tobias Guggenmos 8cb4a48e2e PromQL: Parse label sets using the generated parser (#6432)
* Add grammar for label_sets
* Parse label Sets using the generated parser
* Allow trailing commas for label sets and selectors
* Add test to trigger all possible error messages for label matchers

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 13:58:46 +00:00
Tobias Guggenmos 5c503d85f7 PromQL: export lexer (#6435)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 19:03:31 +00:00
Tobias Guggenmos 35c1f31721 PromQL: Use more standart format for error positions (#6433)
The most common format (used by go, gcc and clang) for compiler error positions seems to be

`filename:line:char:` or `line:char:` if the filename is unknown.

This PR adapts the PromQL parser to use this convention.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 16:39:03 +00:00
Tobias Guggenmos cd39ebe7c6 PromQL: export function list (#6434)
This PR exports the list of supported PromQL functions and their signatures.

The reason for that is that the PromQL language server likes to use that list.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 15:33:43 +00:00
Mark Nevill b0a5c51b95 Return unused point slice to pool in Call and VectorSelector eval. (#6427)
Signed-off-by: Mark Nevill <mark.nevill@gmail.com>
2019-12-09 10:32:40 +00:00
Tobias Guggenmos 3bb715031f PromQL: Use generated parser to parse label matchers (#6410)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-05 16:16:12 +00:00
Garrett 5a9c4acfbf Pushdown aggregator group by through read hint (#6401)
* Pushdown aggregator group by through read hint

Implement https://github.com/prometheus/prometheus/issues/6400

* add temporal aggregation pushdown support

Signed-off-by: xiancli <xiancli@ebay.com>
2019-12-05 14:06:28 +00:00
Tobias Guggenmos 408574a6e1 promql: Allow injecting fake tokens into the generated parser (#6381)
* promql: Allow injecting fake tokens into the generated parser

Yacc grammars do not support having multiple start symbols.

To work around that restriction, it is possible to inject fake tokens into the lexer stream,
as described here https://www.gnu.org/software/bison/manual/html_node/Multiple-start_002dsymbols.html .

This is part of the parser rewrite effort described in #6256.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-27 12:59:03 +00:00
Tobias Guggenmos bbd92b85da promql: Use capitalized names for item types (#6371)
For yacc generated parsers there is the convention to capitalize the names of item types provided by the lexer, which makes it easy to distinct lexer tokens (capitalized) from nonterminal symbols (not capitalized) in language grammars.

This convention is also followed by the (non generated) go compiler (see https://golang.org/pkg/go/token/#Token).

Part of the parser rewrite described in #6256.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-26 13:29:42 +00:00
Tobias Guggenmos c229ed17e2 promql: Implement yyLexer interface (#6370)
This is the first step towards a generated lexer as described in #6256.

It adds methods to the parser struct, that make it implement the yyLexer interface required by a yacc generated parser, as described here: https://godoc.org/golang.org/x/tools/cmd/goyacc .

The yyLexer interface is implemented by the parser struct instead of the lexer struct for the following reasons:

* Both parsers have a lookahead that the lexer does not know about. This solution makes it possible to synchronize these lookaheads when switching parsers.
* The routines to handle parser errors are not accessible to the lexer.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-26 13:28:36 +00:00
Bartosz Jakubski 80ba03c67d fix subquery with offset conversion to string (#6373)
Signed-off-by: Bartosz Jakubski <b.jakubski@gmail.com>
2019-11-26 06:45:51 +00:00
Tobias Guggenmos c63259b83c promql: Clean up parser struct (#6360)
* promql: Clean up parser struct

The parser struct used two have two somewhat misused fields:

peekCount int
token     [3]item

By reading the code carefully one notices, that peekCount always has the value 0 or 1 and that only the first element of token is ever accessed.

To make this clearer, this commit replaces the token array with a single variable and the peekCount int with a boolean.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-25 12:29:14 +00:00
Tobias Guggenmos d0cff29749 PromQL: Reduce Code duplication in AST traversion (#6362)
* promql: Add Children function for easier AST traversion

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-25 11:41:59 +00:00
Tobias Guggenmos ac3932ea35 Improve PromQL parser performance by making it non-concurrent (#6356)
Before this commit, the PromQL parser ran in two goroutines:
* The lexer goroutine that splits the input into tokens and sent them over a channel to
* the parser goroutine which produces the abstract syntax tree

The Problem with this approach is that the parser spends more time on goroutine creation
and syncronisation than on actual parsing.

This commit removes that concurrency and replaces the channel by a slice based buffer.

Benchmarks show that this makes the up to 7 times faster than before.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-21 18:43:09 +00:00
Harkishen Singh 37d666949c fixes space issue in duration range promql (#6295)
* fix space issue in duration range promql

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* updated logic

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* fixed lexer to skip over the spaces

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* added unittests for updated lexer

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* added unittests for updated lexer

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2019-11-11 09:56:24 +01:00
Tobias Guggenmos fe80cf4734 promql: Eliminate dead code (#6215)
peek() already ensures to not return a ItemComment so checking for this
is redundant.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-10-25 10:18:20 +02:00
Krasi Georgiev 5b8ff9b8a4
Merge the 2.13 release branch (#6169)
* Include tsdb tool in builds (#6085) (#6089)

Add the tsdb tool to promu so that it's included in the release
tarballs.

Signed-off-by: Ben Kochie <superq@gmail.com>

* web/ui: fix for CVE-2019-10215 (#6098)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* cut 2.13 release (#6099)

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

* Fix panic in ARM builds of Prometheus (#6110)

An extra sync.Pool was added during a refactor which caused some 64 bit,
atomically accessed variables to no longer be 64 bit aligned. By moving
all atomically accessed variables to the beginning of the struct they
are guaranteed to be 64 bit aligned.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

* promql: fix potential panic in the query logger (#6094)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Cut release 2.13.1 (#6145)

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-10-18 02:21:58 +03:00
Alex Dzyoba e3882629ba promql: Move tests to testutil (#6103)
* promql: Move tests to testutil

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>

* promql: Match error type via errors.As in tests

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>

* promql: Remove unused `expectedList` func from lex_test.go

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>
2019-10-09 18:06:53 -06:00
AllenZMC 1e62435960 fix wrong spells in live_reader.go (#5899)
* fix wrong spells in live_reader.go
* fix wrong spells in lex.go

Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-09-21 16:36:33 +03:00
yuxiaobo96 5f6a56a0bb Third correction (#6012)
Signed-off-by: yuxiaobo <yuxiaobogo@163.com>
2019-09-20 10:48:56 +01:00
johncming ac3cedb108 promql: Modify the order of return values. (#5940)
Signed-off-by: johncming <johncming@yahoo.com>
2019-08-24 02:39:15 +01:00
johncming d1b3a6e013 promql: Optimize the writing (#5941)
Signed-off-by: johncming <johncming@yahoo.com>
2019-08-24 02:38:48 +01:00
Krzysztof Kowalczyk 6e53980bbf fix fuzz targets (#5851)
Signed-off-by: Krzysztof Kowalczyk <kkowalczyk@gmail.com>
2019-08-12 20:47:57 +02:00
AllenZMC 3638e4ab18 fix mis-spelling in fuzz.go (#5848)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-09 15:19:45 +01:00
Chris Marchbanks 529ccff07b
Remove all usages of stretchr/testify
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:49:27 -06:00
Chris Marchbanks 0685eb5395
Refactor testutil.NewStorage into a new package
This avoids a circular dependency between the testutil and storage
packages.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:43:04 -06:00
AllenZMC 04cb37fea8 fix wrong spells in query_logger.go (#5843)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-07 15:02:53 +01:00
ethan 8928094b56 func name ref correct "qauntile" -> "quantile" (#5834)
Signed-off-by: ethan <guangming.wang@daocloud.io>
2019-08-06 06:11:16 +01:00
AllenZMC 8d6ec5acdc fix word 'seriess' to 'series' (#5828)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-04 16:52:51 +01:00
AllenZMC ead0933dd9 fix word 'substracting' to 'subtracting' (#5822)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-01 15:44:38 +01:00
Advait Bhatwadekar 5d401f1e1b Added query logging for prometheus. Issue #1315 (#5794)
* Added query logging for prometheus.
Options added:
1) active.queries.filepath: Filename where queries will be recorded
2) active.queries.filesize: Size of the file where queries will be recorded.

Functionality added:
All active queries are now logged in a file. If prometheus crashes unexpectedly, these queries are also printed out on stdout in the rerun.

Queries are written concurrently to an mmaped file, and removed once they are done. Their positions in the file are reused. They are written in json format. However, due to dynamic nature of application, the json has an extra comma after the last query, and is missing an ending ']'. There may also null bytes in the tail of file.

Signed-off-by: Advait Bhatwadekar <advait123@ymail.com>
2019-07-31 16:12:43 +01:00
Ganesh Vernekar 588eb20018
Efficient iteration and search in HashForLabels and HashWithoutLabels (#5707)
* Efficient iteration and search in Labels.HashForLabels

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Better names for variables

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* HashWithoutLabels optimizations

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor HashForLabels and HashWithoutLabels to take sorted names

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-06-28 18:22:51 +05:30
Thomas Jackson 91d7175eaa Add storage.Warnings to LabelValues and LabelNames (#5673)
Fixes #5661

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2019-06-17 08:31:17 +01:00
beorn7 dd81912554 Add objectives to Summaries
With the next release of client_golang, Summaries will not have
objectives by default. To not lose the objectives we have right now,
explicitly state the current default objectives.

Signed-off-by: beorn7 <beorn@grafana.com>
2019-06-12 02:03:13 +02:00
Thomas Jackson a000cec011 Re-use label builder in promql aggregation (#5641)
For my benchmarks on aggregation this reduces allocations by ~5% (~10%
time improvement):

```
benchmark                                                    old ns/op     new ns/op     delta
BenchmarkEvaluations/benchdata/aggregators.test/promxy-4     727692        649626        -10.73%

benchmark                                                    old allocs     new allocs     delta
BenchmarkEvaluations/benchdata/aggregators.test/promxy-4     2566           2434           -5.14%

benchmark                                                    old bytes     new bytes     delta
BenchmarkEvaluations/benchdata/aggregators.test/promxy-4     162760        148854        -8.54%
```

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2019-06-11 09:24:49 +01:00
Brian Brazil 36dbf042e3
Correctly handle {__name__="a"} (#5552)
This can cause problems in alerts.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-05-11 00:46:15 +01:00
Simon Pasquier 45506841e6
*: enable all default linters (#5504)
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-03 15:11:28 +02:00
Björn Rabenstein 0be9388f8d
Merge pull request #5463 from prometheus/beorn7/templating
Follow-up on #5009
2019-04-24 16:42:23 +02:00
Goutham Veeramachaneni 3cc5f9d880 Make sure subquery range is taken into account for selection (#5467)
* Make sure subquery range is taken into account for selection

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2019-04-17 13:52:41 +01:00
Bjoern Rabenstein a92ef68dd8 Fix staticcheck errors
Not sure why they only show up now.

Signed-off-by: Bjoern Rabenstein <bjoern@rabenste.in>
2019-04-17 01:40:10 +02:00
Julius Volz bc1c7f1809 Fix scalar-vector comparisons (#5454)
* Fix scalar-vector comparisons

Fixes https://github.com/prometheus/prometheus/issues/5452

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2019-04-11 10:42:16 +01:00