mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-15 01:54:06 -08:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity
9759 commits 248 branches 440 tags 657 MiB
Commit graph

20 commits

Author SHA1 Message Date
Oleg Zaytsev 3f510ea6c9 Merge prometheus/a961062 2022-01-03 12:00:39 +01:00
Oleg Zaytsev 3947238ce0
Label values with matchers by intersecting postings (#9907) 
* LabelValues w/matchers by intersecting postings

Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).

Pending to be benchmarked.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Benchmark labelValuesWithMatchers

name                                                                   old time/op    new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                         157ms ± 0%        48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                      1.80s ± 0%       0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"                144ms ± 0%        57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"      304ms ± 0%       111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                      761ms ± 0%       164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        6.11µs ± 0%      6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                        117ms ± 0%        62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     1.44s ± 0%       0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              92.1ms ± 0%      70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     196ms ± 0%       115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     1.23s ± 0%       0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       1.06ms ± 0%      0.88ms ± 0%

name                                                                   old alloc/op   new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                        29.5MB ± 0%      26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                     46.8MB ± 0%     251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"               29.5MB ± 0%      22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     46.8MB ± 0%      23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                     10.3kB ± 0%  138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        5.54kB ± 0%      7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                       39.1MB ± 0%      28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     287MB ± 0%       253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              34.3MB ± 0%      23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    51.6MB ± 0%      25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     144MB ± 0%       139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       6.43kB ± 0%      8.66kB ± 0%

name                                                                   old allocs/op  new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                          104k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                       204k ± 0%        600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"                 104k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"       204k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                       66.0 ± 0%       255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                          61.0 ± 0%       205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                         304k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     5.20M ± 0%       0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"                204k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"      304k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     3.00M ± 0%       0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                         61.0 ± 0%       247.0 ± 0%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Don't expand postings to intersect them

Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).

Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.

Still offers 10x latency improvement where previous latencies were
reaching 1s.

Benchmark results:

name \ time/op                                                         old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                       157ms ± 0%        48ms ± 0%              110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                    1.80s ± 0%       0.46s ± 0%              0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"              144ms ± 0%        57ms ± 0%              125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    304ms ± 0%       111ms ± 0%              177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                    761ms ± 0%       164ms ± 0%              134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                      6.11µs ± 0%      6.62µs ± 0%             4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                      117ms ± 0%        62ms ± 0%              120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   1.44s ± 0%       0.24s ± 0%              0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"            92.1ms ± 0%      70.3ms ± 0%            125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"   196ms ± 0%       115ms ± 0%              170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   1.23s ± 0%       0.21s ± 0%              0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                     1.06ms ± 0%      0.88ms ± 0%             0.92ms ± 0%

name \ alloc/op                                                        old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                      29.5MB ± 0%      26.9MB ± 0%             19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                   46.8MB ± 0%     251.5MB ± 0%             36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"             29.5MB ± 0%      22.3MB ± 0%             19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"   46.8MB ± 0%      23.9MB ± 0%             20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                   10.3kB ± 0%  138535.2kB ± 0%              6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                      5.54kB ± 0%      7.09kB ± 0%             4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                     39.1MB ± 0%      28.5MB ± 0%             20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   287MB ± 0%       253MB ± 0%               38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"            34.3MB ± 0%      23.9MB ± 0%             20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"  51.6MB ± 0%      25.5MB ± 0%             22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   144MB ± 0%       139MB ± 0%                0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                     6.43kB ± 0%      8.66kB ± 0%             5.86kB ± 0%

name \ allocs/op                                                       old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                        104k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                     204k ± 0%        600k ± 0%               400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"               104k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     204k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                     66.0 ± 0%       255.0 ± 0%              139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        61.0 ± 0%       205.0 ± 0%               87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                       304k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   5.20M ± 0%       0.70M ± 0%              0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              204k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    304k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   3.00M ± 0%       0.00M ± 0%              0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       61.0 ± 0%       247.0 ± 0%              129.0 ± 0%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Apply comment suggestions from the code review

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* Change else { if } to else if

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Remove sorting of label values

We were not sorting them before, so no need to sort them now

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
 2021-12-28 15:59:03 +01:00
Peter Štibraný 294c155bb6 Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory 
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
 2021-11-09 08:03:10 +01:00
Dieter Plaetinck cda025b5b5
TSDB: demistify SeriesRefs and ChunkRefs (#9536) 
* TSDB: demistify seriesRefs and ChunkRefs

The TSDB package contains many types of series and chunk references,
all shrouded in uint types.  Often the same uint value may
actually mean one of different types, in non-obvious ways.

This PR aims to clarify the code and help navigating to relevant docs,
usage, etc much quicker.

Concretely:

* Use appropriately named types and document their semantics and
  relations.
* Make multiplexing and demuxing of types explicit
  (on the boundaries between concrete implementations and generic
  interfaces).
* Casting between different types should be free.  None of the changes
  should have any impact on how the code runs.

TODO: Implement BlockSeriesRef where appropriate (for a future PR)

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* feedback

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* agent: demistify seriesRefs and ChunkRefs

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
 2021-11-06 15:40:04 +05:30
Marco Pracucci cc97b25f2d
Upgrade upstream 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-11-05 11:08:47 +01:00
Marco Pracucci 309b094b92
Optimized MemPostings.EnsureOrder() (#9673) 
* Optimizes MemPostings.EnsureOrder()

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Ignore linter warning

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-11-05 10:01:23 +00:00
Oleg Zaytsev 14cb8d8696
Use ExpandPostings() in NewPostingsCloner() 
We should reuse that method, not only because it saves code, but also
because that method is a good candidate to use a pool of slices some
day.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
 2021-10-13 15:26:36 +02:00
Christian Simon 50c1060328
Implement PostingsCloner and implement promise/future pattern for PostingsForMatchers (#14) 
* Fix tests not closing head/block properly

Close Head properly in TestWalRepair_DecodingError
Close Block properly in  TestReadIndexFormatV1

Co-authored-by: Christian Simon <simon@swine.de>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Implement PostingsCloner

PostingsCloner allows obtaining independend clones of a Postings, that
can be used for subsequent calls.

Co-authored-by: Christian Simon <simon@swine.de>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* IndexReader.PostingsForMatchers() and implement Cache

IndexReader now offers the PostingsForMatchers functionality, which is
provided by PostingsForMatchersCache.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
 2021-10-07 15:16:09 +02:00
Bryan Boreham 1fb3c1b598
Replace calls to strings.Compare (#9397) 
< is clearer and faster. As the documentation says,
"Basically no one should use strings.Compare."

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-09-27 17:33:53 +05:30
Bryan Boreham 87d909df4a
Remove symbols map from TSDB head (#9301) 
This saves memory, effort and locking.

Since every symbol is also added to postings, `Symbols()` can be
implemented there instead. This now has to build a map for
deduplication, but `Symbols()` is only called for compaction, and `gc()`
used to rebuild the symbols map after every compaction so not an
additional cost.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-09-08 14:48:48 +05:30
Nguyen Le Vu Long cd1dafc2fd
tsdb: Expose total number of label pairs in head in TSDB stats page (#8343) 
* tsdb: Expose total number of label pairs in head

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>

* fix: add comment for NumLabelPairs

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>

* fix: remove comment

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
 2021-01-07 12:11:32 +05:30
Xiaochao Dong a282d25099
tsdb: remove duplicate values set to reduce memory usage(map overhead) (#7915) 
Signed-off-by: Xiaochao Dong (@damnever) <dxc.wolf@gmail.com>
 2020-09-10 20:35:47 +05:30
ga 05038b48bd
Goroutine: Fix ambiguous variable (#7175) 
Signed-off-by: Gaurav Singh <gaurav1086@gmail.com>
 2020-04-28 11:02:26 +01:00
Josh Soref 91d76c8023 Spelling (#6517) 
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-01-02 15:54:09 +01:00
Tim Bart 2e77f3a52b Fix typo in posting stats. (#6343) 
Signed-off-by: Tim Bart <tbart@cloudflare.com>
 2019-11-19 21:03:24 +00:00
Tom Wilkie de0a772b8e Port tsdb to use pkg/labels. (#6326) 
* Port tsdb to use pkg/labels.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Get tests passing.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Remove useless cast.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Appease linters.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-11-18 11:53:33 -08:00
Sharad Gaur e94503ff5c Head Cardinality Status Page (#6125) 
* Adding TSDB Head Stats like cardinality to Status Page

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Moving mutx to Head

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Renaming variabls

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Renaming variabls and html

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Removing unwanted whitespaces

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Adding Tests, Banchmarks and Max Heap for Postings Stats

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Adding more tests for postingstats and web handler

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Adding more tests for postingstats and web handler

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Remove generated asset file that is no longer used

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

* Changing comment and variable name for more readability

Signed-off-by: Sharad Gaur <sgaur@splunk.com>

* Using time.Duration in postings status function and removing refresh button from web page

Signed-off-by: Sharad Gaur <sgaur@splunk.com>
 2019-11-04 19:06:13 -07:00
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-08-13 14:04:14 +05:30
Ganesh Vernekar 7cf09b0395
Moving tsdb into its own subdirectory 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-08-13 13:58:49 +05:30
Renamed from index/postings.go (Browse further)