mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2025-03-05 20:59:13 -08:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
9299 commits 313 branches 456 tags 672 MiB
Commit graph

410 commits

Author SHA1 Message Date
Christian Simon 50c1060328
Implement PostingsCloner and implement promise/future pattern for PostingsForMatchers (#14) 
* Fix tests not closing head/block properly

Close Head properly in TestWalRepair_DecodingError
Close Block properly in  TestReadIndexFormatV1

Co-authored-by: Christian Simon <simon@swine.de>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Implement PostingsCloner

PostingsCloner allows obtaining independend clones of a Postings, that
can be used for subsequent calls.

Co-authored-by: Christian Simon <simon@swine.de>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* IndexReader.PostingsForMatchers() and implement Cache

IndexReader now offers the PostingsForMatchers functionality, which is
provided by PostingsForMatchersCache.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
 2021-10-07 15:16:09 +02:00
Cyril Tovena 9173cade01
Merge pull request #15 from grafana/improvesetmatches 
Improve `findSetMatches `to support concatenation.
 2021-10-07 14:48:53 +02:00
Cyril Tovena ce28912439
Fixes a test now that we support more setMatches. 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
 2021-10-07 14:04:52 +02:00
Peter Štibraný b3ae917f96
Merge pull request #17 from grafana/symbols-cleanup 
Splitting compactor: remove unused symbols from index of compacted block
 2021-10-06 14:36:45 +02:00
Cyril Tovena 92736182ce
Adding benchmark. 
```
benchstat before.txt after.txt
name                                                      old time/op    new time/op    delta
PostingsForMatchers/Head/n="1"-16                            480ns ± 4%     469ns ± 4%      ~     (p=0.095 n=5+5)
PostingsForMatchers/Head/n="1",j="foo"-16                    720ns ± 1%     705ns ± 2%      ~     (p=0.056 n=5+5)
PostingsForMatchers/Head/j="foo",n="1"-16                    712ns ± 3%     726ns ± 7%      ~     (p=0.841 n=5+5)
PostingsForMatchers/Head/n="1",j!="foo"-16                   833ns ± 3%     827ns ± 1%      ~     (p=0.548 n=5+5)
PostingsForMatchers/Head/i=~"1[0-9]",j=~"foo|bar"-16        6.66ms ± 4%    0.00ms ± 1%   -99.97%  (p=0.008 n=5+5)
PostingsForMatchers/Head/j=~"foo|bar"-16                     897ns ± 2%     675ns ± 2%   -24.80%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~"(1|2|3|4|5|6|20|55)"-16        20.1ms ± 2%     0.0ms ± 2%   -99.99%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*"-16                         74.4ms ± 3%    81.3ms ± 3%    +9.29%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~"1.*"-16                        20.3ms ±27%    17.2ms ± 2%   -15.15%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*1"-16                        6.22ms ±10%    5.60ms ± 1%    -9.95%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".+"-16                          146ms ± 4%     126ms ± 1%   -13.63%  (p=0.016 n=4+5)
PostingsForMatchers/Head/i=~""-16                           77.0ms ±23%    68.2ms ± 5%   -11.47%  (p=0.032 n=5+5)
PostingsForMatchers/Head/i!=""-16                           57.0ms ± 4%    56.9ms ± 6%      ~     (p=1.000 n=5+5)
PostingsForMatchers/Head/n="1",i=~".*",j="foo"-16           77.1ms ± 2%    79.2ms ± 4%      ~     (p=0.095 n=5+5)
PostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-16    77.2ms ± 3%    81.2ms ± 3%    +5.17%  (p=0.016 n=5+5)
PostingsForMatchers/Head/n="1",i!=""-16                     64.7ms ±34%    58.4ms ± 7%      ~     (p=0.421 n=5+5)

name                                                      old alloc/op   new alloc/op   delta
PostingsForMatchers/Head/n="1"-16                            80.0B ± 0%     80.0B ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",j="foo"-16                     208B ± 0%      208B ± 0%      ~     (all equal)
PostingsForMatchers/Head/j="foo",n="1"-16                     208B ± 0%      208B ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",j!="foo"-16                    240B ± 0%      240B ± 0%      ~     (all equal)
PostingsForMatchers/Head/i=~"1[0-9]",j=~"foo|bar"-16        1.61MB ± 0%    0.00MB ± 0%      ~     (p=0.079 n=4+5)
PostingsForMatchers/Head/j=~"foo|bar"-16                      360B ± 0%      232B ± 0%   -35.56%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~"(1|2|3|4|5|6|20|55)"-16        1.61MB ± 0%    0.00MB ± 0%   -99.97%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*"-16                         1.61MB ± 0%    1.61MB ± 0%      ~     (all equal)
PostingsForMatchers/Head/i=~"1.*"-16                        3.15MB ± 0%    3.15MB ± 0%    -0.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*1"-16                        1.61MB ± 0%    1.61MB ± 0%    -0.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".+"-16                         17.3MB ± 0%    17.3MB ± 0%    -0.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~""-16                           17.3MB ± 0%    17.3MB ± 0%      ~     (p=0.238 n=5+5)
PostingsForMatchers/Head/i!=""-16                           17.3MB ± 0%    17.3MB ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",i=~".*",j="foo"-16           1.61MB ± 0%    1.61MB ± 0%    -0.00%  (p=0.032 n=5+5)
PostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-16    1.61MB ± 0%    1.61MB ± 0%      ~     (p=0.651 n=5+4)
PostingsForMatchers/Head/n="1",i!=""-16                     17.3MB ± 0%    17.3MB ± 0%      ~     (all equal)

name                                                      old allocs/op  new allocs/op  delta
PostingsForMatchers/Head/n="1"-16                             4.00 ± 0%      4.00 ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",j="foo"-16                     9.00 ± 0%      9.00 ± 0%      ~     (all equal)
PostingsForMatchers/Head/j="foo",n="1"-16                     9.00 ± 0%      9.00 ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",j!="foo"-16                    10.0 ± 0%      10.0 ± 0%      ~     (all equal)
PostingsForMatchers/Head/i=~"1[0-9]",j=~"foo|bar"-16          19.0 ± 0%      23.0 ± 0%   +21.05%  (p=0.008 n=5+5)
PostingsForMatchers/Head/j=~"foo|bar"-16                      13.0 ± 0%       7.0 ± 0%   -46.15%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~"(1|2|3|4|5|6|20|55)"-16          4.00 ± 0%     13.00 ± 0%  +225.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*"-16                           7.00 ± 0%      7.00 ± 0%      ~     (all equal)
PostingsForMatchers/Head/i=~"1.*"-16                         11.1k ± 0%     11.1k ± 0%    -0.01%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".*1"-16                          4.00 ± 0%      3.00 ± 0%   -25.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~".+"-16                           100k ± 0%      100k ± 0%    -0.00%  (p=0.008 n=5+5)
PostingsForMatchers/Head/i=~""-16                             100k ± 0%      100k ± 0%      ~     (all equal)
PostingsForMatchers/Head/i!=""-16                             100k ± 0%      100k ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",i=~".*",j="foo"-16             12.0 ± 0%      12.0 ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-16      18.0 ± 0%      18.0 ± 0%      ~     (all equal)
PostingsForMatchers/Head/n="1",i!=""-16                       100k ± 0%      100k ± 0%      ~     (all equal)
```

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
 2021-10-06 14:16:14 +02:00
Peter Štibraný b7b70066ae Address review feedback. 2021-10-06 11:04:37 +02:00
Ganesh Vernekar e37ff8f6b6
Merge remote-tracking branch 'upstream/main' into sync 2021-10-06 13:59:18 +05:30
Peter Štibraný d116268e59 Always include empty symbol in the symbol table. 2021-10-06 10:17:16 +02:00
Peter Štibraný 9a50267ea5 Delete symbols files. 2021-10-05 17:56:20 +02:00
Ganesh Vernekar 10d4cb6dc0
Merge remote-tracking branch 'upstream/main' into release-2.30-merge 2021-10-05 20:35:14 +05:30
Peter Štibraný 259e09fe5f When doing compaction with splitting, only use symbols from series that belong to given sharded block. 2021-10-05 16:06:38 +02:00
Cyril Tovena bd4d997bee
lint 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
 2021-10-05 14:00:18 +02:00
Cyril Tovena c751cfbfce
Improve findSetMatches to support concatenation. 
This improves how we find `SetMatches` for regexp. Notably it allows to support concatenation
such as `api_(v1|prom)_push` for which the resulting set matches are `api_v1_push` and `api_prom_push`.

I had to support characters classes too since the syntax may try to optimize alternates with them.

In the end the code is also more robust than the previous implementation relying on the stringyfied version of the regexp.

This could be upstreamed later.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
 2021-10-05 13:43:41 +02:00
Ganesh Vernekar 10bc6e80ee
Fix panic on failed snapshot replay and don't hard fail replay on disabled exemplars (#9438) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-10-05 10:51:25 +05:30
Ganesh Vernekar b30db03f35
Cut v2.30.2 (#9426) 
* Don't error on overlapping m-mapped chunks during WAL replay (#9381)

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Reduce log level during WAL replay on overlapping m-map chunks (#9425)

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Cut v2.30.2

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-10-01 17:00:22 +05:30
Ganesh Vernekar a7d499e19a
Reduce log level during WAL replay on overlapping m-map chunks (#9425) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-10-01 15:33:29 +05:30
Ganesh Vernekar 8c597e5166
Don't error on overlapping m-mapped chunks during WAL replay (#9381) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-10-01 14:34:12 +05:30
Peter Štibraný 57daf79192 More review feedback. 2021-09-28 10:29:54 +02:00
Peter Štibraný db7fa7621c Use <ix+1>_of_<shardCount> formatting for better readibility. 2021-09-28 10:13:48 +02:00
Peter Štibraný 861f9083d8 Fix directory cleanup in case of compaction failure. 2021-09-27 17:05:14 +02:00
Peter Štibraný ffd281ab9d Address feedback. 2021-09-27 16:33:43 +02:00
Peter Štibraný 58dab3de8a Source blocks are deletable only if they are ALL empty. 2021-09-27 16:24:46 +02:00
Peter Štibraný 336f4260db Removed commented code. 2021-09-27 14:34:04 +02:00
Peter Štibraný 006c2d7d55 All output blocks will have the same timestamp. 
Minor updates.
 2021-09-27 14:22:51 +02:00
Bryan Boreham 1fb3c1b598
Replace calls to strings.Compare (#9397) 
< is clearer and faster. As the documentation says,
"Basically no one should use strings.Compare."

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-09-27 17:33:53 +05:30
Peter Štibraný 63dbb1c69a Make lint happy. 2021-09-27 12:49:08 +02:00
Peter Štibraný 78396b67dd Compactor with support for splitting input blocks into multiple output blocks. 2021-09-27 12:40:11 +02:00
Ganesh Vernekar 2bcd9f2f69
Link 2 more TSDB blog posts in tsdb/README.md (#9371) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-09-21 21:35:33 +05:30
Ganesh Vernekar 84f8307aa1
Merge remote-tracking branch 'upstream/main' into codesome/syncprom 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-09-17 15:58:45 +05:30
Darshan Chaudhary 1f688657bf
Call delete on head if interval overlaps (#9151) 
* Call delete on head if interval overlaps

Signed-off-by: darshanime <deathbullet@gmail.com>

* Garbage collect tombstones during head gc

Signed-off-by: darshanime <deathbullet@gmail.com>

* Truncate tombstones before min time during head gc

Signed-off-by: darshanime <deathbullet@gmail.com>

* Lock less by deleting all keys in a single pass

Signed-off-by: darshanime <deathbullet@gmail.com>

* Pass map to DeleteTombstones

Signed-off-by: darshanime <deathbullet@gmail.com>

* Create new slice to replace old one

Signed-off-by: darshanime <deathbullet@gmail.com>
 2021-09-16 12:20:03 +05:30
Marco Pracucci 8708b2d086
Update upstream Prometheus 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-09-14 10:28:19 +02:00
Ganesh Vernekar 30534e99d9
Take snapshot only after closing the WAL (#9328) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-09-13 18:30:41 +05:30
Ganesh Vernekar 8944520ccc
Fix deletion of old snapshots (#9314) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-09-08 19:53:44 +05:30
Bryan Boreham 2327236bb5
Decrement active_appenders metric when no samples added (#9230) 
* Decrement active_appenders metric when no samples added

Also add a test that the metric is incremented and decremented as
expected with and without samples.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Fix comment

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-09-08 14:49:58 +05:30
Bryan Boreham 87d909df4a
Remove symbols map from TSDB head (#9301) 
This saves memory, effort and locking.

Since every symbol is also added to postings, `Symbols()` can be
implemented there instead. This now has to build a map for
deduplication, but `Symbols()` is only called for compaction, and `gc()`
used to rebuild the symbols map after every compaction so not an
additional cost.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-09-08 14:48:48 +05:30
Callum Styan 93886d8417
Fix div by 0 panic is resize function. (#9286) 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2021-09-02 11:08:05 -07:00
Ganesh Vernekar 35b1a82594
Exemplars in snapshot (#9255) 
* Exemplars in snapshot

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Add docs

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-08-30 19:34:38 +05:30
Levi Harrison 06afe6162c
Also ignore func1 
Signed-off-by: Levi Harrison <git@leviharrison.dev>
 2021-08-28 22:42:22 -04:00
Julien Pivotto d5676fb9e0
Merge pull request #9254 from prometheus/superq/go1.17 
Build with Go 1.17 / npm 7 / node 16
 2021-08-28 18:36:42 +02:00
SuperQ e167a45c65
Add new Go build tags. 
Add new go:build comments based on 1.17 formatting[0].

[0]: https://golang.org/doc/go1.17#gofmt

Signed-off-by: SuperQ <superq@gmail.com>
 2021-08-27 10:24:14 +02:00
Callum Styan cc55e57c1b
Fix a data race in the loadWAL function caused by reusing the same error var in multiple goroutines (#9259) 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2021-08-27 11:49:34 +05:30
Ganesh Vernekar 8a5d8c15e3
Do not replay checkpoint if it is covered by snapshot (#9226) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-08-25 21:48:55 +05:30
Marco Pracucci 6c38feef8e
Fixed BenchmarkLoadWAL 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-08-25 16:44:53 +02:00
Marco Pracucci 9537ca8b84
Merged upstream 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-08-25 16:30:05 +02:00
Marco Pracucci 175efada22
Avoid deadlock when processing duplicate series record (#9170) (#8) 
* Avoid deadlock when processing duplicate series record

`processWALSamples()` needs to be able to send on its output channel
before it can read the input channel, so reads to allow this in case the
output channel is full.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* processWALSamples: update comment

Previous text seems to relate to an earlier implementation.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
 2021-08-24 09:01:04 +02:00
Marco Pracucci fbe211d3a8
Do not break exported functions signatures (#6) 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-08-20 18:37:47 +02:00
Bryan Boreham 9dfdc3eb36
Speed up BenchmarkPostings_Stats (#9213) 
The previous code re-used series IDs 1-1000 many times over, so a lot
of time was spent ensuring the lists of series were in ascending order.
The intended use of `MemPostings.Add()` is that all series IDs are
unique, and changing the benchmark to do this lets it finish ten times
faster.

(It doesn't affect the benchmark result, just the setup code)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-08-18 10:27:21 +01:00
Ganesh Vernekar 328a74ca36
Fix bugs and add enhancements to the chunk snapshot (#9185) 
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
 2021-08-17 18:08:16 +01:00
Marco Pracucci 481299f4a5
Added series hash cache support to TSDB (#5) 
* Added series hash cache support to TSDB

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed imports grouping

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-08-17 13:31:08 +00:00
Marco Pracucci a3ad212658
Improve sharding tests and comments (#4) 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-08-13 15:16:03 +02:00