naivewong
6ab483071a
Improvement on postings intersection ( #616 )
...
* improvement on postings intersection
Signed-off-by: naivewong <867245430@qq.com>
2019-06-11 10:14:25 +02:00
Brian Brazil
89a90fe96c
Simplify mergedPostings.Seek ( #595 )
...
The current implementation leads to very slow behaviour when there's
many lists, this no worse than n log k, where k is the number of posting
lists.
Adjust benchmark to catch this.
Remove flattening of without lists, not needed anymore.
Benchmark versus 0.4.0 (used in Prometheus 2.7):
```
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1"-8 189907976 188863880 -0.55%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 113950106 110791414 -2.77%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 104965646 102388760 -2.45%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 138743592 104424250 -24.74%
BenchmarkHeadPostingForMatchers/i=~".*"-8 5279594954 5206096267 -1.39%
BenchmarkHeadPostingForMatchers/i=~".+"-8 8004610589 6184527719 -22.74%
BenchmarkHeadPostingForMatchers/i=~""-8 2476042646 1003920432 -59.45%
BenchmarkHeadPostingForMatchers/i!=""-8 7178244655 6059725323 -15.58%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8 199342649 166642946 -16.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8 215774683 167515095 -22.37%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8 2214714769 392943663 -82.26%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 2148727410 322289262 -85.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 2170658009 338458171 -84.41%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 235720135 70597905 -70.05%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 2190570590 343034307 -84.34%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8 2373784439 387297908 -83.68%
benchmark old allocs new allocs delta
BenchmarkHeadPostingForMatchers/n="1"-8 33 33 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 33 33 +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 33 33 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 41 39 -4.88%
BenchmarkHeadPostingForMatchers/i=~".*"-8 56 56 +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-8 251577 100115 -60.21%
BenchmarkHeadPostingForMatchers/i=~""-8 251123 100077 -60.15%
BenchmarkHeadPostingForMatchers/i!=""-8 251525 100112 -60.20%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8 42 39 -7.14%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8 52 42 -19.23%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8 251069 100101 -60.13%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 251473 100101 -60.19%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 250914 100102 -60.11%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 30038 11181 -62.78%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 250813 100105 -60.09%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8 281503 111260 -60.48%
benchmark old bytes new bytes delta
BenchmarkHeadPostingForMatchers/n="1"-8 10887600 10887600 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8 5456416 5456416 +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8 5456416 5456416 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8 5456640 5456544 -0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-8 258254504 258254472 -0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-8 520126192 281554792 -45.87%
BenchmarkHeadPostingForMatchers/i=~""-8 263446640 24908456 -90.55%
BenchmarkHeadPostingForMatchers/i!=""-8 520121144 281553664 -45.87%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8 7062448 7062272 -0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8 7063473 7062384 -0.02%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8 274325656 35793776 -86.95%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8 268926824 30362624 -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8 268882992 30363000 -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8 33193401 4269304 -87.14%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8 268875024 30363096 -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8 300589656 33099784 -88.99%
```
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-05-13 10:51:07 +01:00
Brian Brazil
f9bc0817a1
Avoid allocation in mergedPostings.Seek. ( #586 )
...
See
https://github.com/prometheus/prometheus/issues/5424#issuecomment-485199651
benchmark old
ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1"-4
5266 5267 +0.02%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4
6469 7801 +20.59%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4
4984 4963 -0.42%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4
7137 7527 +5.46%
BenchmarkHeadPostingForMatchers/i=~".*"-4
8055830839 8117651746 +0.77%
BenchmarkHeadPostingForMatchers/i=~".+"-4
9369298293 9742710251 +3.99%
BenchmarkHeadPostingForMatchers/i=~""-4
2363120708 2507789029 +6.12%
BenchmarkHeadPostingForMatchers/i!=""-4
9387069195 11740628557 +25.07%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4
109668312 107644136 -1.85%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4
106022679 105759760 -0.25%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
330201810 266421024 -19.32%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
355627801 292774913 -17.67%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
108132305 106697690 -1.33%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
358836972 362579998 +1.04%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
555879392 575858378 +3.59%
benchmark old
allocs new allocs delta
BenchmarkHeadPostingForMatchers/n="1"-4 8
8 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4 11
11 +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4 11
11 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4 17
17 +0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-4 58
58 +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-4
100115 100115 +0.00%
BenchmarkHeadPostingForMatchers/i=~""-4
100125 100125 +0.00%
BenchmarkHeadPostingForMatchers/i!=""-4
100112 100112 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4 19
19 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4 22
22 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
100109 100079 -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
100110 100079 -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
11176 11167 -0.08%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
100112 100082 -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
111307 111277 -0.03%
benchmark old
bytes new bytes delta
BenchmarkHeadPostingForMatchers/n="1"-4
1360 1360 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4
1392 1392 +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4
1392 1392 +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4
1520 1520 +0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-4
258254040 258254040 +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-4
281554280 281554280 +0.00%
BenchmarkHeadPostingForMatchers/i=~""-4
241554888 241554888 +0.00%
BenchmarkHeadPostingForMatchers/i!=""-4
281553664 281553664 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4
1607277 1607277 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4
1607389 1607389 +0.00%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
73076560 24907600 -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
73076765 24907723 -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
5416925 3794909 -29.94%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
73076779 24907819 -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
99874860 51705900 -48.23%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-04-24 13:12:58 +01:00
Brian Brazil
259847a6b1
Be smarter in how we look at matchers. ( #572 )
...
* Add unittests for PostingsForMatcher.
* Selector methods are all stateless, don't need a reference.
* Be smarter in how we look at matchers.
Look at all matchers to see if a label can be empty.
Optimise Not handling, so i!="2" is a simple lookup
rather than an inverse postings list.
All all the Withouts together, rather than
having to subtract each from all postings.
Change the pre-expand the postings logic to always do it before doing a
Without only. Don't do that if it's already a list.
The initial goal here was that the oft-seen pattern
i=~"something.+",i!="foo",i!="bar" becomes more efficient.
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1"-4 5888 6160 +4.62%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4 7190 6640 -7.65%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4 6038 5923 -1.90%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4 6030884 4850525 -19.57%
BenchmarkHeadPostingForMatchers/i=~".*"-4 887377940 230329137 -74.04%
BenchmarkHeadPostingForMatchers/i=~".+"-4 490316101 319931758 -34.75%
BenchmarkHeadPostingForMatchers/i=~""-4 594961991 130279313 -78.10%
BenchmarkHeadPostingForMatchers/i!=""-4 537542388 318751015 -40.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4 10460243 8565195 -18.12%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4 44964267 8561546 -80.96%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4 42244885 29137737 -31.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4 35285834 32774584 -7.12%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4 8951047 8379024 -6.39%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4 63813335 30672688 -51.93%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4 45381112 44924397 -1.01%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-04-09 11:59:45 +01:00
Bartek Płotka
3ab5f4e579
index: reduce empty postings trees ( #509 )
...
Improved Merge when all is empty.
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-03-21 18:23:00 +02:00
Brian Brazil
62b652fbd0
Improve Merge performance ( #531 )
...
Use a heap for Next for merges, and
pre-compute if there's many postings on the
unset path.
Add posting lookup benchmarks
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-02-28 17:23:55 +00:00
Simon Pasquier
d5d7a097e1
Update Makefile.common
...
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-04 16:08:43 +01:00
Brian Brazil
296f943ec4
More efficient Merge implementation. ( #486 )
...
Avoid a tree of merge objects, which can result in
what I suspect is n^2 calls to Seek when using Without.
With 100k metrics, and a regex of ^$ in BenchmarkHeadPostingForMatchers:
Before:
BenchmarkHeadPostingForMatchers-8 1 51633185216 ns/op 29745528 B/op 200357 allocs/op
After:
BenchmarkHeadPostingForMatchers-8 10 108924996 ns/op 25715025 B/op 101748 allocs/op
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-01-03 16:59:52 +00:00
Brian Brazil
407e12d051
Make MemPostings nested.
...
This saves memory, about a quarter of the size of the postings map
itself with high-cardinality labels (not including the post ids).
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-11-22 09:36:39 +00:00
Ganesh Vernekar
0fb73ed622
Changed recursion into iteration for removedPostings.Next()
2018-02-14 19:27:35 +05:30
Fabian Reinartz
1e55b7987f
Improve comments, handle allPostingsKey properly
2017-12-22 09:43:34 +01:00
Fabian Reinartz
67f0ca8f0e
Move index and chunk encoders to own packages
2017-12-21 11:27:54 +01:00
Fabian Reinartz
6f93a699e6
wipe
2016-12-02 17:49:05 +01:00
Fabian Reinartz
342aa82505
Fix pages and index package
2016-11-15 15:53:48 +01:00
