Bartlomiej Plotka
ad51c649b5
Fixed float conversion difference.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
cfba92a133
Addressed comments.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
c0a9ab0829
Close db properly in tests.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
fb79f515fc
Fixed second bug.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
aadffd1360
Finally found a fix for the bug I was chasing for 2h...
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
849faa407b
Minor fixes.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
2cf637fbf5
Addressed comments.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka
34426766d8
Unify Iterator interfaces. All point to storage now.
...
This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.
* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.
No logic was changed, only different source of abstractions, so no need for benchmarks.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:54 +00:00
Bartlomiej Plotka
88af973663
Merge pull request #6820 from codesome/break-compact
...
Break DB.Compact and DB.CompactHead and DB.CompactBlocks
2020-02-17 13:20:21 +00:00
Zhou Hao
e628fd7735
fix comments spelling ( #6829 )
...
Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
2020-02-17 12:45:11 +01:00
johncming
c30abf1e2b
tsdb/wal: remove unused argument.
...
Signed-off-by: johncming <johncming@yahoo.com>
2020-02-15 21:54:09 +08:00
Ganesh Vernekar
6f1d2ec73e
Break DB.Compact and DB.compactHead and DB.compactBlocks. Add DB.CompactHead.
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-02-14 20:33:26 +05:30
Ganesh Vernekar
bc6fd96979
Refactor TestGCChunkAccess and TestGCSeriesAccess to create chunks by appending samples
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-02-06 18:26:18 +05:30
Bartlomiej Plotka
641676b397
Merge pull request #6687 from pracucci/reduce-tsdb-head-inuse-memory
...
Trim TSDB head chunks after being cut, to reduce inuse memory
2020-02-06 10:55:33 +00:00
Ganesh Vernekar
0a27df92f0
Refactor tsdb/chunks/chunks.go for future PRs ( #6754 )
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-02-05 19:09:40 +05:30
Ganesh Vernekar
56bf0ee4dc
Add OpenMmapFileWithSize method ( #6753 )
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-02-05 19:08:30 +05:30
Marco Pracucci
699f3e8f4d
Added comments to the Chunk interface
...
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2020-02-05 13:07:41 +01:00
Marco Pracucci
0703dae7cc
Compact TSDB head chunks after being cut, to reduce inuse memory
...
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2020-02-05 13:00:39 +01:00
Ben Ye
492414542e
fix matcher for regex ( #6540 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-02-05 10:53:12 +00:00
LongKB
1ebcf3d5e4
Fix some typo in comments ( #6730 )
...
Signed-off-by: Kim Bao Long <longkb@vn.fujitsu.com>
2020-01-31 12:11:52 +05:30
Callum Styan
83601202d9
Add missing param to NewHead call in test. ( #6725 )
...
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2020-01-30 11:45:14 -08:00
Thor
17d8c49919
made stripe size configurable ( #6644 )
...
Signed-off-by: Thor <thansen@digitalocean.com>
2020-01-30 12:42:43 +05:30
Brian Brazil
61262159c4
Simplify benchmark given the new API
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 14:38:09 +00:00
Brian Brazil
38d32e0686
Don't sort postings if we only have one block.
...
Sorting the heads postings can be quite slow.
We only need sorted series when merging with another
querier, so only sort then.
This will make big queries that only touch the head faster,
though queries that touch both the head and a block will still
be the same speed. This probably won't help much with graphing
unless the range is under an hour, however it should make most
recording rules faster.
Add gaurantee that remote read streaming produces sorted series.
PromQL benchmarks for histograms show only 2-3% improvement, but
they're only over 1k series.
benchmark old ns/op new ns/op delta
BenchmarkQuerierSelect/Head/1of1000000-4 1375486282 507657736 -63.09%
BenchmarkQuerierSelect/Head/10of1000000-4 1387859004 507769850 -63.41%
BenchmarkQuerierSelect/Head/100of1000000-4 1387087935 506029110 -63.52%
BenchmarkQuerierSelect/Head/1000of1000000-4 1386869064 504521986 -63.62%
BenchmarkQuerierSelect/Head/10000of1000000-4 1386213685 505210422 -63.55%
BenchmarkQuerierSelect/Head/100000of1000000-4 1392754988 529842406 -61.96%
BenchmarkQuerierSelect/Head/1000000of1000000-4 1569414722 725059506 -53.80%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 1381019902 1370495863 -0.76%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 1375696209 1366789468 -0.65%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 1386009422 1364519297 -1.55%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 1377700532 1364486191 -0.96%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 1383539536 1369545314 -1.01%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 1410089163 1394731339 -1.09%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 1634744148 1581554956 -3.25%
BenchmarkQuerierSelect/Block/1of1000000-4 881741242 879839470 -0.22%
BenchmarkQuerierSelect/Block/10of1000000-4 880381562 882846038 +0.28%
BenchmarkQuerierSelect/Block/100of1000000-4 887519357 881016916 -0.73%
BenchmarkQuerierSelect/Block/1000of1000000-4 902194205 883433524 -2.08%
BenchmarkQuerierSelect/Block/10000of1000000-4 892321964 885130170 -0.81%
BenchmarkQuerierSelect/Block/100000of1000000-4 938604466 933527150 -0.54%
BenchmarkQuerierSelect/Block/1000000of1000000-4 1313510845 1295881124 -1.34%
benchmark old allocs new allocs delta
BenchmarkQuerierSelect/Head/1of1000000-4 4000056 4000018 -0.00%
BenchmarkQuerierSelect/Head/10of1000000-4 4000074 4000036 -0.00%
BenchmarkQuerierSelect/Head/100of1000000-4 4000254 4000216 -0.00%
BenchmarkQuerierSelect/Head/1000of1000000-4 4002054 4002016 -0.00%
BenchmarkQuerierSelect/Head/10000of1000000-4 4020054 4020016 -0.00%
BenchmarkQuerierSelect/Head/100000of1000000-4 4200054 4200016 -0.00%
BenchmarkQuerierSelect/Head/1000000of1000000-4 6000054 6000016 -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 4000071 4000071 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 4000089 4000089 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 4000269 4000269 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 4002069 4002069 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 4020069 4020069 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 4200069 4200069 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 6000069 6000069 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 6000023 6000022 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 6000059 6000058 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 6000419 6000418 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 6004019 6004018 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 6040019 6040018 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 6400019 6400018 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 10000020 10000019 -0.00%
benchmark old bytes new bytes delta
BenchmarkQuerierSelect/Head/1of1000000-4 229192200 176001176 -23.21%
BenchmarkQuerierSelect/Head/10of1000000-4 229193352 176002328 -23.21%
BenchmarkQuerierSelect/Head/100of1000000-4 229204872 176013848 -23.21%
BenchmarkQuerierSelect/Head/1000of1000000-4 229320072 176129048 -23.20%
BenchmarkQuerierSelect/Head/10000of1000000-4 230472072 177281048 -23.08%
BenchmarkQuerierSelect/Head/100000of1000000-4 241992072 188801048 -21.98%
BenchmarkQuerierSelect/Head/1000000of1000000-4 357192072 304001048 -14.89%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 229193928 229193928 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 229195080 229195080 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 229206600 229206600 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 229321800 229321800 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 230473800 230473800 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 241993800 241993800 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 357193800 357193800 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 227201516 227201500 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 227202924 227202908 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 227217036 227217020 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 227358156 227358140 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 228769356 228769340 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 242881356 242881340 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 384001616 384001600 -0.00%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
Brian Brazil
d682731efc
Extend BenchmarkQuerierSelect to use multiple blocks.
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
Ganesh Vernekar
21a5cf5d1d
Bring back tombstones to Head block ( #6542 )
...
* Bring back tombstones to Head block
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add test cases
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Cleanup
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-01-20 21:08:00 +05:30
Julien Pivotto
46d18112a3
tsdb: error on series with duplicate labels ( #6664 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-20 11:05:27 +00:00
Ganesh Vernekar
e0733a99e3
Expose DB.Compact()
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-01-20 12:59:49 +05:30
John McBride
669592a2c4
Exports metric for WAL write errors ( #6647 )
...
* Exports metric for WAL write errors
Signed-off-by: John McBride <jpmmcbride@gmail.com>
* Correct name for counter
Signed-off-by: John McBride <jpmmcbride@gmail.com>
* Move WAL write failure to wal.go
Signed-off-by: John McBride <jpmmcbride@gmail.com>
* WAL write fail metric moved to Log for external consumers
Signed-off-by: John McBride <jpmmcbride@gmail.com>
2020-01-17 12:12:04 -08:00
Ganesh Vernekar
bc42cf6806
Merge pull request #6559 from yeya24/tomb-test
...
Add a tomb interval test case
2020-01-16 11:42:28 +05:30
Julien Pivotto
201491fd1b
Remove old checkpoint dir if it still exists ( #6621 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-14 09:35:24 +00:00
Julien Pivotto
0cff72d756
tsdb: writePostingsToTmpFiles returns nil instead of err ( #6618 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-13 22:40:12 +00:00
Julien Pivotto
398bd84d6f
small tsdb fixes ( #6616 )
...
* tsdb: register compactions_skipped_total
That metric was not registered.
I also reordered the metrics in the list.
* tsdb: display correct error when WAL can't be read
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-13 22:15:45 +00:00
Bartlomiej Plotka
1e64d757f7
Merge pull request #6585 from prometheus/fix-symbols-lookup
...
tsdb: Fixed Symbol Lookup edge case; Added tests.
2020-01-10 11:44:23 +00:00
Guangwen Feng
5e906cc09a
Fix typo in comment for func AddPadding
...
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-01-10 18:11:30 +08:00
Bartlomiej Plotka
59ea320d96
tsdb: Fixed Symbol Lookup edge case; Added tests.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-09 11:35:58 +00:00
Bartlomiej Plotka
863613300d
Exposed FileWriter. ( #6589 )
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-09 11:28:10 +00:00
Nguyen Hai Truong
b7376f3bff
Remove duplicated words ( #6569 )
...
Although it is spelling mistake, it might make an affect while reading.
Signed-off-by: Nguyen Hai Truong <truongnh@fujitsu.com>
2020-01-08 08:43:27 +00:00
yeya24
06fd7cf3ba
add tomb interval testcase
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-01-07 20:24:46 +00:00
Brian Brazil
4708915ac6
Replace StringTuples with []string
...
Benchmarks show slight cpu/allocs improvements.
benchmark old ns/op new ns/op delta
BenchmarkPostingsForMatchers/Head/n="1"-4 269978625 235305110 -12.84%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 129739974 121646193 -6.24%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 123826274 122056253 -1.43%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 126962188 130038235 +2.42%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 6423653989 5991126455 -6.73%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 6934647521 7033370634 +1.42%
BenchmarkPostingsForMatchers/Head/i=~""-4 1177781285 1121497736 -4.78%
BenchmarkPostingsForMatchers/Head/i!=""-4 7033680256 7246094991 +3.02%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 293702332 287440212 -2.13%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 307628268 307039964 -0.19%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 512247746 480003862 -6.29%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 361199794 367066917 +1.62%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 478863761 476037784 -0.59%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 103394659 102902098 -0.48%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 482552781 475453903 -1.47%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 559257389 589297047 +5.37%
BenchmarkPostingsForMatchers/Block/n="1"-4 36492 37012 +1.42%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 557788 611903 +9.70%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 554443 573814 +3.49%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 553227 553826 +0.11%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 113855090 111707221 -1.89%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 133994674 136520728 +1.89%
BenchmarkPostingsForMatchers/Block/i=~""-4 38138091 36299898 -4.82%
BenchmarkPostingsForMatchers/Block/i!=""-4 28861213 27396723 -5.07%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 112699941 110853868 -1.64%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 113198026 111389742 -1.60%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 28994069 27363804 -5.62%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 29709406 28589223 -3.77%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 134695119 135736971 +0.77%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 26783286 25826928 -3.57%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 134733254 134116739 -0.46%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 160713937 158802768 -1.19%
benchmark old allocs new allocs delta
BenchmarkPostingsForMatchers/Head/n="1"-4 36 36 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 38 38 +0.00%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 38 38 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 42 40 -4.76%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 61 59 -3.28%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 100088 100087 -0.00%
BenchmarkPostingsForMatchers/Head/i=~""-4 100053 100051 -0.00%
BenchmarkPostingsForMatchers/Head/i!=""-4 100087 100085 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 44 42 -4.55%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 50 48 -4.00%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 100076 100074 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 100077 100075 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 100077 100074 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 11167 11165 -0.02%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 100082 100080 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 111265 111261 -0.00%
BenchmarkPostingsForMatchers/Block/n="1"-4 6 6 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 11 11 +0.00%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 11 11 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 15 13 -13.33%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 12 10 -16.67%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 100040 100038 -0.00%
BenchmarkPostingsForMatchers/Block/i=~""-4 100045 100043 -0.00%
BenchmarkPostingsForMatchers/Block/i!=""-4 100041 100039 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 17 15 -11.76%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 23 21 -8.70%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 100046 100044 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 100050 100048 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 100049 100047 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 11150 11148 -0.02%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 100055 100053 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 111238 111234 -0.00%
benchmark old bytes new bytes delta
BenchmarkPostingsForMatchers/Head/n="1"-4 10887816 10887817 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 5456648 5456648 +0.00%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 5456648 5456648 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 5456792 5456712 -0.00%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 258254408 258254328 -0.00%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 273912888 273912904 +0.00%
BenchmarkPostingsForMatchers/Head/i=~""-4 17266680 17266600 -0.00%
BenchmarkPostingsForMatchers/Head/i!=""-4 273912416 273912336 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 7062578 7062498 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 7062770 7062690 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 28152346 28152266 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 22721178 22721098 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 22721336 22721224 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 3623804 3623733 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 22721480 22721400 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 24816652 24816444 -0.00%
BenchmarkPostingsForMatchers/Block/n="1"-4 296 296 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 424 424 +0.00%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 424 424 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 1544 1464 -5.18%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 1606114 1606045 -0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 17264709 17264629 -0.00%
BenchmarkPostingsForMatchers/Block/i=~""-4 17264780 17264696 -0.00%
BenchmarkPostingsForMatchers/Block/i!=""-4 17264680 17264600 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 1606253 1606165 -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 1606445 1606348 -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 17264808 17264728 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 17264936 17264856 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 17264965 17264885 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 3148262 3148182 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 17265141 17265061 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 20416944 20416784 -0.00%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-07 12:20:03 +00:00
Brian Brazil
e9ede51753
Remove last vestiges of never used composite index code.
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-07 12:20:03 +00:00
Brian Brazil
000e306f2f
Handle V1 indexes, some of which have unsorted posting offset tables. ( #6564 )
...
Fixes #6535
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-06 14:06:11 +00:00
Brian Brazil
536d416299
Fix tsdb code and tests to work on Windows. ( #6547 )
...
Add back Windows CI, we lost it when tsdb was merged into the prometheus
repo. There's many tests failing outside tsdb, so only test tsdb for
now.
Fixes #6513
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-04 14:55:02 +00:00
Josh Soref
91d76c8023
Spelling ( #6517 )
...
* spelling: alertmanager
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: attributes
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: autocomplete
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: bootstrap
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: caught
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: chunkenc
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: compaction
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: corrupted
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: deletable
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: expected
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: fine-grained
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: initialized
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: iteration
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: javascript
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: multiple
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: number
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: overlapping
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: possible
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: postings
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: procedure
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: programmatic
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: queuing
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: querier
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: repairing
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: received
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: reproducible
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: retention
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: sample
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: segements
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: semantic
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: software [LICENSE]
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: staging
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: timestamp
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: unfortunately
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: uvarint
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: subsequently
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: ressamples
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Bartlomiej Plotka
3b8ef6386c
Fixed race in Chunks method. ( #6515 )
...
Added regression test.
Fixes #6512
Before (not deterministic result due to concurrency):
```
=== RUN TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL
```
After succuess on multiple runs.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2019-12-24 22:55:22 +01:00
Brian Brazil
2b653ee230
Write label indices based on the posting offset table.
...
This avoids having to build it up in RAM, and means that all variable
memory usage for compactions is now 0.25 bytes per symbol plus a few
O(labelnames) structures. So in practice, pretty close to constant
memory for compactions.
benchmark old ns/op new ns/op delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 662974828 667162981 +0.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 2459590377 2131168138 -13.35%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 3808280548 3919290378 +2.91%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 8513884311 8738099339 +2.63%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 1898843003 1944131966 +2.39%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 5601478437 6031391658 +7.67%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 11225096097 11359624463 +1.20%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 23994637282 23919583343 -0.31%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 891042098 826898358 -7.20%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 915949138 902555676 -1.46%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 955138431 879067946 -7.96%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 991447640 958785968 -3.29%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 1068729356 980249080 -8.28%
benchmark old allocs new allocs delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 470778 470556 -0.05%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 791429 791225 -0.03%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 1111514 1111257 -0.02%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 2111498 2111369 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 841433 841220 -0.03%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 1911469 1911202 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 3041558 3041328 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 6741534 6741382 -0.00%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 824856 820873 -0.48%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 887220 885180 -0.23%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 905253 901539 -0.41%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 925148 913632 -1.24%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 1019141 978727 -3.97%
benchmark old bytes new bytes delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 35694744 41523836 +16.33%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 53405264 59499056 +11.41%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 74160320 78151568 +5.38%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 120878480 135364672 +11.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 203832448 209925504 +2.99%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 341029208 346551064 +1.62%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 580217176 582345224 +0.37%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 1356872288 1363495368 +0.49%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 119535672 94815920 -20.68%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 115352280 95980776 -16.79%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 119472320 98724460 -17.37%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 111979312 94325456 -15.77%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 116628584 98566344 -15.49%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 12:05:42 +00:00
Brian Brazil
7d1aad46b8
Put postings in a temporary file during index writing.
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 00:55:29 +00:00
Brian Brazil
dee6981a6c
Move writing of index label indices into IndexWriter.
...
Now you only need to provide symbols and series to IndexWriter.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 22:15:35 +00:00
Brian Brazil
1733724e30
Factor out index file writing code.
...
Now that we have more than one file open at a time, deduplicate a bit.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:54:13 +00:00
Brian Brazil
85964ce567
During compaction spool postings offset table on the side.
...
This avoids building it up in memory.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:20:10 +00:00
