Matthieu MOREL
e2ede285a2
refactor: move from io/ioutil to io and os packages ( #10528 )
...
* refactor: move from io/ioutil to io and os packages
* use fs.DirEntry instead of os.FileInfo after os.ReadDir
Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>
2022-04-27 11:24:36 +02:00
Bryan Boreham
2c1be4df7b
tsdb: more efficient sorting of postings read from WAL at startup ( #10500 )
...
* tsdb: avoid slice-to-interface allocation in EnsureOrder
This is pulling the `seriesRefSlice` out of the loop, so the compiler
doesn't allocate a new one on the heap every time.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* tsdb: use pointer type in Pool for EnsureOrder
As noted by staticcheck, Pool prefers the objects in the pool to have
pointer type. This is a little more fiddly to code, but avoids
allocation of a wrapper object every time a slice is put into the pool.
Removed a comment that said fixing this has a performance penalty: not
borne out by benchmarks.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-03-30 15:10:19 +05:30
Eng Zer Jun
3e67654d37
refactor: use T.TempDir() and B.TempDir to create temporary directory
...
The directory created by `T.TempDir()` and `B.TempDir()` is
automatically removed when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Reference: https://pkg.go.dev/testing#B.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-01-22 18:57:30 +08:00
Oleg Zaytsev
a83d46ee9c
Tidy postingsWithIndexHeap ( #10123 )
...
Unexported postingsWithIndexHeap's methods that don't need to be
exported, and added detailed comments.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2022-01-06 16:03:44 +05:30
Oleg Zaytsev
701545286d
Pop intersected postings heap without popping ( #10092 )
...
See this comment for detailed explanation:
https://github.com/prometheus/prometheus/pull/9907#issuecomment-1002189932
TL;DR: if we don't call Pop() on the heap implementation, we don't need
to return our param as an `interface{}` so we save an allocation.
This would be popped for every label value, so it can be thousands of
saved allocations here (see benchmarks).
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2022-01-05 16:16:43 +05:30
Oleg Zaytsev
3947238ce0
Label values with matchers by intersecting postings ( #9907 )
...
* LabelValues w/matchers by intersecting postings
Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).
Pending to be benchmarked.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark labelValuesWithMatchers
name old time/op new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0%
name old alloc/op new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0%
name old allocs/op new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Don't expand postings to intersect them
Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).
Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.
Still offers 10x latency improvement where previous latencies were
reaching 1s.
Benchmark results:
name \ time/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 157ms ± 0% 48ms ± 0% 110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 1.80s ± 0% 0.46s ± 0% 0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 144ms ± 0% 57ms ± 0% 125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304ms ± 0% 111ms ± 0% 177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 761ms ± 0% 164ms ± 0% 134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 6.11µs ± 0% 6.62µs ± 0% 4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 117ms ± 0% 62ms ± 0% 120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 1.44s ± 0% 0.24s ± 0% 0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 92.1ms ± 0% 70.3ms ± 0% 125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 196ms ± 0% 115ms ± 0% 170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 1.23s ± 0% 0.21s ± 0% 0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 1.06ms ± 0% 0.88ms ± 0% 0.92ms ± 0%
name \ alloc/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 29.5MB ± 0% 26.9MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 46.8MB ± 0% 251.5MB ± 0% 36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 29.5MB ± 0% 22.3MB ± 0% 19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 46.8MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 10.3kB ± 0% 138535.2kB ± 0% 6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 5.54kB ± 0% 7.09kB ± 0% 4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 39.1MB ± 0% 28.5MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 287MB ± 0% 253MB ± 0% 38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 34.3MB ± 0% 23.9MB ± 0% 20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 51.6MB ± 0% 25.5MB ± 0% 22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 144MB ± 0% 139MB ± 0% 0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 6.43kB ± 0% 8.66kB ± 0% 5.86kB ± 0%
name \ allocs/op old.txt intersect.txt intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo" 104k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 204k ± 0% 500k ± 0% 300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo" 66.0 ± 0% 255.0 ± 0% 139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 205.0 ± 0% 87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$" 5.20M ± 0% 0.70M ± 0% 0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo" 204k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo" 304k ± 0% 600k ± 0% 400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo" 3.00M ± 0% 0.00M ± 0% 0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1" 61.0 ± 0% 247.0 ± 0% 129.0 ± 0%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Apply comment suggestions from the code review
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Change else { if } to else if
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Remove sorting of label values
We were not sorting them before, so no need to sort them now
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 15:59:03 +01:00
Oleg Zaytsev
5e746e4e88
Check postings bytes length when decoding ( #9766 )
...
Added validation to expected postings length compared to the bytes slice
length. With 32bit postings, we expect to have 4 bytes per each posting.
If the number doesn't add up, we know that the input data is not
compatible with our code (maybe it's cut, or padded with trash, or even
written in a different coded).
This is needed in downstream projects to correctly identify cached
postings written with an unknown codec, but it's also a good idea to
validate it here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2021-11-24 15:26:37 +05:30
Peter Štibraný
422e7839d4
Add more size checks when writing individual sections in the index. ( #9710 )
...
* Add more size checks when writing individual sections in the index.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
* Use uint and add comment about it.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
2021-11-11 15:44:28 +05:30
beorn7
c954cd9d1d
Move packages out of deprecated pkg directory
...
This creates a new `model` directory and moves all data-model related
packages over there:
exemplar labels relabel rulefmt textparse timestamp value
All the others are more or less utilities and have been moved to `util`:
gate logging modetimevfs pool runtime
Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Dieter Plaetinck
cda025b5b5
TSDB: demistify SeriesRefs and ChunkRefs ( #9536 )
...
* TSDB: demistify seriesRefs and ChunkRefs
The TSDB package contains many types of series and chunk references,
all shrouded in uint types. Often the same uint value may
actually mean one of different types, in non-obvious ways.
This PR aims to clarify the code and help navigating to relevant docs,
usage, etc much quicker.
Concretely:
* Use appropriately named types and document their semantics and
relations.
* Make multiplexing and demuxing of types explicit
(on the boundaries between concrete implementations and generic
interfaces).
* Casting between different types should be free. None of the changes
should have any impact on how the code runs.
TODO: Implement BlockSeriesRef where appropriate (for a future PR)
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* feedback
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* agent: demistify seriesRefs and ChunkRefs
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
2021-11-06 15:40:04 +05:30
Marco Pracucci
309b094b92
Optimized MemPostings.EnsureOrder() ( #9673 )
...
* Optimizes MemPostings.EnsureOrder()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Ignore linter warning
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-05 10:01:23 +00:00
Mateusz Gozdek
1a6c2283a3
Format Go source files using 'gofumpt -w -s -extra'
...
Part of #9557
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Bryan Boreham
1fb3c1b598
Replace calls to strings.Compare ( #9397 )
...
< is clearer and faster. As the documentation says,
"Basically no one should use strings.Compare."
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-09-27 17:33:53 +05:30
Bryan Boreham
87d909df4a
Remove symbols map from TSDB head ( #9301 )
...
This saves memory, effort and locking.
Since every symbol is also added to postings, `Symbols()` can be
implemented there instead. This now has to build a map for
deduplication, but `Symbols()` is only called for compaction, and `gc()`
used to rebuild the symbols map after every compaction so not an
additional cost.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-09-08 14:48:48 +05:30
Bryan Boreham
9dfdc3eb36
Speed up BenchmarkPostings_Stats ( #9213 )
...
The previous code re-used series IDs 1-1000 many times over, so a lot
of time was spent ensuring the lists of series were in ascending order.
The intended use of `MemPostings.Add()` is that all series IDs are
unique, and changing the benchmark to do this lets it finish ten times
faster.
(It doesn't affect the benchmark result, just the setup code)
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-08-18 10:27:21 +01:00
Jupiter
84ab705318
32 should better be replaced by "symbolFactor" ( #9203 )
...
Signed-off-by: tanghengjian <1040104807@qq.com>
2021-08-13 16:38:53 +05:30
Marco Pracucci
84e786ebc1
Fixed Decoder.Series() error checking ( #9201 )
...
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-08-13 16:11:41 +05:30
Oleg Zaytsev
b1ed4a0a66
LabelNames API with matchers ( #9083 )
...
* Push the matchers for LabelNames all the way into the index.
NB This doesn't actually implement it in the index, just plumbs it through for now...
Signed-off-by: Tom Wilkie <tom@grafana.com>
* Hack it up. Does not work.
Signed-off-by: Tom Wilkie <tom@grafana.com>
* Revert changes I don't understand
Can't see why do we need to hold a mutex on symbols, and the purpose of
the LabelNamesFor method.
Maybe I'll need to re-add this later.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Implement LabelNamesFor
This method provides the label names that appear in the postings
provided. We do that deeper than the label values because we know
beforehand that most of the label names we'll be the same across
different postings, and we don't want to go down an up looking up the
same symbols for all different series.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Mutex on symbols should be unlocked
However, I still don't understand why do we need a mutex here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Fix head.LabelNamesFor
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Implement mockIndex LabelNames with matchers
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Nitpick on slice initialisation
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Add tests for LabelNamesWithMatchers
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Fix the mutex mess on head.LabelValues/LabelNames
I still don't see why we need to grab that unrelated mutex, but at least
now we're grabbing it consistently
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Check error after iterating postings
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use the error from posting when there was en error in postings
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Update storage/interface.go comment
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Update tsdb/index/index.go comment
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Update tsdb/index/index.go wrapped error msg
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Update tsdb/index/index.go wrapped error msg
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Update tsdb/index/index.go warpped error msg
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Remove unneeded comment
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Add testcases for LabelNames w/matchers in api.go
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use t.Cleanup() instead of defer in tests
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Tom Wilkie <tom@grafana.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-07-20 18:08:08 +05:30
Jupiter
7337ecf0d3
Log when total symbol size exceeds 2^32 bytes. ( #9104 )
...
* Compaction fails when total symbol size exceeds 2^32 bytes.
Signed-off-by: tanghengjian <1040104807@qq.com>
* Compaction fails when total symbol size exceeds 2^32 bytes.
Signed-off-by: tanghengjian <1040104807@qq.com>
* Compaction fails when total symbol size exceeds 2^32 bytes.
Signed-off-by: root <tanghengjian@oppo.com>
Co-authored-by: root <tanghengjian@oppo.com>
2021-07-20 15:51:36 +05:30
kjinan
e1370eecde
typos correct
...
Signed-off-by: kjinan <2008kongxiangsheng@163.com>
2021-05-20 09:52:33 +08:00
Mauro Stettler
7715fe3219
Add matchers to LabelValues() call ( #8400 )
...
* Accept matchers in querier LabelValues()
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* create matcher to only select metrics which have searched label
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* test case for merge querier with matchers
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* test LabelValues with matchers on head
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* add test for LabelValues on block
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* formatting fix
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Add comments
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* add missing lock release
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* remove unused parameter
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Benchmarks for LabelValues() methods on block/head
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Better comment
Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* update comment
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* minor refactor make code cleaner
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* better comments
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* fix expected errors in test
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Deleting parameter which can only be empty
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* fix comments
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* remove unnecessary lock
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* only lookup label value if label name was looked up
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Return error when there is one
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Call .Get() on decoder before checking errors
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* only lock head.symMtx when necessary
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* remove unnecessary delete()
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* re-use code instead of duplicating it
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Consistently return error from LabelValueFor()
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* move helper func from util.go to querier.go
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* Fix test expectation
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* ensure result de-duplication and sorting works
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
* return named error from LabelValueFor()
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-02-09 23:08:35 +05:30
Nguyen Le Vu Long
cd1dafc2fd
tsdb: Expose total number of label pairs in head in TSDB stats page ( #8343 )
...
* tsdb: Expose total number of label pairs in head
Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
* fix: add comment for NumLabelPairs
Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
* fix: remove comment
Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
2021-01-07 12:11:32 +05:30
Chris Marchbanks
c8f810083d
Cache label name and last value when adding series ( #8192 )
...
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2020-11-19 11:18:12 -07:00
Julien Pivotto
6c56a1faaa
Testify: move to require ( #8122 )
...
* Testify: move to require
Moving testify to require to fail tests early in case of errors.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* More moves
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Bartlomiej Plotka
3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. ( #8066 )
...
* MultiError: Refactored MultiError for more concise and safe usage.
* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.
NOTE: Potentially I would rename package to merrors. (: In different PR.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed review comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Fix after rebase.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-28 15:24:58 +00:00
Julien Pivotto
1282d1b39c
Refactor test assertions ( #8110 )
...
* Refactor test assertions
This pull request gets rid of assert.True where possible to use
fine-grained assertions.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-27 11:06:53 +01:00
Julien Pivotto
4e5b1722b3
Move away from testutil, refactor imports ( #8087 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Xiaochao Dong
a282d25099
tsdb: remove duplicate values set to reduce memory usage(map overhead) ( #7915 )
...
Signed-off-by: Xiaochao Dong (@damnever) <dxc.wolf@gmail.com>
2020-09-10 20:35:47 +05:30
Guangwen Feng
9439b2150d
Fix golint warning caused by misspell ( #7842 )
...
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-08-25 09:17:41 +01:00
Harold Dost
21a753c4e2
Make file permissions set to allow for wider umask options. ( #7782 )
...
0644 -> 0666 on all non vendored code.
Fixes #7717
Signed-off-by: Harold Dost <harolddost@gmail.com>
2020-08-12 23:23:17 +02:00
Robert-André Mauchin
ed6ce7ac98
Convert int to string using rune() ( #7707 )
...
See https://github.com/golang/go/issues/32479
Fix #7706 .
Signed-off-by: Robert-André Mauchin <zebob.m@gmail.com>
2020-08-03 15:10:04 +01:00
Julien Pivotto
62805b2fe9
tsdb: test for leaks ( #7566 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-21 10:08:06 +02:00
Ganesh Vernekar
082c17b691
Introduce SortedLabelValues/LabelValues to speedup queries for high cardinality ( #7448 )
...
* Introduce LabelValuesUnsorted to speedup queries for high cardinality
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add sort check
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-06-25 14:10:29 +01:00
ga
05038b48bd
Goroutine: Fix ambiguous variable ( #7175 )
...
Signed-off-by: Gaurav Singh <gaurav1086@gmail.com>
2020-04-28 11:02:26 +01:00
ZouYu
2b7437d60e
Fix some warnings: 'redundant type from array, slice, or map composite literal' ( #7109 )
...
Signed-off-by: ZouYu <zouy.fnst@cn.fujitsu.com>
2020-04-15 11:17:41 +01:00
Julien Pivotto
0cff72d756
tsdb: writePostingsToTmpFiles returns nil instead of err ( #6618 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-13 22:40:12 +00:00
Bartlomiej Plotka
1e64d757f7
Merge pull request #6585 from prometheus/fix-symbols-lookup
...
tsdb: Fixed Symbol Lookup edge case; Added tests.
2020-01-10 11:44:23 +00:00
Guangwen Feng
5e906cc09a
Fix typo in comment for func AddPadding
...
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
2020-01-10 18:11:30 +08:00
Bartlomiej Plotka
59ea320d96
tsdb: Fixed Symbol Lookup edge case; Added tests.
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-09 11:35:58 +00:00
Bartlomiej Plotka
863613300d
Exposed FileWriter. ( #6589 )
...
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-09 11:28:10 +00:00
Brian Brazil
4708915ac6
Replace StringTuples with []string
...
Benchmarks show slight cpu/allocs improvements.
benchmark old ns/op new ns/op delta
BenchmarkPostingsForMatchers/Head/n="1"-4 269978625 235305110 -12.84%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 129739974 121646193 -6.24%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 123826274 122056253 -1.43%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 126962188 130038235 +2.42%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 6423653989 5991126455 -6.73%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 6934647521 7033370634 +1.42%
BenchmarkPostingsForMatchers/Head/i=~""-4 1177781285 1121497736 -4.78%
BenchmarkPostingsForMatchers/Head/i!=""-4 7033680256 7246094991 +3.02%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 293702332 287440212 -2.13%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 307628268 307039964 -0.19%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 512247746 480003862 -6.29%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 361199794 367066917 +1.62%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 478863761 476037784 -0.59%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 103394659 102902098 -0.48%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 482552781 475453903 -1.47%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 559257389 589297047 +5.37%
BenchmarkPostingsForMatchers/Block/n="1"-4 36492 37012 +1.42%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 557788 611903 +9.70%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 554443 573814 +3.49%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 553227 553826 +0.11%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 113855090 111707221 -1.89%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 133994674 136520728 +1.89%
BenchmarkPostingsForMatchers/Block/i=~""-4 38138091 36299898 -4.82%
BenchmarkPostingsForMatchers/Block/i!=""-4 28861213 27396723 -5.07%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 112699941 110853868 -1.64%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 113198026 111389742 -1.60%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 28994069 27363804 -5.62%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 29709406 28589223 -3.77%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 134695119 135736971 +0.77%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 26783286 25826928 -3.57%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 134733254 134116739 -0.46%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 160713937 158802768 -1.19%
benchmark old allocs new allocs delta
BenchmarkPostingsForMatchers/Head/n="1"-4 36 36 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 38 38 +0.00%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 38 38 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 42 40 -4.76%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 61 59 -3.28%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 100088 100087 -0.00%
BenchmarkPostingsForMatchers/Head/i=~""-4 100053 100051 -0.00%
BenchmarkPostingsForMatchers/Head/i!=""-4 100087 100085 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 44 42 -4.55%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 50 48 -4.00%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 100076 100074 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 100077 100075 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 100077 100074 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 11167 11165 -0.02%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 100082 100080 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 111265 111261 -0.00%
BenchmarkPostingsForMatchers/Block/n="1"-4 6 6 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 11 11 +0.00%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 11 11 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 15 13 -13.33%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 12 10 -16.67%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 100040 100038 -0.00%
BenchmarkPostingsForMatchers/Block/i=~""-4 100045 100043 -0.00%
BenchmarkPostingsForMatchers/Block/i!=""-4 100041 100039 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 17 15 -11.76%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 23 21 -8.70%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 100046 100044 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 100050 100048 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 100049 100047 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 11150 11148 -0.02%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 100055 100053 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 111238 111234 -0.00%
benchmark old bytes new bytes delta
BenchmarkPostingsForMatchers/Head/n="1"-4 10887816 10887817 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j="foo"-4 5456648 5456648 +0.00%
BenchmarkPostingsForMatchers/Head/j="foo",n="1"-4 5456648 5456648 +0.00%
BenchmarkPostingsForMatchers/Head/n="1",j!="foo"-4 5456792 5456712 -0.00%
BenchmarkPostingsForMatchers/Head/i=~".*"-4 258254408 258254328 -0.00%
BenchmarkPostingsForMatchers/Head/i=~".+"-4 273912888 273912904 +0.00%
BenchmarkPostingsForMatchers/Head/i=~""-4 17266680 17266600 -0.00%
BenchmarkPostingsForMatchers/Head/i!=""-4 273912416 273912336 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",j="foo"-4 7062578 7062498 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".*",i!="2",j="foo"-4 7062770 7062690 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!=""-4 28152346 28152266 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i!="",j="foo"-4 22721178 22721098 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",j="foo"-4 22721336 22721224 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~"1.+",j="foo"-4 3623804 3623733 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!="2",j="foo"-4 22721480 22721400 -0.00%
BenchmarkPostingsForMatchers/Head/n="1",i=~".+",i!~"2.*",j="foo"-4 24816652 24816444 -0.00%
BenchmarkPostingsForMatchers/Block/n="1"-4 296 296 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4 424 424 +0.00%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4 424 424 +0.00%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4 1544 1464 -5.18%
BenchmarkPostingsForMatchers/Block/i=~".*"-4 1606114 1606045 -0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4 17264709 17264629 -0.00%
BenchmarkPostingsForMatchers/Block/i=~""-4 17264780 17264696 -0.00%
BenchmarkPostingsForMatchers/Block/i!=""-4 17264680 17264600 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4 1606253 1606165 -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4 1606445 1606348 -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4 17264808 17264728 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4 17264936 17264856 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4 17264965 17264885 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4 3148262 3148182 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4 17265141 17265061 -0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4 20416944 20416784 -0.00%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-07 12:20:03 +00:00
Brian Brazil
e9ede51753
Remove last vestiges of never used composite index code.
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-07 12:20:03 +00:00
Brian Brazil
000e306f2f
Handle V1 indexes, some of which have unsorted posting offset tables. ( #6564 )
...
Fixes #6535
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-06 14:06:11 +00:00
Brian Brazil
536d416299
Fix tsdb code and tests to work on Windows. ( #6547 )
...
Add back Windows CI, we lost it when tsdb was merged into the prometheus
repo. There's many tests failing outside tsdb, so only test tsdb for
now.
Fixes #6513
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-04 14:55:02 +00:00
Josh Soref
91d76c8023
Spelling ( #6517 )
...
* spelling: alertmanager
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: attributes
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: autocomplete
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: bootstrap
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: caught
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: chunkenc
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: compaction
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: corrupted
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: deletable
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: expected
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: fine-grained
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: initialized
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: iteration
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: javascript
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: multiple
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: number
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: overlapping
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: possible
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: postings
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: procedure
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: programmatic
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: queuing
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: querier
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: repairing
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: received
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: reproducible
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: retention
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: sample
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: segements
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: semantic
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: software [LICENSE]
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: staging
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: timestamp
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: unfortunately
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: uvarint
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: subsequently
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
* spelling: ressamples
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Brian Brazil
2b653ee230
Write label indices based on the posting offset table.
...
This avoids having to build it up in RAM, and means that all variable
memory usage for compactions is now 0.25 bytes per symbol plus a few
O(labelnames) structures. So in practice, pretty close to constant
memory for compactions.
benchmark old ns/op new ns/op delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 662974828 667162981 +0.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 2459590377 2131168138 -13.35%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 3808280548 3919290378 +2.91%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 8513884311 8738099339 +2.63%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 1898843003 1944131966 +2.39%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 5601478437 6031391658 +7.67%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 11225096097 11359624463 +1.20%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 23994637282 23919583343 -0.31%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 891042098 826898358 -7.20%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 915949138 902555676 -1.46%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 955138431 879067946 -7.96%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 991447640 958785968 -3.29%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 1068729356 980249080 -8.28%
benchmark old allocs new allocs delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 470778 470556 -0.05%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 791429 791225 -0.03%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 1111514 1111257 -0.02%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 2111498 2111369 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 841433 841220 -0.03%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 1911469 1911202 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 3041558 3041328 -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 6741534 6741382 -0.00%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 824856 820873 -0.48%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 887220 885180 -0.23%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 905253 901539 -0.41%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 925148 913632 -1.24%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 1019141 978727 -3.97%
benchmark old bytes new bytes delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 35694744 41523836 +16.33%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 53405264 59499056 +11.41%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 74160320 78151568 +5.38%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 120878480 135364672 +11.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4 203832448 209925504 +2.99%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4 341029208 346551064 +1.62%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4 580217176 582345224 +0.37%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4 1356872288 1363495368 +0.49%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4 119535672 94815920 -20.68%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4 115352280 95980776 -16.79%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4 119472320 98724460 -17.37%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4 111979312 94325456 -15.77%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4 116628584 98566344 -15.49%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 12:05:42 +00:00
Brian Brazil
7d1aad46b8
Put postings in a temporary file during index writing.
...
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 00:55:29 +00:00
Brian Brazil
dee6981a6c
Move writing of index label indices into IndexWriter.
...
Now you only need to provide symbols and series to IndexWriter.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 22:15:35 +00:00
Brian Brazil
1733724e30
Factor out index file writing code.
...
Now that we have more than one file open at a time, deduplicate a bit.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:54:13 +00:00
Brian Brazil
85964ce567
During compaction spool postings offset table on the side.
...
This avoids building it up in memory.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:20:10 +00:00
