Commit graph

7577 commits

Author SHA1 Message Date
Chris Marchbanks 847c66a843 Add sharding test
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2020-01-06 14:53:23 -07:00
Bartlomiej Plotka f9fef91b59
Merge pull request #6566 from prometheus/pull-2.15.2
Pull commit Cut release 2.15.2 from release-2.15.2 branch.
2020-01-06 17:56:21 +00:00
Bartlomiej Plotka 801eaa22a5 Cut release 2.15.2
Cherry-picked 2 commits.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-06 17:40:15 +00:00
Bartlomiej Plotka d9613e5c46
Merge pull request #6565 from prometheus/fixes-for-2.15.2
Cut release 2.15.2; cherry-picked windows fix and index with unsorted postings support.
2020-01-06 14:46:42 +00:00
Bartlomiej Plotka bfe3dafcf0 Cut release 2.15.2
Cherry-picked 2 commits.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-01-06 14:36:22 +00:00
Brian Brazil 13e88e5a49 Handle V1 indexes, some of which have unsorted posting offset tables. (#6564)
Fixes #6535

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-06 14:28:05 +00:00
Brian Brazil ada0945b8f Fix tsdb code and tests to work on Windows. (#6547)
Add back Windows CI, we lost it when tsdb was merged into the prometheus
repo. There's many tests failing outside tsdb, so only test tsdb for
now.

Fixes #6513

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-06 14:28:02 +00:00
Brian Brazil 000e306f2f
Handle V1 indexes, some of which have unsorted posting offset tables. (#6564)
Fixes #6535

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-06 14:06:11 +00:00
Julien Pivotto 577e738986 Cleanup PromQL functions (#6551)
* Cleanup PromQL functions

The engine ensures, for Matrix functions, that functions are called with exactly one series at the time.
Therefore a lot of code can be inlined and we can directly assume the first element of the arguments exists and contains all the samples needed.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-06 10:33:36 +00:00
Brian Brazil 536d416299
Fix tsdb code and tests to work on Windows. (#6547)
Add back Windows CI, we lost it when tsdb was merged into the prometheus
repo. There's many tests failing outside tsdb, so only test tsdb for
now.

Fixes #6513

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-04 14:55:02 +00:00
Julien Pivotto e0afec906f add absent_over_time (#6490)
* Implement absent_over_time

Fixes #2882

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-03 15:26:12 +00:00
Josh Soref 91d76c8023 Spelling (#6517)
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Brian Brazil a9ed3b980d
Optimise labels.Compare (#6539)
strings.Compare isn't meant to be used, and this way we save one
comparison which is thus very slightly cheaper.

benchmark                                     old ns/op     new ns/op delta
BenchmarkPostingsForMatchers/Head/n="1"-4     236305440     233515705 -1.18%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-01 15:45:01 +00:00
Harkishen Singh edf8f135bc Adds support service discovery page in react ui (#6394)
* active targets component completed

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* support for service-discovery in react ui

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* restored prev files

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* used fc

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* removed trivial keys

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* FC based labels

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implmented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* minor word change

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* before dropped addressed

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* linted

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions. removed false styles

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions. Unified buttons with targets screen.

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* component for ToggleButton

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* removed false Button

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions.

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* tests for ToggleMoreLess component

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* linted

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* fixed nested h3. implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* linted

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>
2019-12-31 18:41:50 +01:00
Julius Volz ae93bae88f Fix: exclude metric name in labels.MatchLabels() (#6523)
This function is only used in one place to format an error message when
encountering multiple matches on the "one" side of a
one-to-many/many-to-one vector match, which is probably why nobodoy has
noticed this before. The hashing is already done correctly and excludes
the metric name label when using the "ignoring" matching modifier.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2019-12-27 09:32:19 +00:00
Bartlomiej Plotka 199f062c31
Merge pull request #6519 from prometheus/pull-2.15.1
Merge release-2.15 branch to master: Cut release 2.15.1
2019-12-25 10:55:29 +01:00
Bartlomiej Plotka 54654340ad Cut release 2.15.1
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2019-12-25 09:06:37 +01:00
Bartlomiej Plotka 8744510c63
Merge pull request #6518 from prometheus/fix-2.15.1
Cut 2.15.1 + cherry-picked fix.
2019-12-25 01:58:51 +01:00
Bartlomiej Plotka e6e48e6dce Cut release 2.15.1
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2019-12-24 22:57:04 +01:00
Bartlomiej Plotka c2e083c30f Fixed race in Chunks method. (#6515)
Added regression test.

Fixes #6512

Before (not deterministic result due to concurrency):
```
=== RUN   TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
    db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
    db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL
```

After succuess on multiple runs.


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2019-12-24 22:56:59 +01:00
Bartlomiej Plotka 3b8ef6386c
Fixed race in Chunks method. (#6515)
Added regression test.

Fixes #6512

Before (not deterministic result due to concurrency):
```
=== RUN   TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
    db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
    db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL
```

After succuess on multiple runs.


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2019-12-24 22:55:22 +01:00
Julien Pivotto 9398b9ac1b OpenMetrics: force input to end with # EOF (#6505)
Fixes #1169

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-24 08:48:28 +00:00
Bartlomiej Plotka fa4658e4de
Pull release-2.15 branch to master. (#6510)
* Cut 2.15.0-rc.0 release. (#6479)

* Cut 2.15.0-rc.0

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

* Cut 2.15.0 release. (#6509)

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-12-23 16:19:22 +01:00
Bartlomiej Plotka ec1868b026
Cut 2.15.0 release. (#6509)
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-12-23 11:57:27 +00:00
Tobias Guggenmos 213a8fe89a PromQL: Parse Series descriptions using the generated parser (#6494)
* Use generated parser for series descriptions

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-21 08:44:12 +00:00
Julien Pivotto f363a71ef7 promql: function: fix typo (#6501)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 22:30:55 +00:00
Julien Pivotto 4b3bab890d Fix absent output where two label matchers are identical (#6493)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 14:33:33 +00:00
Tobias Guggenmos a48acbcb23 PromQL: Fix parser benchmark (#6495)
The parser benchmarks called the `ParseMetric` function instead of the `ParseExpr` function, which resulted in parsing failing every time.

This means only the case of PromQL parser failure was benchmarked.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:30:41 +00:00
Tobias Guggenmos 1e0cd28bf6 PromQL: Support trailing commas in grouping opts (#6480)
* PromQL: Support trailing commas in grouping opts

Fixes #6470

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:28:56 +00:00
Julien Pivotto 06b4744793 docs: fix absent documentation (#6492)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 07:33:08 +00:00
Julien Pivotto f315c69129 promql: fix typo in value.go (#6491)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-19 22:30:32 +00:00
Ganesh Vernekar 34e0376ef8
Merge pull request #6485 from roidelapluie/fullstop
manager: add full stops on comments
2019-12-19 16:21:54 +05:30
Julien Pivotto e079c9ed45 manager: add full stops on comments
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-19 11:46:22 +01:00
alburthoffman 156dcb8cca avoid stopping rule groups if new rule groups are as same as old rule… (#6450)
* avoid stopping rule groups if new rule groups are as same as old rule groups

Signed-off-by: alburthoffman <alburthoffman@gmail.com>
2019-12-19 10:41:11 +00:00
Tobias Guggenmos db1258f2a5 PromQL: Refactor error message generation (#6481)
* Add parser method to produce errors messages about unexpected items
* PromQL: use parser.unexpected in generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 17:36:43 +00:00
Bartlomiej Plotka f5b9bc6db3 docs: Be more precise when pushing tags on release. (#6483)
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-12-18 17:32:33 +00:00
Tom Janson 4f1cfd6cfe remove redundant/accidental line in query doc (#6482)
Signed-off-by: Tom Janson <tom.janson@lancom.de>
2019-12-18 17:31:28 +00:00
Bartlomiej Plotka 61f1953c50
Cut 2.15.0-rc.0 release. (#6479)
* Cut 2.15.0-rc.0

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-12-18 16:57:51 +00:00
Tobias Guggenmos 9e34f08ac3 PromQL: Parse grouping opts with the generated parser (#6472)
* PromQL: Parse grouping opts with the generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 14:18:52 +00:00
Simon Pasquier f174ae1f09 Revert "Added s390x support for docker image (#6307)" (#6478)
This reverts commit f9bf8e4e4c.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-12-18 13:31:42 +00:00
Brian Brazil 92f31c3ec2
Merge pull request #6475 from prometheus/constant-compaction
Constant memory compaction
2019-12-18 13:22:02 +00:00
Julien Pivotto 2d7c8069d0 Check that rules don't contain metrics with the same labelset (#6469)
Closes #5529

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-18 12:29:35 +00:00
Brian Brazil 2b653ee230 Write label indices based on the posting offset table.
This avoids having to build it up in RAM, and means that all variable
memory usage for compactions is now 0.25 bytes per symbol plus a few
O(labelnames) structures. So in practice, pretty close to constant
memory for compactions.

benchmark                                                                                   old ns/op       new ns/op       delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        662974828       667162981       +0.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       2459590377      2131168138      -13.35%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       3808280548      3919290378      +2.91%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       8513884311      8738099339      +2.63%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      1898843003      1944131966      +2.39%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     5601478437      6031391658      +7.67%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     11225096097     11359624463     +1.20%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     23994637282     23919583343     -0.31%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               891042098       826898358       -7.20%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               915949138       902555676       -1.46%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               955138431       879067946       -7.96%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               991447640       958785968       -3.29%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               1068729356      980249080       -8.28%

benchmark                                                                                   old allocs     new allocs     delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        470778         470556         -0.05%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       791429         791225         -0.03%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       1111514        1111257        -0.02%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       2111498        2111369        -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      841433         841220         -0.03%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     1911469        1911202        -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     3041558        3041328        -0.01%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     6741534        6741382        -0.00%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               824856         820873         -0.48%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               887220         885180         -0.23%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               905253         901539         -0.41%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               925148         913632         -1.24%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               1019141        978727         -3.97%

benchmark                                                                                   old bytes      new bytes      delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        35694744       41523836       +16.33%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       53405264       59499056       +11.41%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       74160320       78151568       +5.38%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       120878480      135364672      +11.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      203832448      209925504      +2.99%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     341029208      346551064      +1.62%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     580217176      582345224      +0.37%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     1356872288     1363495368     +0.49%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               119535672      94815920       -20.68%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               115352280      95980776       -16.79%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               119472320      98724460       -17.37%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               111979312      94325456       -15.77%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               116628584      98566344       -15.49%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 12:05:42 +00:00
vitt-bagal f9bf8e4e4c Added s390x support for docker image (#6307)
Signed-off-by: Vitthal Bagal <vitthalb@us.ibm.com>
2019-12-18 11:21:03 +01:00
Brian Brazil 7d1aad46b8 Put postings in a temporary file during index writing.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-18 00:55:29 +00:00
Brian Brazil dee6981a6c Move writing of index label indices into IndexWriter.
Now you only need to provide symbols and series to IndexWriter.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 22:15:35 +00:00
Brian Brazil 1733724e30 Factor out index file writing code.
Now that we have more than one file open at a time, deduplicate a bit.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:54:13 +00:00
Brian Brazil 85964ce567 During compaction spool postings offset table on the side.
This avoids building it up in memory.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 21:20:10 +00:00
Julien Pivotto 31700a05df Improve testutil.ErrorEqual (#6471)
Also improves TestPopulateLabels: testutil.ErrorEqual just returned a
bool without failing the test.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-17 21:11:33 +00:00
Brian Brazil d782387f81
Stream symbols during compaction. (#6468)
Rather than buffer up symbols in RAM, do it one by one
during compaction. Then use the reader's symbol handling
for symbol lookups during the rest of the index write.

There is some slowdown in compaction, due to having to look through a file
rather than a hash lookup. This is noise to the overall cost of compacting
series with thousands of samples though.

benchmark                                                                                   old ns/op       new ns/op       delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        539917175       675341565       +25.08%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       2441815993      2477453524      +1.46%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       3978543559      3922909687      -1.40%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       8430219716      8586610007      +1.86%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      1786424591      1909552782      +6.89%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     5328998202      6020839950      +12.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     10085059958     11085278690     +9.92%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     25497010155     27018079806     +5.97%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               2427391406      2817217987      +16.06%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               2592965497      2538805050      -2.09%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               2437388343      2668012858      +9.46%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               2317095324      2787423966      +20.30%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               2600239857      2096973860      -19.35%

benchmark                                                                                   old allocs     new allocs     delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        500851         470794         -6.00%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       821527         791451         -3.66%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       1141562        1111508        -2.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       2141576        2111504        -1.40%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      871466         841424         -3.45%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     1941428        1911415        -1.55%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     3071573        3041510        -0.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     6771648        6741509        -0.45%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               731493         824888         +12.77%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               793918         887311         +11.76%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               811842         905204         +11.50%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               832244         925081         +11.16%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               921553         1019162        +10.59%

benchmark                                                                                   old bytes      new bytes      delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        40532648       35698276       -11.93%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       60340216       53409568       -11.49%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       81087336       72065552       -11.13%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       142485576      120878544      -15.16%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      208661368      203831136      -2.31%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     347345904      340484696      -1.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     585185856      576244648      -1.53%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     1357641792     1358966528     +0.10%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               126486664      119666744      -5.39%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               122323192      115117224      -5.89%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               126404504      119469864      -5.49%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               119047832      112230408      -5.73%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               136576016      116634800      -14.60%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 19:49:54 +00:00