mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-09 23:24:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Brian Brazil d782387f81
Stream symbols during compaction. (#6468) 
Rather than buffer up symbols in RAM, do it one by one
during compaction. Then use the reader's symbol handling
for symbol lookups during the rest of the index write.

There is some slowdown in compaction, due to having to look through a file
rather than a hash lookup. This is noise to the overall cost of compacting
series with thousands of samples though.

benchmark                                                                                   old ns/op       new ns/op       delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        539917175       675341565       +25.08%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       2441815993      2477453524      +1.46%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       3978543559      3922909687      -1.40%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       8430219716      8586610007      +1.86%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      1786424591      1909552782      +6.89%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     5328998202      6020839950      +12.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     10085059958     11085278690     +9.92%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     25497010155     27018079806     +5.97%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               2427391406      2817217987      +16.06%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               2592965497      2538805050      -2.09%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               2437388343      2668012858      +9.46%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               2317095324      2787423966      +20.30%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               2600239857      2096973860      -19.35%

benchmark                                                                                   old allocs     new allocs     delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        500851         470794         -6.00%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       821527         791451         -3.66%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       1141562        1111508        -2.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       2141576        2111504        -1.40%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      871466         841424         -3.45%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     1941428        1911415        -1.55%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     3071573        3041510        -0.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     6771648        6741509        -0.45%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               731493         824888         +12.77%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               793918         887311         +11.76%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               811842         905204         +11.50%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               832244         925081         +11.16%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               921553         1019162        +10.59%

benchmark                                                                                   old bytes      new bytes      delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        40532648       35698276       -11.93%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       60340216       53409568       -11.49%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       81087336       72065552       -11.13%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       142485576      120878544      -15.16%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      208661368      203831136      -2.31%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     347345904      340484696      -1.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     585185856      576244648      -1.53%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     1357641792     1358966528     +0.10%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               126486664      119666744      -5.39%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               122323192      115117224      -5.89%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               126404504      119469864      -5.49%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               119047832      112230408      -5.73%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               136576016      116634800      -14.60%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 19:49:54 +00:00
..
chunkenc Correct spelling mistakes 2019-10-10 18:46:27 +08:00
chunks return err instead of panic for corrupted chunk (#6040) 2019-12-04 09:37:49 +02:00
cmd/tsdb Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
docs/format Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
encoding Avoid WriteAt for Postings. 2019-12-16 15:30:49 +00:00
errors Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
fileutil Compute WAL size and use it during retention size checks (#5886) 2019-11-12 09:40:16 +07:00
goversion fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
index Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
record Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
test Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
testdata Load only some offsets into the symbol table into memory. 2019-12-17 18:56:58 +00:00
tombstones remove debug fmt.Println in tombstones. (#6135) 2019-10-14 14:45:26 +03:00
tsdbutil Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
wal Simplify duration check for watcher WAL replay. 2019-11-26 16:53:11 -08:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
block_test.go Load only some offsets into the symbol table into memory. 2019-12-17 18:56:58 +00:00
CHANGELOG.md fix-up tsdb-typo (#5954) 2019-08-28 14:43:02 +01:00
compact.go Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
compact_test.go Add contexts to index writer to fix test races. 2019-12-16 17:28:29 +00:00
db.go tsdb: add error details in log. (#6415) 2019-12-09 10:37:01 +00:00
db_test.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
head.go Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
head_bench_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
head_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
mocks_test.go Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
querier.go Load only some offsets into the symbol table into memory. 2019-12-17 18:56:58 +00:00
querier_bench_test.go Add benchmark for Querier.Select over blocks and head. 2019-12-17 18:56:58 +00:00
querier_test.go Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
README.md Edit TSDB README badges 2019-10-24 15:35:47 +05:30
repair.go tsdb: add block meta version constant (#5994) 2019-09-09 12:28:01 +03:00
repair_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
tsdbblockutil.go Added CreateBlock and CreateHead functions to new file (#6331) 2019-11-21 19:10:25 +07:00
wal.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
wal_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00

README.md

TSDB

GoDoc

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.