prometheus/tsdb
Chris Marchbanks c5b3f0221f Decode WAL in Separate Goroutine (#6230)
* Make WAL replay benchmark more representative

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

* Move decoding records from the WAL into goroutine

Decoding the WAL records accounts for a significant amount of time on
startup, and can be done in parallel with creating series/samples to
speed up startup. However, records still must be handled in order, so
only a single goroutine can do the decoding.

benchmark
old ns/op     new ns/op     delta
BenchmarkLoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200-8
481607033     391971490     -18.61%
BenchmarkLoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50-8
836394378     629067006     -24.79%
BenchmarkLoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480-8
348238658     234218667     -32.74%

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-11-07 17:26:45 +01:00
..
chunkenc Correct spelling mistakes 2019-10-10 18:46:27 +08:00
chunks Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
cmd/tsdb Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
docs/format new world spelling mistake 2019-10-17 19:09:54 +08:00
encoding Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
errors Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
fileutil Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
goversion fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
index Head Cardinality Status Page (#6125) 2019-11-04 19:06:13 -07:00
labels Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
record Move WAL watcher code to tsdb/wal package. (#5999) 2019-09-19 14:45:41 +05:30
test Cleanup after merging tsdb into prometheus 2019-08-13 14:04:14 +05:30
testdata Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
tombstones remove debug fmt.Println in tombstones. (#6135) 2019-10-14 14:45:26 +03:00
tsdbutil Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
wal Garbage collect asynchronously in the WAL Watcher 2019-10-07 14:36:10 -06:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
block_test.go Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
CHANGELOG.md fix-up tsdb-typo (#5954) 2019-08-28 14:43:02 +01:00
compact.go Move WAL watcher code to tsdb/wal package. (#5999) 2019-09-19 14:45:41 +05:30
compact_test.go new world spelling mistake 2019-10-17 19:09:54 +08:00
db.go Merge the 2.13 release branch to master (#6117) 2019-10-09 17:41:46 +02:00
db_test.go Correct spelling mistakes 2019-10-10 18:46:27 +08:00
head.go Decode WAL in Separate Goroutine (#6230) 2019-11-07 17:26:45 +01:00
head_bench_test.go Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
head_test.go Decode WAL in Separate Goroutine (#6230) 2019-11-07 17:26:45 +01:00
mocks_test.go Move WAL watcher code to tsdb/wal package. (#5999) 2019-09-19 14:45:41 +05:30
querier.go fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
querier_test.go Move WAL watcher code to tsdb/wal package. (#5999) 2019-09-19 14:45:41 +05:30
README.md Edit TSDB README badges 2019-10-24 15:35:47 +05:30
repair.go tsdb: add block meta version constant (#5994) 2019-09-09 12:28:01 +03:00
repair_test.go tsdb: add block meta version constant (#5994) 2019-09-09 12:28:01 +03:00
wal.go fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
wal_test.go fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00

TSDB

GoDoc

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.