prometheus/tsdb
Mauro Stettler cc9ddf50c9
Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57)
* write chunks via queue, predicting chunkrefs

* fixes

* linter

* cleanup

* better test coverage

* deduplicate operations when cutting file

* better comments

* fix race

* improvements based on PR feedback

* comments

* concurrent correctness test of write queue

* linter

* use isStarted flag in chunk write queue

* separate mutex for isStarted

* fixing tests

* fix race in test

* PR feedback

* add license headers

* PR feedback

* pr feedback

* dont recreate workerCtrl channel

* address PR feedback in high concurrency read and write test

* refactor the high concurrency write queue test

* pr feedback

* use require.Eventually

* typo

* Fixing comment

Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

* use buffered chan for chunk write queue

* fix races

* address feedback on PR

* better comment

* less type casts

* dont reeimplement varint

* move mtx above property it protects

* improve tests by using wg instead of inspecting queue

* add comment

* add comment

* writeChunkF comment

* rename isStarted -> isRunning

* prevent panic when double stopping

* fix variable name in comment

* delete outdated comment

* naming and comment fixes in TestChunkWriteQueue_WrappingAroundSizeLimit

* use require.False instead of require.Equal(t, false

* always enable chunk write queue

* dont call callback with lock held

* undo rename of curFileSequence to curFileSeq

* fix accidental change

* fix comment

* Better syntax

Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

* Better wording in comment

Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

* better comments to explain the use of coordinator chans in test

* fix test which broke to the async writing of chunks

* undo previous renaming of shadowing variable

* rename var threadID to workerID

* rename variables based on PR feedback

* rename pos -> ts

* better implementation of whileNotCanceled

* update querySeriesRef before using it

* rename labels -> lbls

* initialize the chunk write queue operations metric with all possible labels

* dont return error from CutNewFile

* recreate chunk ref map regularly to free

* limit how often we recreate the chunk ref map

* fix typo in comment

* better comment

Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
2021-12-08 13:14:26 +05:30
..
agent Share TSDB locker code with agent (#9623) 2021-11-11 11:45:25 -05:00
chunkenc improve bstream comments and doc (#9560) 2021-10-25 18:44:15 +05:30
chunks Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
docs clarify HeadChunkID type and usage (#9726) 2021-11-17 18:35:10 +05:30
encoding Format Go source files using 'gofumpt -w -s -extra' 2021-11-02 19:52:34 +01:00
errors Address golint failures from revive 2021-10-23 00:53:11 +02:00
fileutil tsdb: Add windows arm64 support. 2021-11-09 11:07:27 +01:00
goversion Add new Go build tags. 2021-08-27 10:24:14 +02:00
hashcache Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
index Merge remote-tracking branch 'prometheus/main' into upgrade-upstream 2021-11-25 12:01:25 +01:00
record Add basic initial developer docs for TSDB (#9451) 2021-11-17 15:51:27 +05:30
test Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones TSDB: demistify SeriesRefs and ChunkRefs (#9536) 2021-11-06 15:40:04 +05:30
tsdbutil Share TSDB locker code with agent (#9623) 2021-11-11 11:45:25 -05:00
wal fix: panic when checkpoint directory is empty (#9687) 2021-11-17 16:39:04 +05:30
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
async_block_writer.go Introduced some options for compactor concurrency (#66) 2021-12-02 10:34:52 +01:00
block.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
block_test.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
blockwriter.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
blockwriter_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go Compactor: Open blocks concurrently (#67) 2021-12-02 11:42:29 +00:00
compact_test.go Compactor: Open blocks concurrently (#67) 2021-12-02 11:42:29 +00:00
db.go Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
db_test.go Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
example_test.go Add basic initial developer docs for TSDB (#9451) 2021-11-17 15:51:27 +05:30
exemplar.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
exemplar_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
head.go Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
head_append.go Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
head_append_test.go Merge pull request #29 from grafana/add-jitter-to-chunk-end 2021-11-16 11:05:07 +01:00
head_bench_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
head_read.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-19 12:11:26 +01:00
head_test.go Write m-map chunks in async via a queue while predicting the m-map chunk refs (#57) 2021-12-08 13:14:26 +05:30
head_wal.go track OOO-ness in tsdb into a histogram (#56) 2021-11-29 16:44:35 +05:30
isolation.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
isolation_test.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
mocks_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
postings_for_matchers_cache.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
postings_for_matchers_cache_test.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
querier.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
querier_bench_test.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
querier_test.go Merge remote-tracking branch 'upstream/main' into merge-upstream 2021-11-18 15:48:40 +01:00
README.md Add basic initial developer docs for TSDB (#9451) 2021-11-17 15:51:27 +05:30
repair.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
repair_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
symbols_batch.go Introduced some options for compactor concurrency (#66) 2021-12-02 10:34:52 +01:00
symbols_batch_test.go Introduced some options for compactor concurrency (#66) 2021-12-02 10:34:52 +01:00
tsdbblockutil.go TSDB: demistify SeriesRefs and ChunkRefs (#9536) 2021-11-06 15:40:04 +05:30
wal.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
wal_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Due to an issue with versioning, the "latest" docs shown on Godoc are outdated. Instead you may use the docs for v2.31.1

Documentation

External resources

A series of blog posts explaining different components of TSDB: