mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2025-02-02 08:31:11 -08:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Paschalis Tsilias d1122e0743
Introduce TSDB changes for appending metadata to the WAL (#10972) 
* Append metadata to the WAL

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove extra whitespace; Reword some docstrings and comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use RLock() for hasNewMetadata check

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use single byte for metric type in RefMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update proposed WAL format for single-byte type metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implementa MetadataAppender interface for the Agent

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of review comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Amend description of metadata in wal.md

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Correct key used to retrieve metadata from cache

When we're setting metadata entries in the scrapeCace, we're using the
p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and
use it as the cache key. When checking for cache entries though, we used
p.Series() as the key, which included the metric name _with_ its labels.
That meant that we were never actually hitting the cache. We're fixing
this by utiling the __name__ internal label for correctly getting the
cache entries after they've been set by setHelp(), setType() or
setUnit().

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Put feature behind a feature flag

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix AppendMetadata docstring

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reorder WAL format document

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse temporary buffer in Metadata encoder

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Only keep latest metadata for each refID during checkpointing

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test that's referencing decoding metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Avoid creating metadata block if no new metadata are present

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for corrupt metadata block and relevant record type

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix CR comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Extract logic about changing metadata in an anonymous function

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implement new proposed WAL format and amend relevant tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use 'const' for metadata field names

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Apply metadata to head memSeries in Commit, not in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add docstring and rename extracted helper in scrape.go

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for tsdb-related cases

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol1

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol2

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix Windows test by closing WAL reader files

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use switch instead of two if statements in metadata decoding

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments around TestMetadata* tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add code for replaying WAL; test correctness of in-memory data after a replay

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove scrape-loop related code from PR

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify tests by sorting slices before comparison

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test to use separate transactions

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Empty out buffer and record slices after encoding latest metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linting issue

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update calculation for DroppedMetadata metric

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse buffer when encoding latest metadata for each series

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments; Check all returned error values using two helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify use of helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Satisfy linter

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-07-19 10:58:52 +02:00
..
agent Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
chunkenc Add SyncForState Implementation for Ruler HA (#10070) 2022-03-29 02:16:46 +02:00
chunks Reduce chunk write queue memory usage 2 (#10874) 2022-06-29 17:51:27 +05:30
docs Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
encoding Format Go source files using 'gofumpt -w -s -extra' 2021-11-02 19:52:34 +01:00
errors Address golint failures from revive 2021-10-23 00:53:11 +02:00
fileutil Make necessary change to compile promql parser to wasm (#10683) 2022-05-12 09:12:05 +02:00
goversion Add new Go build tags. 2021-08-27 10:24:14 +02:00
index Fix typo in symbol table size exceeded error message (#10746) 2022-05-25 10:40:36 +02:00
record Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
test Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
tsdbutil refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
wal Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go fix typo (#10696) 2022-05-25 18:01:45 +02:00
block_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
blockwriter.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
blockwriter_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108) 2022-01-05 15:10:00 +05:30
compact_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
db.go Don't increment prometheus_tsdb_compactions_failed_total when context is canceled (#10772) 2022-06-17 11:21:43 +05:30
db_test.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
example_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
exemplar.go tsdb: use simpler map key to improve exemplar ingest performance (#10111) 2022-01-06 15:58:58 +05:30
exemplar_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
head.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
head_append.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
head_bench_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
head_read.go Label values with matchers by intersecting postings (#9907) 2021-12-28 15:59:03 +01:00
head_test.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
head_wal.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
isolation.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
isolation_test.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
mocks_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
querier.go Label values with matchers by intersecting postings (#9907) 2021-12-28 15:59:03 +01:00
querier_bench_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
querier_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
README.md Update godoc links and remove note about TSDB versioning (#10754) 2022-05-26 18:34:43 +10:00
repair.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
repair_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
tsdbblockutil.go tsdb: commit data periodically in CreateBlock (#10788) 2022-06-17 11:26:19 +05:30
wal.go fix potential goroutine leaks 2021-12-17 18:35:30 -05:00
wal_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00

README.md

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A series of blog posts explaining different components of TSDB: