prometheus/storage/remote
Paschalis Tsilias d1122e0743
Introduce TSDB changes for appending metadata to the WAL (#10972)
* Append metadata to the WAL

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove extra whitespace; Reword some docstrings and comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use RLock() for hasNewMetadata check

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use single byte for metric type in RefMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update proposed WAL format for single-byte type metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implementa MetadataAppender interface for the Agent

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of review comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Amend description of metadata in wal.md

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Correct key used to retrieve metadata from cache

When we're setting metadata entries in the scrapeCace, we're using the
p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and
use it as the cache key. When checking for cache entries though, we used
p.Series() as the key, which included the metric name _with_ its labels.
That meant that we were never actually hitting the cache. We're fixing
this by utiling the __name__ internal label for correctly getting the
cache entries after they've been set by setHelp(), setType() or
setUnit().

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Put feature behind a feature flag

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix AppendMetadata docstring

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reorder WAL format document

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse temporary buffer in Metadata encoder

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Only keep latest metadata for each refID during checkpointing

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test that's referencing decoding metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Avoid creating metadata block if no new metadata are present

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for corrupt metadata block and relevant record type

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix CR comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Extract logic about changing metadata in an anonymous function

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implement new proposed WAL format and amend relevant tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use 'const' for metadata field names

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Apply metadata to head memSeries in Commit, not in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add docstring and rename extracted helper in scrape.go

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for tsdb-related cases

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol1

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol2

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix Windows test by closing WAL reader files

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use switch instead of two if statements in metadata decoding

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments around TestMetadata* tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add code for replaying WAL; test correctness of in-memory data after a replay

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove scrape-loop related code from PR

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify tests by sorting slices before comparison

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test to use separate transactions

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Empty out buffer and record slices after encoding latest metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linting issue

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update calculation for DroppedMetadata metric

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse buffer when encoding latest metadata for each series

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments; Check all returned error values using two helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify use of helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Satisfy linter

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-07-19 10:58:52 +02:00
..
chunked.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
chunked_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
client.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
client_test.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
codec.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
codec_test.go storage: expose bug in iterators #10027 2021-12-16 12:02:15 +01:00
ewma.go Replace sync/atomic with uber-go/atomic (#7683) 2020-07-30 13:15:42 +05:30
intern.go Move away from testutil, refactor imports (#8087) 2020-10-22 11:00:08 +02:00
intern_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
max_timestamp.go Remote: Do not collect non-initialized timestamp metrics (#8060) 2020-10-15 23:53:59 +02:00
metadata_watcher.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
metadata_watcher_test.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
queue_manager.go Remote Write: Rename confusing walDir parameter to dir (#10464) 2022-05-30 21:45:30 -07:00
queue_manager_test.go Removing global state modification on unit tests (fix #10033 #10034) (#10935) 2022-06-30 10:20:16 -06:00
read.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
read_handler.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
read_handler_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
read_test.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
storage.go Add an option to use the external labels as selectors for the remote read endpoint (#10254) 2022-02-16 22:12:47 +01:00
storage_test.go Removing global state modification on unit tests (fix #10033 #10034) (#10935) 2022-06-30 10:20:16 -06:00
write.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
write_handler.go (storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946) 2022-07-01 18:59:50 +02:00
write_handler_test.go Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
write_test.go Removing global state modification on unit tests (fix #10033 #10034) (#10935) 2022-06-30 10:20:16 -06:00