prometheus/tsdb
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
..
chunkenc Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
chunks Stop the bleed on chunk mapper panic (#8723) 2021-05-06 14:18:59 -06:00
docs/format Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
encoding Avoid WriteAt for Postings. 2019-12-16 15:30:49 +00:00
errors MultiError: Refactored MultiError for more concise and safe usage. (#8066) 2020-10-28 15:24:58 +00:00
fileutil More info in mmap error message (#8058) 2020-10-15 12:58:13 +05:30
goversion tsdb: test for leaks (#7566) 2020-07-21 10:08:06 +02:00
index Add matchers to LabelValues() call (#8400) 2021-02-09 23:08:35 +05:30
record Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
test labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) 2020-10-15 11:31:28 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
tsdbutil Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
wal Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go Rolling tombstones clean up (#8007) 2021-02-17 11:02:43 +05:30
block_test.go Combine Appender.Add and AddFast into a single Append method. (#8489) 2021-02-18 17:37:00 +05:30
blockwriter.go add changes per comments, fix tests 2021-03-20 12:38:30 -07:00
blockwriter_test.go fix failing blockwriter tests 2021-03-01 06:16:05 -08:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go Add support for smaller block chunk segment allocations (#8478) 2021-04-15 14:25:01 +05:30
compact_test.go Combine Appender.Add and AddFast into a single Append method. (#8489) 2021-02-18 17:37:00 +05:30
db.go Add support for smaller block chunk segment allocations (#8478) 2021-04-15 14:25:01 +05:30
db_test.go Stop the bleed on chunk mapper panic (#8723) 2021-05-06 14:18:59 -06:00
exemplar.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
exemplar_test.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
head.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
head_bench_test.go TSDB: optimize series creation on PreCreation() failure (#8620) 2021-03-18 20:53:50 +05:30
head_test.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
isolation.go Optimise lowWatermark in Isolation (#7332) 2020-06-03 20:09:05 +02:00
isolation_test.go Optimise lowWatermark in Isolation (#7332) 2020-06-03 20:09:05 +02:00
mocks_test.go Move away from testutil, refactor imports (#8087) 2020-10-22 11:00:08 +02:00
querier.go Correct spelling of "iterable" (#8713) 2021-04-12 21:43:42 +01:00
querier_bench_test.go Combine Appender.Add and AddFast into a single Append method. (#8489) 2021-02-18 17:37:00 +05:30
querier_test.go Combine Appender.Add and AddFast into a single Append method. (#8489) 2021-02-18 17:37:00 +05:30
README.md Link blog post on TSDB queries in tsdb/README.md (#8391) 2021-01-21 17:14:40 +05:30
repair.go MultiError: Refactored MultiError for more concise and safe usage. (#8066) 2020-10-28 15:24:58 +00:00
repair_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
tsdbblockutil.go Combine Appender.Add and AddFast into a single Append method. (#8489) 2021-02-18 17:37:00 +05:30
wal.go Update golangci-lint to 1.36.0 2021-02-04 08:53:33 -06:00
wal_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00

TSDB

GoDoc

This directory contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.

A series of blog posts explaining different components of TSDB: