prometheus/tsdb
Robert Fratto bc72a718c4
Initial draft of prometheus-agent (#8785)
* Initial draft of prometheus-agent

This commit introduces a new binary, prometheus-agent, based on the
Grafana Agent code. It runs a WAL-only version of prometheus without the
TSDB, alerting, or rule evaluations. It is intended to be used to
remote_write to Prometheus or another remote_write receiver.

By default, prometheus-agent will listen on port 9095 to not collide
with the prometheus default of 9090.

Truncation of the WAL cooperates on a best-effort case with Remote
Write. Every time the WAL is truncated, the minimum timestamp of data to
truncate is determined by the lowest sent timestamp of all samples
across all remote_write endpoints. This gives loose guarantees that data
from the WAL will not try to be removed until the maximum sample
lifetime passes or remote_write starts functionining.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* add tests for Prometheus agent (#22)

* add tests for Prometheus agent

* add tests for Prometheus agent

* rearranged tests as per the review comments

* update tests for Agent

* changes as per code review comments

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* incremental changes to prometheus agent

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* changes as per code review comments

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* Commit feedback from code review

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Port over some comments from grafana/agent

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Rename agent.Storage to agent.DB for tsdb consistency

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Consolidate agentMode ifs in cmd/prometheus/main.go

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Document PreAction usage requirements better for agent mode flags

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* remove unnecessary defaultListenAddr

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* `go fmt ./tsdb/agent` and fix lint errors

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

Co-authored-by: SriKrishna Paparaju <paparaju@gmail.com>
2021-10-29 16:25:05 +01:00
..
agent Initial draft of prometheus-agent (#8785) 2021-10-29 16:25:05 +01:00
chunkenc improve bstream comments and doc (#9560) 2021-10-25 18:44:15 +05:30
chunks TSDB: Use a dedicated head chunk reference type (#9501) 2021-10-13 17:44:32 +05:30
docs improve bstream comments and doc (#9560) 2021-10-25 18:44:15 +05:30
encoding Snapshot in-memory chunks on shutdown for faster restarts (#7229) 2021-08-06 17:51:01 +01:00
errors Address golint failures from revive 2021-10-23 00:53:11 +02:00
fileutil Add new Go build tags. 2021-08-27 10:24:14 +02:00
goversion Add new Go build tags. 2021-08-27 10:24:14 +02:00
index Replace calls to strings.Compare (#9397) 2021-09-27 17:33:53 +05:30
record Exemplars in snapshot (#9255) 2021-08-30 19:34:38 +05:30
test labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) 2020-10-15 11:31:28 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones Call delete on head if interval overlaps (#9151) 2021-09-16 12:20:03 +05:30
tsdbutil storage: Split chunks if more than 120 samples (#8582) 2021-05-18 18:37:16 +02:00
wal Address golint failures from revive 2021-10-23 00:53:11 +02:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
block_test.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
blockwriter.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
blockwriter_test.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
compact_test.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
db.go Call delete on head if interval overlaps (#9151) 2021-09-16 12:20:03 +05:30
db_test.go fix: possible race on shared variables in test (#9470) 2021-10-25 18:44:40 +05:30
exemplar.go Address golint failures from revive 2021-10-23 00:53:11 +02:00
exemplar_test.go Fix div by 0 panic is resize function. (#9286) 2021-09-02 11:08:05 -07:00
head.go TSDB: Use a dedicated head chunk reference type (#9501) 2021-10-13 17:44:32 +05:30
head_append.go Fix remote write receiver endpoint for exemplars (#9414) 2021-10-21 22:58:40 +02:00
head_bench_test.go React UI: Add Starting Screen (#8662) 2021-06-05 15:29:32 +01:00
head_read.go Remove symbols map from TSDB head (#9301) 2021-09-08 14:48:48 +05:30
head_test.go close tsdb.head in test case (#9580) 2021-10-26 11:36:25 +05:30
head_wal.go Fix panic on failed snapshot replay and don't hard fail replay on disabled exemplars (#9438) 2021-10-05 10:51:25 +05:30
isolation.go Add present_over_time (#9097) 2021-07-29 12:38:11 +02:00
isolation_test.go tsdb: Block Head GC till pending readers are done reading (#9081) 2021-07-20 14:17:20 +05:30
mocks_test.go Move away from testutil, refactor imports (#8087) 2020-10-22 11:00:08 +02:00
querier.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
querier_bench_test.go React UI: Add Starting Screen (#8662) 2021-06-05 15:29:32 +01:00
querier_test.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
README.md improve bstream comments and doc (#9560) 2021-10-25 18:44:15 +05:30
repair.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
repair_test.go React UI: Add Starting Screen (#8662) 2021-06-05 15:29:32 +01:00
tsdbblockutil.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
wal.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
wal_test.go Add new Go build tags. 2021-08-27 10:24:14 +02:00

TSDB

GoPkg

This directory contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation and bstream details.

A series of blog posts explaining different components of TSDB: