mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-09-20 07:47:31 -07:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Brian Brazil 971dafdfbe Coalesce series reads where we can. 
When compacting rather than doing a read of all
series in the index per label name, do many at once
but only when it won't use (much) more ram than writing the
special all index does.

original in-memory postings:
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                  1        1202383447 ns/op        158936496 B/op   1031511 allocs/op
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                  1        1141792706 ns/op        154453408 B/op   1093453 allocs/op
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                  1        1169288829 ns/op        161072336 B/op   1110021 allocs/op
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                  1        1115700103 ns/op        149480472 B/op   1129180 allocs/op
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                  1        1283813141 ns/op        162937800 B/op   1202771 allocs/op

before:
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                  1        1145195941 ns/op        131749984 B/op    834400 allocs/op
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                  1        1233526345 ns/op        127889416 B/op    897033 allocs/op
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                  1        1821942296 ns/op        131665648 B/op    914836 allocs/op
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                  1        8035568665 ns/op        123811832 B/op    934312 allocs/op
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                  1       71325926267 ns/op        140722648 B/op   1016824 allocs/op

after:
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                  1        1101429174 ns/op        129063496 B/op    832571 allocs/op
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                  1        1074466374 ns/op        124154888 B/op    894875 allocs/op
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                  1        1166510282 ns/op        128790648 B/op    912931 allocs/op
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                  1        1075013071 ns/op        120570696 B/op    933511 allocs/op
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                  1        1231673790 ns/op        138754288 B/op   1022791 allocs/op

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-12 08:38:14 +00:00
..
chunkenc Correct spelling mistakes 2019-10-10 18:46:27 +08:00
chunks return err instead of panic for corrupted chunk (#6040) 2019-12-04 09:37:49 +02:00
cmd/tsdb Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
docs/format Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
encoding Coalesce series reads where we can. 2019-12-12 08:38:14 +00:00
errors Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
fileutil Compute WAL size and use it during retention size checks (#5886) 2019-11-12 09:40:16 +07:00
goversion fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
index Coalesce series reads where we can. 2019-12-12 08:38:14 +00:00
record Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
test Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
testdata Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
tombstones remove debug fmt.Println in tombstones. (#6135) 2019-10-14 14:45:26 +03:00
tsdbutil Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
wal Simplify duration check for watcher WAL replay. 2019-11-26 16:53:11 -08:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go Reread index series rather than storing in memory. 2019-12-11 22:23:39 +00:00
block_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
CHANGELOG.md fix-up tsdb-typo (#5954) 2019-08-28 14:43:02 +01:00
compact.go Reread index series rather than storing in memory. 2019-12-11 22:23:39 +00:00
compact_test.go Coalesce series reads where we can. 2019-12-12 08:38:14 +00:00
db.go tsdb: add error details in log. (#6415) 2019-12-09 10:37:01 +00:00
db_test.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
head.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
head_bench_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
head_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
mocks_test.go Coalesce series reads where we can. 2019-12-12 08:38:14 +00:00
querier.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
querier_bench_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
querier_test.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
README.md Edit TSDB README badges 2019-10-24 15:35:47 +05:30
repair.go tsdb: add block meta version constant (#5994) 2019-09-09 12:28:01 +03:00
repair_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
tsdbblockutil.go Added CreateBlock and CreateHead functions to new file (#6331) 2019-11-21 19:10:25 +07:00
wal.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
wal_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00

README.md

TSDB

GoDoc

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.