mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-10 07:34:04 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Jesus Vazquez 6cfe44d7fd
WaitUntilIdle optimize idling time (#10878) 
Relates to @bboreham optimization in https://github.com/prometheus/prometheus/pull/10859

Bryan did reduce the sleep time improving the deltas on the benchmark by
quite a lot. However I've been working on a similar implementation for
out of order and I noticed that we actually get into this method
thousands of times.

@ywwg had the brilliant idea of not always sleeping before the select
but actually make it a case in the select so we only sleep if we need
to.

The benchmark deltas are amazing

```
❯ benchstat old_implementation.txt new_implementation_using_time_after.txt
name                                                                                                     old time/op  new time/op  delta
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0,mmappedChunkT=0-8        521ms ±25%   253ms ± 6%  -51.47%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=36,mmappedChunkT=0-8       773ms ± 3%   369ms ±31%  -52.23%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=72,mmappedChunkT=0-8       592ms ±28%   297ms ±28%  -49.80%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=360,mmappedChunkT=0-8      547ms ± 2%  999ms ±187%     ~     (p=0.690 n=5+5)
LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0,mmappedChunkT=0-8        11.3s ± 4%    1.3s ±44%  -88.48%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=2,mmappedChunkT=0-8        11.1s ± 1%    1.2s ±20%  -89.08%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=0-8        1.24s ± 3%   0.18s ± 7%  -85.76%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=0-8        1.24s ± 2%   0.18s ± 5%  -85.24%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=0-8        1.23s ± 5%   0.27s ±33%  -77.73%  (p=0.008 n=5+5)
LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=0-8       1.28s ± 1%   0.36s ± 7%  -71.51%  (p=0.008 n=5+5)
LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=3800-8    12.1s ± 1%    3.1s ± 6%  -74.33%  (p=0.008 n=5+5)
LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=3800-8    12.1s ± 1%    3.4s ± 4%  -71.94%  (p=0.008 n=5+5)
LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=3800-8    12.1s ± 1%    3.8s ±17%  -68.35%  (p=0.008 n=5+5)
LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=3800-8   12.4s ± 1%    4.0s ±18%  -67.71%  (p=0.008 n=5+5)
```

Benchmarked on Linux
```
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/tsdb
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
```

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2022-06-30 15:00:04 +02:00
..
agent unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
chunkenc Add SyncForState Implementation for Ruler HA (#10070) 2022-03-29 02:16:46 +02:00
chunks Reduce chunk write queue memory usage 2 (#10874) 2022-06-29 17:51:27 +05:30
docs Update godoc links and remove note about TSDB versioning (#10754) 2022-05-26 18:34:43 +10:00
encoding Format Go source files using 'gofumpt -w -s -extra' 2021-11-02 19:52:34 +01:00
errors Address golint failures from revive 2021-10-23 00:53:11 +02:00
fileutil Make necessary change to compile promql parser to wasm (#10683) 2022-05-12 09:12:05 +02:00
goversion Add new Go build tags. 2021-08-27 10:24:14 +02:00
index Fix typo in symbol table size exceeded error message (#10746) 2022-05-25 10:40:36 +02:00
record Add basic initial developer docs for TSDB (#9451) 2021-11-17 15:51:27 +05:30
test Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
tsdbutil refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
wal Remote Write: Rename confusing walDir parameter to dir (#10464) 2022-05-30 21:45:30 -07:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go fix typo (#10696) 2022-05-25 18:01:45 +02:00
block_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
blockwriter.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
blockwriter_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108) 2022-01-05 15:10:00 +05:30
compact_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
db.go Don't increment prometheus_tsdb_compactions_failed_total when context is canceled (#10772) 2022-06-17 11:21:43 +05:30
db_test.go test(tsdb): break up repeated test to avoid timeout (#10671) 2022-05-09 00:39:26 +02:00
example_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
exemplar.go tsdb: use simpler map key to improve exemplar ingest performance (#10111) 2022-01-06 15:58:58 +05:30
exemplar_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
head.go Fix panic if series is not found when deleting series 2022-06-24 15:55:32 +08:00
head_append.go Fix chunk overflow appending samples at a variable rate (#10607) 2022-04-20 14:54:20 +02:00
head_bench_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
head_read.go Label values with matchers by intersecting postings (#9907) 2021-12-28 15:59:03 +01:00
head_test.go unit tests: make all Labels sorted alphabetically (#10532) 2022-05-04 23:41:36 +02:00
head_wal.go WaitUntilIdle optimize idling time (#10878) 2022-06-30 15:00:04 +02:00
isolation.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
isolation_test.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
mocks_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
querier.go Label values with matchers by intersecting postings (#9907) 2021-12-28 15:59:03 +01:00
querier_bench_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
querier_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
README.md Update godoc links and remove note about TSDB versioning (#10754) 2022-05-26 18:34:43 +10:00
repair.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
repair_test.go refactor: use T.TempDir() and B.TempDir to create temporary directory 2022-01-22 18:57:30 +08:00
tsdbblockutil.go tsdb: commit data periodically in CreateBlock (#10788) 2022-06-17 11:26:19 +05:30
wal.go fix potential goroutine leaks 2021-12-17 18:35:30 -05:00
wal_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00

README.md

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A series of blog posts explaining different components of TSDB: