mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-12-25 05:34:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
973 commits 243 branches 444 tags 649 MiB
Commit graph

20 commits

Author SHA1 Message Date
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev 2ae0620205 rename some vars and use Gauge instead of Counter for metrics 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-07 10:09:42 +02:00
Krasi Georgiev 00d5e19baf Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 13:47:50 +02:00
Krasi Georgiev a44d15798e
remove a changelog double entry (#507) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-23 17:10:19 +03:00
Brian Brazil 5db162568b Remove _total from prometheus_tsdb_storage_blocks_bytes (#506) 
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-01-23 16:46:58 +03:00
Krasi Georgiev 0b31b8af88 Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-18 16:58:18 +02:00
Krasi Georgiev 10ba228e6b
release 0.4.0 (#501) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-18 11:42:59 +03:00
Ganesh Vernekar 1a9d08adc5 Don't write empty blocks (#374) 
* Dont write empty blocks when a compaction results in a block with no samples.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-18 11:35:16 +03:00
Callum Styan 3929359302
add live reader for WAL (#481) 
* add live reader for WAL

Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2019-01-16 10:09:08 -08:00
mknapphrt ebf5d74325 Added storage size based retention method and new metrics (#343) 
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.

2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
 2019-01-16 13:03:52 +03:00
Krasi Georgiev 090b6852e1
remove unused PrefixMatcher (#474) 
* remove unused `PrefixMatcher`

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-28 21:13:02 +03:00
Krasi Georgiev 2e0571caba
remove unused WALFlushInterval option and NopWAL struct (#468) 
The WALFlushInterval is not used anywhere in the code base.
The WAL is not an interface anymore to save some lookup time so can't use NopWAL in the tests. Instead can just pass nil as the code checks for that and it is essentially a noop.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-28 20:42:46 +03:00
Brian Brazil 915d7cf937
Add a tsdbutil command to analyse churn etc. (#478) 
This reports the cardinality of each label,
the total number of label pairs,
and how much series worth of time is "uncovered"
by series data. Which is basically how much churn there is.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2018-12-28 17:06:12 +00:00
glutamatt 22e3aeb107 Add WALSegmentSize as an option of tsdb creation (#450) 
Expose `WALSegmentSize` option to allow overriding the `DefaultOptions.WALSegmentSize`.
 2018-12-18 21:56:51 +03:00
Krasi Georgiev 79aa611d56
release 0.3.1 (#477) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-13 17:02:32 +03:00
Krasi Georgiev db127a60e0 changelog 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-10 12:48:30 +02:00
Krasi Georgiev bd5ccee5c1 use context to cancel compactions 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-10 12:48:30 +02:00
Krasi Georgiev b4a2103a12
Release v0.3.0 (#464) 2018-12-04 16:50:11 +03:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461) 
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-04 13:30:49 +03:00
Krasi Georgiev 48efdf8b81
refactor NewSegmentsRangeReader to take multi WAL ranges (#449) 
* refactor NewSegmentsRangeReader to take multi WAL ranges

In case of an error when checkpointing the WAL the error doesn't show
the exact WAL index that is corrupter. this is because it uses
MultiReader to read multiply WAL files.
This refactoring allows the NewSegmentsRangeReader to take more than a
single WAL range and it reads all of the ranges by iterating each one.

this changes the logs from
create checkpoint: read segments: corruption after 4841144384 bytes:...
to
create checkpoint: read segments: corruption in segment
data/wal/00017351 at 123142208: ...

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-30 16:46:16 +02:00