mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-10 07:34:04 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
1054 commits 245 branches 436 tags 650 MiB
Commit graph

130 commits

Author SHA1 Message Date
zhulongcheng 19d402d154 cleanup tmp files (#570) 
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
 2019-04-26 11:27:36 +03:00
Krasi Georgiev e46ec89de6
use fsync to force persisting the meta file. (#573) 
* use fsync to force persisting the meta file.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-04-01 16:23:45 +03:00
Krasi Georgiev c3ffdf1a99
Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 
Testing that createBlock creates blocks that can be opened.

and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.

Many missing .Close() calls were added for fixing failing os.RemoveAll

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-03-19 15:31:57 +02:00
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev bf2239079d refactor multi errors 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-11 12:28:46 +02:00
Krasi Georgiev d48606827c simplify closers 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-08 13:35:32 +02:00
Krasi Georgiev 315de4c782 fix windows tests 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 11:40:40 +02:00
Krasi Georgiev 53c18e7a41 use a global indexFilename constant 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-29 10:32:32 +02:00
Ganesh Vernekar 1a9d08adc5 Don't write empty blocks (#374) 
* Dont write empty blocks when a compaction results in a block with no samples.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-18 11:35:16 +03:00
mknapphrt ebf5d74325 Added storage size based retention method and new metrics (#343) 
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.

2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
 2019-01-16 13:03:52 +03:00
Brian Brazil c93e261466 Reduce memory taken up by posting/symbol tables. 
Reuse the string already allocated for symbols
in the posting tables.

Use a slice for symbols in v2 format.

Move symbol size logic into the index code.
Avoid duplication of lookupSymbol logic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2018-11-22 09:36:39 +00:00
Krasi Georgiev 5a9ddeecef
fix lint errors (#439) 
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-14 18:40:01 +02:00
Ganesh Vernekar 3a08a71d86 LabelNames() method to get all unique label names (#369) 
* LabelNames() method to get all unique label names

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-11-07 17:52:41 +02:00
Ganesh Vernekar 6e712963e2 Fix updating of NumTombstones in block.Delete(..) (#385) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-27 14:43:22 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-21 11:07:35 +03:00
Julius Volz 5ae6c60d39 Handle a bunch of unchecked errors (#365) 
As discovered by "gosec".

Signed-off-by: Julius Volz <julius.volz@gmail.com>
 2018-09-20 11:33:52 +03:00
Ganesh Vernekar 77d2c7cf3d
Added metric for symbol table size 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-08 23:58:36 +05:30
codwu 667e539a7a Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete 2018-07-06 20:21:32 +08:00
Benoît Knecht 1e1b2e163d Make interval overlap comparisons more explicit 
Blocks are half-open intervals [a, b), while all other intervals
(chunks, head, ...) are closed intervals [a, b].

Make that distinction explicit by defining `OverlapsClosedInterval()`
methods for blocks and chunks, and using them in place of the more
generic `intervalOverlap()` function.

This change also fixes `db.Querier()` and `db.Delete()`, which could
previously return one extraneous block at the end of the specified
interval.

Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
 2018-07-02 10:35:08 +02:00
Fabian Reinartz d907928e57 Clarify docs, error on unexpected meta read errors 
Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-06-27 12:05:21 -04:00
Fabian Reinartz af9003dcef Add resilience to crashes during deletion 
Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-06-27 09:50:31 -04:00
codwu cd145c90d5 remove put function and use RLock in Iter function 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-25 21:52:11 +08:00
codwu 84a45cb79a add rwmutex to prevent concurrent map read when delete series 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-08 19:52:01 +08:00
Ganesh Vernekar 528439aa93 Cleanup new blocks on 'CleanTombstones' faliure. 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-06-01 15:48:38 -04:00
Goutham Veeramachaneni f82a1fe4f2
Simplify stones counting. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2018-04-03 18:17:02 +05:30
Goutham Veeramachaneni 8a301b126a
Merge pull request #307 from mjtrangoni/fixes 
Fix some megacheck and unconvert issues
 2018-04-03 12:27:27 +05:30
Mario Trangoni c2182820ed fix megacheck issue: should omit values from range 2018-03-21 22:39:15 +01:00
Simon Pasquier 79defa54df Fix crash when a series has no block 2018-02-21 16:45:06 +01:00
Fabian Reinartz d09283f60a Add repair of bad index version 2018-02-09 13:11:03 +01:00
Goutham Veeramachaneni 2c3400ab4e Add block ULID to index reader errors 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2018-01-16 23:48:58 +05:30
Shubheksha Jalan 7b27dc2109 address review comments, fix index tests 2018-01-11 13:20:42 +05:30
Shubheksha Jalan 44052bc937 Make version for index format configurable 2018-01-11 00:49:16 +05:30
Shubheksha Jalan 129773b41a merge BlockMeta and blockMeta struct, implement v1 and v2 index readers 2018-01-08 23:03:35 +05:30
Fabian Reinartz 1e55b7987f Improve comments, handle allPostingsKey properly 2017-12-22 09:43:34 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Goutham Veeramachaneni 3158b03e6c Select series with label unset for != and !~ 
Fixes https://github.com/prometheus/prometheus/issues/3575

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2017-12-19 02:47:31 +00:00
Goutham Veeramachaneni 8a5ea9db74
Merge branch 'master' into tomb-clean 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-30 13:15:23 +05:30
Goutham Veeramachaneni 24ff293dc9
Merge pull request #207 from Gouthamve/compact-fail 
Don't retry failed compactions.
 2017-11-30 12:58:19 +05:30
Goutham Veeramachaneni e7445d00b0
Add a function to cleanup tombstones. 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-27 20:03:37 +05:30
Goutham Veeramachaneni 1fc94a02d1
Don't retry failed compactions. 
Fixes prometheus/prometheus#3487

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-23 15:16:20 +05:30
Fabian Reinartz f1512a368a Expose ChunkSeriesSet and lookups methods. 2017-11-13 14:02:32 +01:00
Fabian Reinartz 3ef4326114 Refactor tombstone reader types 2017-11-13 13:38:07 +01:00
Fabian Reinartz e5ce2bef43 Add explicit error to Querier.Select 
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
 2017-11-13 12:16:58 +01:00
Fabian Reinartz d578b10d5c chunk: make reader accept abstract ByteSlice 2017-11-10 10:38:22 +00:00
Fabian Reinartz b7c3cfecbf index: abstract ByteSlice and adjust indexReader 
This replaces the builtin byte slice with an interface for the index
reader. This allows the complex decoding of the index file format
to be used against more generalized implementations.
 2017-11-09 17:38:32 +00:00
Sunny Klair 4fdf9b195c Validate index TOC checksum on read 2017-10-25 18:12:13 -04:00
Fabian Reinartz fb9da52b11 Add more verbose error handling for closing, reduce locking 
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.

Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
 2017-10-10 12:13:37 +02:00
Fabian Reinartz 78df406dac Allocate and cache strings for persisted blocks 
This change loads the full symbol table when we open a persisted block
and allocates a string for each. This ensures that strings retrieved
through the index can be used after the block was closed.
Before we backed the strings by the mmap'd byte regions which would
segfault in this case.

Also remove an inconsistency in the disk format and move both offset
tables to the end (breaking change).
 2017-10-02 15:56:57 +02:00
Goutham Veeramachaneni 8919baef03
Expose NewIndexReader() and cleanups 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-09-13 13:47:20 +05:30
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00