mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-12-27 14:39:40 -08:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity
1076 commits 243 branches 444 tags 649 MiB
Commit graph

55 commits

Author SHA1 Message Date
Ganesh Vernekar b1cd829030
Reuse Chunk Iterator (#642) 
* Reset method for chunkenc.Iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reset method only for XORIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Use Reset(...) in querier.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reuse deletedIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Another way of reusing chunk iterators

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Unexport xorIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix memSeries.iterator(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add some comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-07-09 15:19:34 +05:30
Krasi Georgiev 31f7990d1d
Re-encode chunks that are still being appended to when snapshoti… (#641) 
* re encode all head chunks outside the time range.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
 2019-07-03 13:47:31 +03:00
Krasi Georgiev 8eeb70fee1
remove Fsync workaround for macos. (#574) 
since golang 1.12 no special handling is required for file.Sync()

@pborzenkov thanks for the pointer.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-04-03 11:16:54 +03:00
Pavel Borzenkov 4f204dcbc1 chunks: close all opened MmapFiles on NewDirReader error path (#575) 
Signed-off-by: Pavel Borzenkov <pavel.borzenkov@gmail.com>
 2019-04-02 15:16:29 +03:00
zhulongcheng e11e01f68d Fix exported function comments (#566) 
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
 2019-03-25 10:17:28 +02:00
zhulongcheng 4d03c70800 Verify chunk format version (#544) 
Verify chunk format version.
 2019-03-24 22:33:08 +02:00
zhulongcheng b7cac504c9 Remove unused NewReader function (#547) 
The `NewReader` function is not used anywhere.
 2019-03-14 15:05:40 +02:00
Alec 0861a9b14f [bugfix] update 'last' variable in chunks.MergeOverlappingChunks() (#539) 
* update

Signed-off-by: naivewong <867245430@qq.com>

* update comment

Signed-off-by: naivewong <867245430@qq.com>

* Update failing test.

Signed-off-by: naivewong <867245430@qq.com>

* Update comment

Signed-off-by: naivewong <867245430@qq.com>

* Add changelog entry.

Signed-off-by: naivewong <867245430@qq.com>

* Update CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-03-05 19:18:55 +05:30
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev bf2239079d refactor multi errors 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-11 12:28:46 +02:00
Simon Pasquier 95334f13c5 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-02-07 12:10:22 +01:00
Krasi Georgiev ee99718ff6
rename chunk reader vars to make it easier to follow. (#508) 
* rename chunk reader vars to make it easyer to understand.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-29 20:46:12 +03:00
Alec 051a7ae1a7 Missing the length of the encoding byte when calling b.Range 
Signed-off-by: naivewong <867245430@qq.com>
 2019-01-28 13:33:44 +03:00
Simon Pasquier 2e69508536 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-22 10:47:31 +01:00
mknapphrt ebf5d74325 Added storage size based retention method and new metrics (#343) 
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.

2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
 2019-01-16 13:03:52 +03:00
naivewong bff5aa4d21 Missing the len of crc32 when calculating maxLen in WriteChunks (#494) 
Signed-off-by: naivewong <867245430@qq.com>
 2019-01-14 20:58:03 +05:30
Simon Pasquier d5d7a097e1 Update Makefile.common 
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-04 16:08:43 +01:00
Benoît Knecht 1e1b2e163d Make interval overlap comparisons more explicit 
Blocks are half-open intervals [a, b), while all other intervals
(chunks, head, ...) are closed intervals [a, b].

Make that distinction explicit by defining `OverlapsClosedInterval()`
methods for blocks and chunks, and using them in place of the more
generic `intervalOverlap()` function.

This change also fixes `db.Querier()` and `db.Delete()`, which could
previously return one extraneous block at the end of the specified
interval.

Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
 2018-07-02 10:35:08 +02:00
Simon Pasquier f55ccd4ecb Add unit tests 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2018-06-25 11:25:22 +02:00
Simon Pasquier ee5fe8ea9f chunks: fix potential "index out of range" error 
When binary.Uvarint() fails, the returned number of bytes is less than
or equal to zero.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2018-06-08 10:28:27 +02:00
Mario Trangoni c0e888e82b fix megacheck issues: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd. 2018-03-21 22:39:15 +01:00
Mario Trangoni 09142e4dd1 fix unconvert issues: unnecessary conversion 2018-03-21 22:39:14 +01:00
Fabian Reinartz 1e55b7987f Improve comments, handle allPostingsKey properly 2017-12-22 09:43:34 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Callum Styan ff0b0ac4b6 use test utils in chunk_test 2017-12-06 19:05:58 -08:00
Daniel Swarbrick 0585ab3fcd Use Go 1.9 math/bits implementation for counting leading / trailing zeros 2017-11-08 23:18:33 +01:00
Fabian Reinartz ea817e169b Return nop iterator for invalid chunk references 2017-10-20 09:43:52 +02:00
Fabian Reinartz 905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz 66ff7b12e9 Pool Chunk objects during compaction 2017-08-08 17:35:34 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set 
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
 2017-08-06 12:06:41 +02:00
Goutham Veeramachaneni 401bdfd4a6 Make Chunks return the number of samples inside 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-12 18:45:12 +02:00
Julius Volz 0e2f8917ed Also add original license info to bstream.go 
Plus, extend the messages a bit to mention the origin repo.
 2017-05-03 01:32:09 +02:00
Fabian Reinartz 87676f4a26 Add dgryski/go-tsz LICENSE to header 2017-05-02 11:26:01 +02:00
Julius Volz 8d1fb4fa01 Minor comment fixes and additions. 2017-04-28 15:41:42 +02:00
Fabian Reinartz 778103b450 Add liecence file and headers 2017-04-10 20:59:45 +02:00
Goutham Veeramachaneni 761e4768f3
Lint and Vet Fixes 2017-03-19 21:35:01 +05:30
Fabian Reinartz 91b65b55e7 Run persistence in separate goroutine 2017-01-02 22:24:35 +01:00
Fabian Reinartz 62b8ded0a5 Rename iterator value getters to At() 2017-01-02 13:33:06 +01:00
Fabian Reinartz 5d75a3dc7b Add basic compaction 
This adds a basic compactor that will merge two persisted blocks into
one. It simply fully rewrites the index and concatenates the chunk
lists.

It just writes into the current working dir and doesn't properly handle
which blocks to compact for now.
 2017-01-02 10:34:55 +01:00
Fabian Reinartz 675f0886f0 Append to chunks cannot error 2016-12-31 10:10:27 +01:00
Fabian Reinartz ce7f4106c2 Reda correct label number, fix buffered iterator panic 2016-12-20 14:21:50 +01:00
Fabian Reinartz 70a0224f19 Change chunk sample number to BigEndian 2016-12-12 08:11:53 +01:00
Fabian Reinartz 40a451694f Refactor persistence into interfaces 2016-12-09 20:45:46 +01:00
Fabian Reinartz b845f8d3a1 Reduce test data allocations 2016-12-08 11:59:54 +01:00
Fabian Reinartz c5945177fb chunks: helper for bit range 2016-12-07 15:37:37 +01:00
Fabian Reinartz 9b459458d0 Docs and interface definitions 2016-12-05 21:26:19 +01:00
Fabian Reinartz 0b6d621471 chunks: bring back lead/trail reuse, truncate incomplete sample 2016-11-30 23:01:01 +01:00
Fabian Reinartz 8c48dc2ca5 chunks: varint encoding in first full 64bit numbers 
This saves about 7 bytes per chunk
 2016-11-30 22:14:23 +01:00
Fabian Reinartz 5e76fd3126 chunks: drop shift, remove branch 
This removes the need for the shift value and just calculates
everything on the fly through count.
Removes a branch condition.
 2016-11-30 21:50:39 +01:00
Fabian Reinartz c9ee572709 chunks: don't mutate in bstream iterator 
This replaces mutation of underlying bytes in the iterated slice
with a shift counter, which is used when reading the head byte.
This is avoids having to copy the entire slice for every new iterator.
 2016-11-30 19:39:22 +01:00