mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-12 16:44:05 -08:00
Code Issues Actions 7 Packages Projects Releases Wiki Activity
1041 commits 247 branches 438 tags 655 MiB
Commit graph

97 commits

Author SHA1 Message Date
Brian Brazil 6ac81cc7a9
Correctly handle empty labels. (#594) 
Currently a time series with empty labels is not treated the same
as one with missing labels. Currently this can only come from
ALERTS&ALERT_FOR_STATE so it's unlikely anyone has actually hit it.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-05-07 11:00:16 +01:00
naivewong 7ab060c864 selectOverlappingDirs selects wrong dirs where there are 2 disjoint sets of overlapping dirs 
Signed-off-by: naivewong <867245430@qq.com>
 2019-04-08 15:27:06 +03:00
Vishnunarayan K I 7757fe6f21 Added ability to create db with NopWal (#519) 
Signed-off-by: Vishnunarayan K I <appukuttancr@gmail.com>
 2019-03-26 01:38:12 +02:00
Krasi Georgiev c3ffdf1a99
Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 
Testing that createBlock creates blocks that can be opened.

and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.

Many missing .Close() calls were added for fixing failing os.RemoveAll

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-03-19 15:31:57 +02:00
Krasi Georgiev ab8476dd06
Release 0.6.0 (#537) 2019-03-02 15:54:49 +02:00
Ganesh Vernekar 28c73f531f
Make vertical compaction and query merge optional 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-26 11:50:50 -08:00
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev 07df4fd383 nits 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-11 11:25:57 +02:00
Krasi Georgiev e138c7ed7e Merge remote-tracking branch 'upstream/master' into delete-compact-block-on-reload-error 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-08 13:26:28 +02:00
Krasi Georgiev 0b72f9af4c
Merge pull request #270 from codesome/master 
Head: don't create stones, delete samples directly
 2019-02-08 12:35:01 +02:00
Ganesh Vernekar 5481549324
Merge remote-tracking branch 'upstream/master' 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-07 22:15:06 +05:30
Simon Pasquier 95334f13c5 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-02-07 12:10:22 +01:00
Krasi Georgiev 1603222bbc small refactor of openTestDB to handle errors properly. 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 11:40:12 +02:00
Krasi Georgiev 7245c6dc33 Merge remote-tracking branch 'upstream/master' into delete-compact-block-on-reload-error 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 11:39:19 +02:00
Krasi Georgiev 3ec08eac50 use camelcase for rangeToTriggerCompaction 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-29 10:32:59 +02:00
yeya24 6181d1f18f fix two typos in db_test 
Signed-off-by: yeya24 <ben.ye@daocloud.io>
 2019-01-29 10:25:12 +08:00
Krasi Georgiev dac2b97dfd
make createBlock more generic so it can be used in other tests. (#489) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-28 14:24:49 +03:00
Simon Pasquier 2e69508536 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-22 10:47:31 +01:00
Ganesh Vernekar 1a9d08adc5 Don't write empty blocks (#374) 
* Dont write empty blocks when a compaction results in a block with no samples.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-18 11:35:16 +03:00
mknapphrt ebf5d74325 Added storage size based retention method and new metrics (#343) 
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.

2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
 2019-01-16 13:03:52 +03:00
Ganesh Vernekar d7e505db34
Dont store stones in head, delete samples directly 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-08 22:38:41 +05:30
Simon Pasquier d5d7a097e1 Update Makefile.common 
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-04 16:08:43 +01:00
Krasi Georgiev 48c439d26d
fix statick check errors (#475) 
fix the tests for `check_license` and `staticcheck`

the static check also found some actual bugs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-02 19:48:42 +03:00
Krasi Georgiev eb6586f513
rename createPopulatedBlock to createBlock and use it instead ot createEmptyBlock (#488) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-29 14:20:51 +03:00
Krasi Georgiev 6d489a1004
fix TestWALSegmentSizeOption for windows. (#482) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-19 12:40:47 +03:00
glutamatt 22e3aeb107 Add WALSegmentSize as an option of tsdb creation (#450) 
Expose `WALSegmentSize` option to allow overriding the `DefaultOptions.WALSegmentSize`.
 2018-12-18 21:56:51 +03:00
Krasi Georgiev a2779cc901
fix flaky tests: TestDisableAutoCompactions,TestBlockRanges (#472) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-12 14:49:03 +03:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461) 
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-04 13:30:49 +03:00
Brian Brazil d50b9a5619
Reload after reading the WAL. (#460) 
This causes the head to be GCed at startup,
removing any series that were read from the WAL
but have since been written to a block. In
systems with low ingestion rates, this potentially
could be many many hours of data.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2018-11-28 09:23:50 +00:00
Ganesh Vernekar 7f30395115 LabelNames() for Querier (#455) 
* LabelNames() for Querier

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-11-16 19:02:24 +01:00
Brian Brazil 41b54585d9
Use already open blocks while compacting. (#441) 
This roughly halves the RAM requirements of compaction.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2018-11-15 12:20:54 +00:00
Krasi Georgiev 5a9ddeecef
fix lint errors (#439) 
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-14 18:40:01 +02:00
Ganesh Vernekar 3a08a71d86 LabelNames() method to get all unique label names (#369) 
* LabelNames() method to get all unique label names

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-11-07 17:52:41 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419) 
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-10-25 21:06:19 +01:00
Ganesh Vernekar 61d0868966 Fix TestCorrectNumTombstones (#399) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-28 13:26:29 +03:00
Ganesh Vernekar 6e712963e2 Fix updating of NumTombstones in block.Delete(..) (#385) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-27 14:43:22 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-21 11:07:35 +03:00
Krasi Georgiev d38516b1c2
remove unused changes variable (#391) 
This was added in
55a9b5428a

and later not used after some refactoring in following PRs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-21 09:24:01 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2018-09-21 09:23:01 +03:00
Goutham Veeramachaneni 9c8ca47399
Fix filehandling for windows (#392) 
* Fix filehandling for windows

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix more windows filehandling issues

Windows: Close files before deleting Checkpoints.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close writers in case of errors so they can be deleted

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close block so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close file to delete it

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close dir so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: close files so that they can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2018-09-21 11:01:22 +05:30
Fabian Reinartz 45071c657c Properly initialize head time 
This fixes various issues when initializing the head time range
under different starting conditions.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-07-19 07:41:02 -04:00
Fabian Reinartz 7841d417b3 Ensure blocks are time-ordered in memory 
We assume in multiple places that the block list held by DB
has blocks sequential by time.
A regression caused us to hold them ordered by ULID, i.e. by creation
time instead.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-07-19 07:25:30 -04:00
codwu 667e539a7a Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete 2018-07-06 20:21:32 +08:00
Benoît Knecht 24b223c161 db: add test for Querier returning too many blocks 
Due to the way blocks used to overlap by 1 millisecond (see #347), when
requesting a 2-hour interval starting at `blocks[1].MinTime`, the
`Querier` would consider three blocks: `blocks[0]`, `blocks[1]` and
`blocks[2]`, because `blocks[0].MaxTime` and `blocks[2].MinTime` were in
that interval.

However, if the blocks don't overlap, only two blocks should be
returned: `blocks[1]` and `blocks[2]`. This test ensures that it's
indeed the case.

Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
 2018-07-02 10:35:21 +02:00
Benoît Knecht 4ed6b9ed72 db: add test for chunks that span beyond a block's boundaries 
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
 2018-07-02 10:32:05 +02:00
Fabian Reinartz af9003dcef Add resilience to crashes during deletion 
Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-06-27 09:50:31 -04:00
codwu cd145c90d5 remove put function and use RLock in Iter function 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-25 21:52:11 +08:00
codwu 84a45cb79a add rwmutex to prevent concurrent map read when delete series 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-08 19:52:01 +08:00
Krasi Georgiev 6094f35aa2 simplify if-else,test before the tombstone failure, more comments 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-06-04 20:18:44 +01:00
Krasi Georgiev f31a0d6457 add Test for Tombstone Cleaning after a failure 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-06-04 18:34:30 +01:00