mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-10 07:34:04 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
1075 commits 245 branches 436 tags 650 MiB
Commit graph

93 commits

Author SHA1 Message Date
Ganesh Vernekar b1cd829030
Reuse Chunk Iterator (#642) 
* Reset method for chunkenc.Iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reset method only for XORIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Use Reset(...) in querier.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reuse deletedIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Another way of reusing chunk iterators

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Unexport xorIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix memSeries.iterator(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add some comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-07-09 15:19:34 +05:30
Chris Marchbanks b40cc43958 Provide option to compress WAL records (#609) 
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2019-06-19 16:46:24 +03:00
Krasi Georgiev 69740485c1
move the wal repair logic in db.Open (#633) 
* move the wal repair logic in db.Open

This is to allow opening a wal in a read oly mode without triggering a
repair.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
 2019-06-14 17:39:22 +02:00
naivewong b05ca5c36e fix two benchmarks (#631) 
* fix two benchmarks

Signed-off-by: naivewong <867245430@qq.com>
 2019-06-14 14:30:49 +02:00
Krasi Georgiev 3549c097b4
add some comments for the different WAL repair tests (#630) 
The WAL repair tests look similar so added some extra comments to
clarify their purpose.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
 2019-06-12 16:10:37 +02:00
Brian Brazil be4edbe174
Start a new WAL segement on head truncation. (#605) 
This reduces disk space usage to not be a minimum of 3 128MB files
in small setups. This will possibly also help debug wal data issues,
by making things a bit more deterministic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-06-07 11:35:02 +01:00
Brian Brazil 149c5dc73a
Handle multiple refs for the same series when WAL reading. (#623) 
This can happen if a given series is created/truncated/recreated.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-06-06 14:28:54 +01:00
Brian Brazil 30d0ea59d7 Don't crash on an unknown tombstone ref. (#604) 
Fixes https://github.com/prometheus/prometheus/issues/5562

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-05-16 16:36:44 +03:00
Krasi Georgiev 288f67efbf
fix some unchecked errors and remove unused vars. (#592) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-04-25 13:07:04 +03:00
Brian Brazil dfed85e4a4
Keep series that are still in WAL in checkpoints (#577) 
If all the samples are deleted for a series,
we should still keep the series in the WAL as
anything else reading the WAL will still care
about it in order to understand the samples.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-04-09 14:16:24 +01:00
Krasi Georgiev c3ffdf1a99
Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 
Testing that createBlock creates blocks that can be opened.

and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.

Many missing .Close() calls were added for fixing failing os.RemoveAll

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-03-19 15:31:57 +02:00
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev 0b72f9af4c
Merge pull request #270 from codesome/master 
Head: don't create stones, delete samples directly
 2019-02-08 12:35:01 +02:00
Ganesh Vernekar 5481549324
Merge remote-tracking branch 'upstream/master' 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-07 22:15:06 +05:30
Simon Pasquier 95334f13c5 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-02-07 12:10:22 +01:00
Krasi Georgiev dac2b97dfd
make createBlock more generic so it can be used in other tests. (#489) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-28 14:24:49 +03:00
Ganesh Vernekar 4592b77035
Patch by Krasi (https://github.com/codesome/tsdb/pull/3) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-11 22:04:09 +05:30
Ganesh Vernekar d7e505db34
Dont store stones in head, delete samples directly 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-08 22:38:41 +05:30
Simon Pasquier d5d7a097e1 Update Makefile.common 
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-04 16:08:43 +01:00
Krasi Georgiev 48c439d26d
fix statick check errors (#475) 
fix the tests for `check_license` and `staticcheck`

the static check also found some actual bugs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-02 19:48:42 +03:00
Krasi Georgiev 520ab7dc53
re-add the missing prometheus_tsdb_wal_corruptions_total (#473) 
closes https://github.com/prometheus/tsdb/issues/471

after implementing the new WAL this metric was missing so adding it again.
Also added it in a test to make sure it works as expected.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-18 13:24:56 +03:00
Krasi Georgiev 2962202ed3
fix windows tests (#469) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-13 16:29:29 +03:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461) 
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-04 13:30:49 +03:00
Krasi Georgiev 0493efb7c5
repair wal when the record cannot be decoded (#453) 
* repair wal when the record cannot be decoded

Currently repair is run only when the error happens in the reader.

A corruption can occur after the record is read and when it is decoded.
This change wraps the error at decoding as a CorruptionErr as this error
is expected to trigger a repair.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-30 13:37:04 +02:00
Krasi Georgiev 5a9ddeecef
fix lint errors (#439) 
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-14 18:40:01 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419) 
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-10-25 21:06:19 +01:00
Ben a8351dc9d0 Using filepath.Join() instead of strings with slashes (#428) 
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
 2018-10-25 10:32:57 +01:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-21 11:07:35 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2018-09-21 09:23:01 +03:00
Ganesh Vernekar e779603633 Reuse testutil (#393) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-21 11:47:59 +05:30
Chris Marchbanks a8966cb53d Fix race condition between gc and committing (#378) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2018-09-17 19:58:42 +03:00
Krasi Georgiev ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e (#382) 
no logic changes just uncommented and fixed the issues.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-17 14:28:55 +03:00
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345) 
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
 2018-09-13 19:34:26 +03:00
Fabian Reinartz def912ce0e Integrate new WAL and checkpoints 
Remove the old WAL and drop in the new one

Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-07-19 07:25:30 -04:00
codwu 667e539a7a Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete 2018-07-06 20:21:32 +08:00
Fabian Reinartz ea607b9fc3 Log series on rollback 
Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-06-28 09:04:07 -04:00
codwu 84a45cb79a add rwmutex to prevent concurrent map read when delete series 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-08 19:52:01 +08:00
Alexey Palazhchenko 64cce683cc gofmt -s. 
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
 2018-05-07 15:39:54 +03:00
Mario Trangoni b7173eb0e5 fix some comments typos (#315) 2018-04-08 10:28:30 +01:00
Simon Pasquier 79defa54df Fix crash when a series has no block 2018-02-21 16:45:06 +01:00
Fabian Reinartz adb6be73dd move label read function int labels pkg 2017-12-21 11:55:58 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Goutham Veeramachaneni e6e3fa566a Remove verbose log line that is causing travis issues 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-12-21 12:16:11 +05:30
Goutham Veeramachaneni 9f2935c5cf
Merge pull request #224 from Gouthamve/fix-214 
Make sure gc'ed chunks are handled properly
 2017-12-21 12:10:24 +05:30
Callum Styan bcde61b237 we shouldn't assume we'll receive the labels in sorted order 2017-12-18 15:32:39 -08:00
Callum Styan 8326e410d0 decouple tsdb main.go (and tests) from prometheus/pkg 2017-12-18 15:26:16 -08:00
Goutham Veeramachaneni 05d62ca842 Make sure gc'ed chunks are handled properly 
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2017-12-14 19:48:39 +00:00
Callum Styan df7cc4dff5 remove Nil/NotNil functions 2017-12-08 13:42:08 -08:00
Callum Styan ca4e817372 move test utils to it's own package so we can use it in tests for other packages in tsdb repo 2017-12-06 17:06:14 -08:00
Callum Styan 5aa2af255b use test utils in head_test 2017-12-06 16:00:14 -08:00