mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-09 23:24:05 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
1093 commits 245 branches 436 tags 650 MiB
Commit graph

94 commits

Author SHA1 Message Date
Ganesh Vernekar 7cf09b0395
Moving tsdb into its own subdirectory 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-08-13 13:58:49 +05:30
Ganesh Vernekar b1cd829030
Reuse Chunk Iterator (#642) 
* Reset method for chunkenc.Iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reset method only for XORIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Use Reset(...) in querier.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Reuse deletedIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Another way of reusing chunk iterators

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Unexport xorIterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix memSeries.iterator(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add some comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-07-09 15:19:34 +05:30
Chris Marchbanks b40cc43958 Provide option to compress WAL records (#609) 
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2019-06-19 16:46:24 +03:00
Krasi Georgiev 69740485c1
move the wal repair logic in db.Open (#633) 
* move the wal repair logic in db.Open

This is to allow opening a wal in a read oly mode without triggering a
repair.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
 2019-06-14 17:39:22 +02:00
naivewong b05ca5c36e fix two benchmarks (#631) 
* fix two benchmarks

Signed-off-by: naivewong <867245430@qq.com>
 2019-06-14 14:30:49 +02:00
Krasi Georgiev 3549c097b4
add some comments for the different WAL repair tests (#630) 
The WAL repair tests look similar so added some extra comments to
clarify their purpose.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
 2019-06-12 16:10:37 +02:00
Brian Brazil be4edbe174
Start a new WAL segement on head truncation. (#605) 
This reduces disk space usage to not be a minimum of 3 128MB files
in small setups. This will possibly also help debug wal data issues,
by making things a bit more deterministic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-06-07 11:35:02 +01:00
Brian Brazil 149c5dc73a
Handle multiple refs for the same series when WAL reading. (#623) 
This can happen if a given series is created/truncated/recreated.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-06-06 14:28:54 +01:00
Brian Brazil 30d0ea59d7 Don't crash on an unknown tombstone ref. (#604) 
Fixes https://github.com/prometheus/prometheus/issues/5562

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-05-16 16:36:44 +03:00
Krasi Georgiev 288f67efbf
fix some unchecked errors and remove unused vars. (#592) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-04-25 13:07:04 +03:00
Brian Brazil dfed85e4a4
Keep series that are still in WAL in checkpoints (#577) 
If all the samples are deleted for a series,
we should still keep the series in the WAL as
anything else reading the WAL will still care
about it in order to understand the samples.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2019-04-09 14:16:24 +01:00
Krasi Georgiev c3ffdf1a99
Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 
Testing that createBlock creates blocks that can be opened.

and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.

Many missing .Close() calls were added for fixing failing os.RemoveAll

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-03-19 15:31:57 +02:00
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev 0b72f9af4c
Merge pull request #270 from codesome/master 
Head: don't create stones, delete samples directly
 2019-02-08 12:35:01 +02:00
Ganesh Vernekar 5481549324
Merge remote-tracking branch 'upstream/master' 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-07 22:15:06 +05:30
Simon Pasquier 95334f13c5 Merge branch 'master' into update-makefile-common 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-02-07 12:10:22 +01:00
Krasi Georgiev dac2b97dfd
make createBlock more generic so it can be used in other tests. (#489) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-28 14:24:49 +03:00
Ganesh Vernekar 4592b77035
Patch by Krasi (https://github.com/codesome/tsdb/pull/3) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-11 22:04:09 +05:30
Ganesh Vernekar d7e505db34
Dont store stones in head, delete samples directly 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-01-08 22:38:41 +05:30
Simon Pasquier d5d7a097e1 Update Makefile.common 
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
 2019-01-04 16:08:43 +01:00
Krasi Georgiev 48c439d26d
fix statick check errors (#475) 
fix the tests for `check_license` and `staticcheck`

the static check also found some actual bugs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-02 19:48:42 +03:00
Krasi Georgiev 520ab7dc53
re-add the missing prometheus_tsdb_wal_corruptions_total (#473) 
closes https://github.com/prometheus/tsdb/issues/471

after implementing the new WAL this metric was missing so adding it again.
Also added it in a test to make sure it works as expected.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-18 13:24:56 +03:00
Krasi Georgiev 2962202ed3
fix windows tests (#469) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-13 16:29:29 +03:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461) 
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-04 13:30:49 +03:00
Krasi Georgiev 0493efb7c5
repair wal when the record cannot be decoded (#453) 
* repair wal when the record cannot be decoded

Currently repair is run only when the error happens in the reader.

A corruption can occur after the record is read and when it is decoded.
This change wraps the error at decoding as a CorruptionErr as this error
is expected to trigger a repair.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-30 13:37:04 +02:00
Krasi Georgiev 5a9ddeecef
fix lint errors (#439) 
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-14 18:40:01 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419) 
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-10-25 21:06:19 +01:00
Ben a8351dc9d0 Using filepath.Join() instead of strings with slashes (#428) 
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
 2018-10-25 10:32:57 +01:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-21 11:07:35 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2018-09-21 09:23:01 +03:00
Ganesh Vernekar e779603633 Reuse testutil (#393) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2018-09-21 11:47:59 +05:30
Chris Marchbanks a8966cb53d Fix race condition between gc and committing (#378) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2018-09-17 19:58:42 +03:00
Krasi Georgiev ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e (#382) 
no logic changes just uncommented and fixed the issues.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-09-17 14:28:55 +03:00
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345) 
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
 2018-09-13 19:34:26 +03:00
Fabian Reinartz def912ce0e Integrate new WAL and checkpoints 
Remove the old WAL and drop in the new one

Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-07-19 07:25:30 -04:00
codwu 667e539a7a Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete 2018-07-06 20:21:32 +08:00
Fabian Reinartz ea607b9fc3 Log series on rollback 
Signed-off-by: Fabian Reinartz <freinartz@google.com>
 2018-06-28 09:04:07 -04:00
codwu 84a45cb79a add rwmutex to prevent concurrent map read when delete series 
Signed-off-by: codwu <wuhan9087@163.com>
 2018-06-08 19:52:01 +08:00
Alexey Palazhchenko 64cce683cc gofmt -s. 
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
 2018-05-07 15:39:54 +03:00
Mario Trangoni b7173eb0e5 fix some comments typos (#315) 2018-04-08 10:28:30 +01:00
Simon Pasquier 79defa54df Fix crash when a series has no block 2018-02-21 16:45:06 +01:00
Fabian Reinartz adb6be73dd move label read function int labels pkg 2017-12-21 11:55:58 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Goutham Veeramachaneni e6e3fa566a Remove verbose log line that is causing travis issues 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-12-21 12:16:11 +05:30
Goutham Veeramachaneni 9f2935c5cf
Merge pull request #224 from Gouthamve/fix-214 
Make sure gc'ed chunks are handled properly
 2017-12-21 12:10:24 +05:30
Callum Styan bcde61b237 we shouldn't assume we'll receive the labels in sorted order 2017-12-18 15:32:39 -08:00
Callum Styan 8326e410d0 decouple tsdb main.go (and tests) from prometheus/pkg 2017-12-18 15:26:16 -08:00
Goutham Veeramachaneni 05d62ca842 Make sure gc'ed chunks are handled properly 
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2017-12-14 19:48:39 +00:00
Callum Styan df7cc4dff5 remove Nil/NotNil functions 2017-12-08 13:42:08 -08:00
Callum Styan ca4e817372 move test utils to it's own package so we can use it in tests for other packages in tsdb repo 2017-12-06 17:06:14 -08:00