Commit graph

72 commits

Author SHA1 Message Date
Krasi Georgiev c3ffdf1a99
Test createBlock and check all os.RemoveAll in the tests for errors. (#549)
Testing that createBlock creates blocks that can be opened.

and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.

Many missing .Close() calls were added for fixing failing os.RemoveAll

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-03-19 15:31:57 +02:00
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370)
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-02-14 14:29:41 +01:00
Simon Pasquier 95334f13c5 Merge branch 'master' into update-makefile-common
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-02-07 12:10:22 +01:00
Krasi Georgiev dac2b97dfd
make createBlock more generic so it can be used in other tests. (#489)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-28 14:24:49 +03:00
Simon Pasquier 2e69508536 Merge branch 'master' into update-makefile-common
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-22 10:47:31 +01:00
mknapphrt ebf5d74325 Added storage size based retention method and new metrics (#343)
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.

2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
2019-01-16 13:03:52 +03:00
Simon Pasquier d5d7a097e1 Update Makefile.common
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-04 16:08:43 +01:00
Krasi Georgiev eb6586f513
rename createPopulatedBlock to createBlock and use it instead ot createEmptyBlock (#488)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-29 14:20:51 +03:00
Krasi Georgiev a90a7195d4
createPopulatedBlock returns the block dir instead of the block (#487)
It is easy to forget to close the block returned by createPopulatedBlock
which causes failures for windows so instead it returns the block dir
and which can be used by OpenBlock explicitly.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-29 10:23:56 +03:00
Krasi Georgiev 090b6852e1
remove unused PrefixMatcher (#474)
* remove unused `PrefixMatcher`

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-28 21:13:02 +03:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461)
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-04 13:30:49 +03:00
Krasi Georgiev 5a9ddeecef
fix lint errors (#439)
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-14 18:40:01 +02:00
Ganesh Vernekar 3a08a71d86 LabelNames() method to get all unique label names (#369)
* LabelNames() method to get all unique label names

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-07 17:52:41 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419)
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-25 21:06:19 +01:00
Ben a8351dc9d0 Using filepath.Join() instead of strings with slashes (#428)
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
2018-10-25 10:32:57 +01:00
Chris Marchbanks f4afc7dff2 Add benchmark for querying a persisted block (#425)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-10-24 00:35:52 +03:00
Bartek Płotka 047b1b1357 compact: Verify for chunks outside of compacted time range. Added unit test for populateBlocs. (#349)
* compact: Verify for chunks outside of compacted time range. 
 Unit test for populateBlocs.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-12 12:45:19 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345)
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-09-13 19:34:26 +03:00
codwu bc6ef0b94e rename mts to intvlGroups
Signed-off-by: codwu <wuhan9087@163.com>
2018-07-10 21:24:13 +08:00
codwu e4444ca48c update addInterval function and test.
Signed-off-by: codwu <wuhan9087@163.com>
2018-07-06 20:30:27 +08:00
codwu cd145c90d5 remove put function and use RLock in Iter function
Signed-off-by: codwu <wuhan9087@163.com>
2018-06-25 21:52:11 +08:00
codwu 84a45cb79a add rwmutex to prevent concurrent map read when delete series
Signed-off-by: codwu <wuhan9087@163.com>
2018-06-08 19:52:01 +08:00
Fabian Reinartz 54205903f7 Move index and chunk encoders to own packages 2017-12-21 12:02:05 +01:00
Fabian Reinartz adb6be73dd move label read function int labels pkg 2017-12-21 11:55:58 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Callum Styan ca4e817372 move test utils to it's own package so we can use it in tests for other packages in tsdb repo 2017-12-06 17:06:14 -08:00
Callum Styan a1a91ea96a use test utils in querier_test 2017-12-06 16:27:09 -08:00
Fabian Reinartz 3ef4326114 Refactor tombstone reader types 2017-11-13 13:38:07 +01:00
Fabian Reinartz e5ce2bef43 Add explicit error to Querier.Select
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
2017-11-13 12:16:58 +01:00
Fabian Reinartz 665955da48 Clarify postings index semantics, handle staleness
The postings list index may point to series that no longer
exist during garbage collection. This clarifies that this is valid
behavior.
It would be possible, though more complex, to always keep them in sync.
However, series existance means nothing in itself as the queried time
range defines whether there's actual data. Thus our definition is sane
overall as long as drift is kept small.
2017-10-11 09:37:19 +02:00
Fabian Reinartz c36d574290 Replace single head lock with granular locks
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00
Fabian Reinartz 3901b6e70b Remove multiple heads
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Goutham Veeramachaneni 7438ed7035 Expose Intervals type for use by TombstoneReader.
TombstoneReader is exposed but Intervals is not.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-25 16:06:36 +05:30
Fabian Reinartz 2644c8665c Don't allocate ChunkMetas, reuse postings slices 2017-08-06 20:41:24 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Dmitry Ilyevsky 37194b7a30 Add prefix label matcher.
Implement labels.PrefixMatcher and use interface conversion in querier
to optimize label tuples search.

[unit-tests]: Fix bug and populate label index for mock index.

Signed-off-by: Dmitry Ilyevsky <ilyevsky@gmail.com>
2017-07-22 01:06:30 -07:00
Fabian Reinartz bc3fee4820 Add benchmark for mergedSeriesSet 2017-07-05 16:19:28 +02:00
Fabian Reinartz bda3ed20ac Fix omitting of chunk on Seek() 2017-06-30 15:06:27 +02:00
Goutham Veeramachaneni a280169427
Add regression test
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-13 13:21:22 +05:30
Goutham Veeramachaneni f29fb62fba
Make TombstoneReader a Getter.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-24 11:24:24 +05:30
Goutham Veeramachaneni 244b73fce1
Rename for clarity and consistency.
Misc. changes for code cleanliness.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 16:42:36 +05:30
Goutham Veeramachaneni 7ccf9d0211
Add test on blockQuerier which uses tombstones.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 13:31:57 +05:30
Goutham Veeramachaneni d6bd64357b
Fix Delete on HeadBlock
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-19 22:54:29 +05:30
Goutham Veeramachaneni 22c1b5b492
Make SeriesSets use tombstones.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-17 14:49:42 +05:30
Goutham Veeramachaneni 7bbbd55aad
Fix bug where having one chunk can cause panic
When we have only one chunk that is out of range, then we are returning
it unpopulated (w/o calling `Chunk(ref)`). This would cause a panic
downstream.

Fixes: prometheus/prometheus#2629

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-03 22:45:28 +05:30
Fabian Reinartz 085991c9da Merge pull request #71 from Gouthamve/tests
Fix #59
2017-05-01 11:46:16 +02:00
Goutham Veeramachaneni 8b43b0d2c1
Fix broken tests
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-01 15:01:17 +05:30
Goutham Veeramachaneni 6169c33fb8
Fix #59
Mutating the chunks can change their length. Hence referencing using
previous indices can cause panics.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-01 14:33:56 +05:30