Commit graph

49 commits

Author SHA1 Message Date
Fabian Reinartz 54205903f7 Move index and chunk encoders to own packages 2017-12-21 12:02:05 +01:00
Fabian Reinartz adb6be73dd move label read function int labels pkg 2017-12-21 11:55:58 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Callum Styan ca4e817372 move test utils to it's own package so we can use it in tests for other packages in tsdb repo 2017-12-06 17:06:14 -08:00
Callum Styan a1a91ea96a use test utils in querier_test 2017-12-06 16:27:09 -08:00
Fabian Reinartz 3ef4326114 Refactor tombstone reader types 2017-11-13 13:38:07 +01:00
Fabian Reinartz e5ce2bef43 Add explicit error to Querier.Select
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
2017-11-13 12:16:58 +01:00
Fabian Reinartz 665955da48 Clarify postings index semantics, handle staleness
The postings list index may point to series that no longer
exist during garbage collection. This clarifies that this is valid
behavior.
It would be possible, though more complex, to always keep them in sync.
However, series existance means nothing in itself as the queried time
range defines whether there's actual data. Thus our definition is sane
overall as long as drift is kept small.
2017-10-11 09:37:19 +02:00
Fabian Reinartz c36d574290 Replace single head lock with granular locks
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00
Fabian Reinartz 3901b6e70b Remove multiple heads
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Goutham Veeramachaneni 7438ed7035 Expose Intervals type for use by TombstoneReader.
TombstoneReader is exposed but Intervals is not.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-25 16:06:36 +05:30
Fabian Reinartz 2644c8665c Don't allocate ChunkMetas, reuse postings slices 2017-08-06 20:41:24 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Dmitry Ilyevsky 37194b7a30 Add prefix label matcher.
Implement labels.PrefixMatcher and use interface conversion in querier
to optimize label tuples search.

[unit-tests]: Fix bug and populate label index for mock index.

Signed-off-by: Dmitry Ilyevsky <ilyevsky@gmail.com>
2017-07-22 01:06:30 -07:00
Fabian Reinartz bc3fee4820 Add benchmark for mergedSeriesSet 2017-07-05 16:19:28 +02:00
Fabian Reinartz bda3ed20ac Fix omitting of chunk on Seek() 2017-06-30 15:06:27 +02:00
Goutham Veeramachaneni a280169427
Add regression test
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-13 13:21:22 +05:30
Goutham Veeramachaneni f29fb62fba
Make TombstoneReader a Getter.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-24 11:24:24 +05:30
Goutham Veeramachaneni 244b73fce1
Rename for clarity and consistency.
Misc. changes for code cleanliness.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 16:42:36 +05:30
Goutham Veeramachaneni 7ccf9d0211
Add test on blockQuerier which uses tombstones.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 13:31:57 +05:30
Goutham Veeramachaneni d6bd64357b
Fix Delete on HeadBlock
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-19 22:54:29 +05:30
Goutham Veeramachaneni 22c1b5b492
Make SeriesSets use tombstones.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-17 14:49:42 +05:30
Goutham Veeramachaneni 7bbbd55aad
Fix bug where having one chunk can cause panic
When we have only one chunk that is out of range, then we are returning
it unpopulated (w/o calling `Chunk(ref)`). This would cause a panic
downstream.

Fixes: prometheus/prometheus#2629

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-03 22:45:28 +05:30
Fabian Reinartz 085991c9da Merge pull request #71 from Gouthamve/tests
Fix #59
2017-05-01 11:46:16 +02:00
Goutham Veeramachaneni 8b43b0d2c1
Fix broken tests
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-01 15:01:17 +05:30
Goutham Veeramachaneni 6169c33fb8
Fix #59
Mutating the chunks can change their length. Hence referencing using
previous indices can cause panics.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-01 14:33:56 +05:30
Fabian Reinartz a5322c50fc Merge pull request #36 from Gouthamve/tests
Tests For Querier and Postings
2017-04-24 10:50:59 +02:00
Goutham Veeramachaneni 6178de9acc
Removed unused feild
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-22 01:43:18 +05:30
Goutham Veeramachaneni 725b69caa1
Incorporate feedback on #36
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-22 01:38:26 +05:30
Goutham Veeramachaneni 239f8b9eb5
Modify tests to check for mint, maxt
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-14 00:37:21 +05:30
Goutham Veeramachaneni b60c2068bc
Make blockQuerier return data in valid time-range
Fixes #43

Added mint, maxt to chunkSeriesIterator. Adding a field there is
inevitable as something similar is required for ignoring deleted
time-ranges.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-14 00:36:14 +05:30
Goutham Veeramachaneni b9868c9f0b
Initial test for blockQuerier
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-13 19:57:31 +05:30
Fabian Reinartz 778103b450 Add liecence file and headers 2017-04-10 20:59:45 +02:00
Goutham Veeramachaneni 0c5c4f07c5
Test "Stitching" SeriesIterators
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-09 19:45:22 +05:30
Fabian Reinartz 10c7c9acbe Adjust import names to new repository organisation 2017-04-04 11:27:26 +02:00
Fabian Reinartz 3be4ef94ce Move BufferedSeriesIterator in own package
This functionality is useful for a lot of clients but not relevant to
the TSDB's core features.
2017-03-24 13:23:32 +01:00
Fabian Reinartz d6fb6aaaa8 Rename paritionSeriesSet to mergedSeriesSet 2017-03-14 15:24:08 +01:00
Fabian Reinartz 4590b61343 Rename shard to partition 2017-01-06 08:08:02 +01:00
Fabian Reinartz 40cf215fba Hotfix missing sorting of head block series index 2017-01-04 09:47:20 +01:00
Fabian Reinartz ec99f99d3d Fix and test bug in shardSeriesSet 2017-01-03 19:02:42 +01:00
Fabian Reinartz 62b8ded0a5 Rename iterator value getters to At() 2017-01-02 13:33:06 +01:00
Fabian Reinartz 5efe1d178e labels: add Compare and String methods 2016-12-24 13:49:35 +01:00
Fabian Reinartz 869cccf080 Test and fixes for buffered iterator 2016-12-21 16:06:33 +01:00
Fabian Reinartz 0a94f58f1a Fix test import of labels, simplify constructor names 2016-12-21 15:12:26 +01:00
Fabian Reinartz cddc29fa17 Fix labels comparison, fetch correct labels 2016-12-20 14:54:52 +01:00
Fabian Reinartz aabb21f4b9 Add shard series set test 2016-12-19 11:44:11 +01:00
Fabian Reinartz e561c91d53 Implement proper buffered iterator
This adds a proper duration based lookback buffer for series iterators
to allow advancing sequentially while remaining able to calculate time
aggregating functions such as `rate` backwards.

It uses an array ring buffer to minimize heap allocations for
potentially hundreds of thousands of series for a single query.
2016-12-14 21:14:44 +01:00
Fabian Reinartz 725385ea05 Fix compareLabels, add test 2016-12-14 15:47:05 +01:00