Commit graph

103 commits

Author SHA1 Message Date
Fabian Reinartz d09283f60a Add repair of bad index version 2018-02-09 13:11:03 +01:00
Goutham Veeramachaneni 6c62437101 Fix metric unit (#275)
Fixes #274

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2018-02-05 08:36:18 +00:00
Simon Pasquier 5d47710f8a Fix panic on nil index writer 2018-01-24 15:55:04 +01:00
Shubheksha Jalan 44052bc937 Make version for index format configurable 2018-01-11 00:49:16 +05:30
Fabian Reinartz 1e55b7987f Improve comments, handle allPostingsKey properly 2017-12-22 09:43:34 +01:00
Fabian Reinartz 7087f68647 Return new block ULID in compactor 2017-12-21 12:02:05 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Goutham Veeramachaneni f35e9c7fec
Merge branch 'master' into fix-dels
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-30 13:30:13 +05:30
Goutham Veeramachaneni 24ff293dc9
Merge pull request #207 from Gouthamve/compact-fail
Don't retry failed compactions.
2017-11-30 12:58:19 +05:30
Fabian Reinartz cf7f6108fd Order postings lists in index file by key
Aligning postings list for similar keys close to each other improves
page cache hit rates in typical queries that select postings for
multiple label pairs with the same name.
2017-11-29 08:28:55 +01:00
Goutham Veeramachaneni 1fc94a02d1
Don't retry failed compactions.
Fixes prometheus/prometheus#3487

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-23 15:16:20 +05:30
Goutham Veeramachaneni 1627a47640
A copy is being assigned newChunk not the original
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-22 17:58:06 +05:30
Fabian Reinartz a031cf7424 Return ULID in Compactor 2017-11-14 15:25:30 +01:00
Fabian Reinartz f1512a368a Expose ChunkSeriesSet and lookups methods. 2017-11-13 14:02:32 +01:00
Fabian Reinartz 3ef4326114 Refactor tombstone reader types 2017-11-13 13:38:07 +01:00
ranbochen a27cf34a36 fix bugs on platform windows to pass all test case. (#192)
* fix bugs on platform windows to pass all test case.

* fix bugs on platform windows to pass all test case

* clean up codes
2017-10-31 15:37:41 +01:00
Fabian Reinartz d17104f1f0 Prefix all metrics with prometheus_* 2017-10-20 12:32:32 +02:00
Fabian Reinartz fb9da52b11 Add more verbose error handling for closing, reduce locking
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.

Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
2017-10-10 12:13:37 +02:00
Goutham Veeramachaneni da565f975e Merge pull request #161 from prometheus/fileutil
Remove dependency on etcd/pkg/fileutil
2017-10-04 17:08:54 +05:30
Fabian Reinartz f04ec031eb compact: sync temporary directory 2017-10-04 12:22:09 +02:00
Fabian Reinartz bbe72dccb9 Remove dependency on etcd/pkg/fileutil 2017-10-04 10:23:41 +02:00
Goutham Veeramachaneni c35d3a65bd
Add levels to all log lines.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-28 12:49:34 +05:30
Fabian Reinartz 6ee254e353 Ensure postings are always sorted
IDs for new series are handed out before the postings are locked. Thus
series are not indexed in order of their IDs, which could result in only
partially sorted postings list.
Iterating over those silently skipped elements as the sort invariant was
violated.
2017-09-21 09:38:18 +02:00
Fabian Reinartz f904cd385f Do not build a superflous 'all' postings 2017-09-08 18:41:43 +02:00
Fabian Reinartz c36d574290 Replace single head lock with granular locks
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00
Fabian Reinartz 9f41d9fd3c Instrument chunks on level 1 compactions 2017-09-01 16:10:10 +02:00
Fabian Reinartz 4cc37eecab Refactor and add tests for compactor 2017-09-01 11:50:58 +02:00
Fabian Reinartz 3901b6e70b Remove multiple heads
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Goutham Veeramachaneni 7438ed7035 Expose Intervals type for use by TombstoneReader.
TombstoneReader is exposed but Intervals is not.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-25 16:06:36 +05:30
Goutham Veeramachaneni 178f840295 Fix panic caused by 0 division
Introduced in #108

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-18 07:45:29 +02:00
Fabian Reinartz 905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz 66ff7b12e9 Pool Chunk objects during compaction 2017-08-08 17:35:34 +02:00
Fabian Reinartz 2644c8665c Don't allocate ChunkMetas, reuse postings slices 2017-08-06 20:41:24 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Fabian Reinartz 3951d8cc29 Fix selection of compactable blocks 2017-08-04 12:02:34 +02:00
Fabian Reinartz 5d7b5994d6 Fix compaction range selection 2017-07-13 16:13:59 +02:00
Goutham Veeramachaneni 401bdfd4a6 Make Chunks return the number of samples inside
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:45:12 +02:00
Goutham Veeramachaneni 1263a68875 Recompact blocks with large number of tombstones
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:27:31 +02:00
Goutham Veeramachaneni 9a2ab732b5 Choose ranges by retention and min-block-duration
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 17:30:34 +02:00
Goutham Veeramachaneni ce23adb579 Initial compaction implementation for block-ranges
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-07 16:32:18 +02:00
Fabian Reinartz 03f95ee096 Fix directory file descriptor leak. 2017-06-12 00:05:04 +02:00
Fabian Reinartz 05e411a8eb Improve heuristic to spread chunks across block 2017-06-08 11:30:32 +02:00
Fabian Reinartz f006e2d1ab Merge pull request #92 from prometheus/sources
Add list of source head blocks to meta.json
2017-06-07 11:50:44 +02:00
Fabian Reinartz 683b4c8c2a Add list of source head blocks to meta.json 2017-06-07 11:06:45 +02:00
Goutham Veeramachaneni a1c8425357
Initial implementation of HeadBlock Snapshots
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-05 13:48:31 +05:30
Goutham Veeramachaneni 3a5ae6b1a4 Merge pull request #82 from Gouthamve/deletes-1
Implement deletions
2017-05-27 22:06:45 +05:30
Goutham Veeramachaneni 44e9ae38b5
Incorporate PR feedback.
* Expose Stone as it is used in an exported method.
* Move from tombstoneReader to []Stone for the same reason as above.
* Make WAL reading a little cleaner.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-26 21:26:31 +05:30
Fabian Reinartz b3b9245ad6 Add and fix metrics 2017-05-26 15:13:03 +02:00
Goutham Veeramachaneni 6febabeb28
Final delete fixes.
* Make sure no reads happen on the block when delete is in progress.
* Fix bugs in compaction.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-26 16:31:45 +05:30