Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
It is easy to forget to close the block returned by createPopulatedBlock
which causes failures for windows so instead it returns the block dir
and which can be used by OpenBlock explicitly.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
* compact: Verify for chunks outside of compacted time range.
Unit test for populateBlocs.
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
The postings list index may point to series that no longer
exist during garbage collection. This clarifies that this is valid
behavior.
It would be possible, though more complex, to always keep them in sync.
However, series existance means nothing in itself as the queried time
range defines whether there's actual data. Thus our definition is sane
overall as long as drift is kept small.
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.
As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
Implement labels.PrefixMatcher and use interface conversion in querier
to optimize label tuples search.
[unit-tests]: Fix bug and populate label index for mock index.
Signed-off-by: Dmitry Ilyevsky <ilyevsky@gmail.com>
When we have only one chunk that is out of range, then we are returning
it unpopulated (w/o calling `Chunk(ref)`). This would cause a panic
downstream.
Fixes: prometheus/prometheus#2629
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
Mutating the chunks can change their length. Hence referencing using
previous indices can cause panics.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
