Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
This causes the head to be GCed at startup,
removing any series that were read from the WAL
but have since been written to a block. In
systems with low ingestion rates, this potentially
could be many many hours of data.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* Fix filehandling for windows
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Fix more windows filehandling issues
Windows: Close files before deleting Checkpoints.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close writers in case of errors so they can be deleted
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close block so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close file to delete it
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close dir so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: close files so that they can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Review feedback
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
This fixes various issues when initializing the head time range
under different starting conditions.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
We assume in multiple places that the block list held by DB
has blocks sequential by time.
A regression caused us to hold them ordered by ULID, i.e. by creation
time instead.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
Due to the way blocks used to overlap by 1 millisecond (see #347), when
requesting a 2-hour interval starting at `blocks[1].MinTime`, the
`Querier` would consider three blocks: `blocks[0]`, `blocks[1]` and
`blocks[2]`, because `blocks[0].MaxTime` and `blocks[2].MinTime` were in
that interval.
However, if the blocks don't overlap, only two blocks should be
returned: `blocks[1]` and `blocks[2]`. This test ensures that it's
indeed the case.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
Due to a regression blocks were no longer ordered by time before
being stored in memory. This made data intermittently become unavailable
for queries.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
