* refactor block size calculation
The block size is kept in memory and not in the meta file anymore.
It now includes the size of the meta file itself for a more
correct block size.
It fixes a bug where the size didn't change when calling `block.Delete()`.
Adds a dedicated test to ensure correct block sizes.
This allows opening a db in a read only mode as it doesn't write to the meta file anymore.
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
Currently a time series with empty labels is not treated the same
as one with missing labels. Currently this can only come from
ALERTS&ALERT_FOR_STATE so it's unlikely anyone has actually hit it.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
Testing that createBlock creates blocks that can be opened.
and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.
Many missing .Close() calls were added for fixing failing os.RemoveAll
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
This causes the head to be GCed at startup,
removing any series that were read from the WAL
but have since been written to a block. In
systems with low ingestion rates, this potentially
could be many many hours of data.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* Fix filehandling for windows
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Fix more windows filehandling issues
Windows: Close files before deleting Checkpoints.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close writers in case of errors so they can be deleted
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close block so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close file to delete it
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close dir so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: close files so that they can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Review feedback
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
This fixes various issues when initializing the head time range
under different starting conditions.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
We assume in multiple places that the block list held by DB
has blocks sequential by time.
A regression caused us to hold them ordered by ULID, i.e. by creation
time instead.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
Due to the way blocks used to overlap by 1 millisecond (see #347), when
requesting a 2-hour interval starting at `blocks[1].MinTime`, the
`Querier` would consider three blocks: `blocks[0]`, `blocks[1]` and
`blocks[2]`, because `blocks[0].MaxTime` and `blocks[2].MinTime` were in
that interval.
However, if the blocks don't overlap, only two blocks should be
returned: `blocks[1]` and `blocks[2]`. This test ensures that it's
indeed the case.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
