When executing promql expressions, the engine first collects all series
and chunk metas. The chunk metas need to refer to immutable chunks,
otherwise reading the same series twice during evaluation of an
expression could yield different values. Head chunks are not immutable as
opposed to chunks from blocks or mmap-ed chunks, thus they have to be
copied.
Before native histograms, the copy of the OOO head could only result in a
single chunk, which was attached to chunks.Meta.Chunk field by
OOOHeadIndexReader.series method. It would be a major rewrite to make
chunks.Meta.Chunk field an array or an Iterable.
It also seems hard/impossible and definitely less optimal to generate
more chunks.Meta from the OOOHeadIndexReader.series() function as the
metas need to be non overlapping (this can be done), but also tell
memSeries.oooMergedChunks later what to return from overlapping chunks.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
There was a bug where the min time of a chunk was adjusted even
if it was only recoded and not completely new.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
The test was assuming that you can tell the sample type in
chunks.Sample interface by looking at the H() and FH() method output.
That turns out to be not true, you need to look at Type() function.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
series() via Select() I think on OOOHeadIndexReader was not handling
the chunk ranges correctly, creating overlapping chunk metas.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* chunkenc: allow missing empty buckets on histogram append
Allow appending to chunks when the histogram to be added is missing
some buckets, but the missing buckets are empty in the chunk.
For example bucket at index 5 is present in the chunk, but its value
is 0 and the new histogram doesn't have a bucket at index 5.
This fixes an issue of merging chunks where one chunk was recoded to
retroactively have some empty buckets in all the histograms and we are
merging in a histogram that doesn't have the empty bucket (because it
was not recoded yet).
The operation alters the histogram that is being added, however this has
already been the case when appending gauge histograms. Thus the test
TestHistogramSeriesToChunks in storage package is changed to explicitly
test what happened to the appended histogram - Compact(0) call is removed.
The new expandIntSpansAndBuckets and expandFloatSpansAndBuckets functions
are a merge of expandSpansForward and counterResetInAnyBucket and
counterResetInAnyFloatBucket.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
The histogram stats iterator does not fully clear the histogram object
and is not resilient to new fields being added to the histogram type.
To resolve the issue, the commit uses the CopyTo methods which should
be future proof to new fields being added.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Don't reshard if we haven't successfully sent a sample in the last
shardUpdateDuration seconds.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: kushagra Shukla <kushalshukla110@gmail.com>
We don't use seriesShard during DB initialization, so we can use the
same 8 bytes to store mmMaxTime, and save those during the rest of the
lifetime of the database.
This doesn't affect CPU performance.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Believed to trigger segmentation faults due to memory-mapped block
data still being accessed by iterators after the querier is closed.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
The histogram stats decoder keeps track of the last seen histogram sample
in order to properly detect counter resets. We are seeing an issue where
a histogram with UnknownResetHint gets treated as a counter reset when it follows
a stale histogram sample.
I believe that this is incorrect since stale samples should be completely ignored
in PromQL. As a result, they should not be stored in the histogram stats iterator
and the counter reset detection needs to be done against the last non-stale sample.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
