Commit graph

826 commits

Author SHA1 Message Date
Alex Le 01d0dda4fc Rename PopulateBlockFunc to BlockPopulator
Signed-off-by: Alex Le <leqiyue@amazon.com>
2023-04-12 14:18:20 -07:00
Björn Rabenstein 8ed90b567b
Merge pull request #12234 from aknuds1/chore/improve-histogram-comments
tsdb: Improve a couple of histogram documentation comments
2023-04-12 10:55:22 +02:00
Björn Rabenstein 6e0a46900b
Merge pull request #12192 from leizor/leizor/prometheus/issues/11204
Add support for native histograms to concreteSeriesIterator
2023-04-11 12:30:35 +02:00
Arve Knudsen cca7178a12 tsdb: Improve a couple of histogram documentation comments
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-04-07 18:06:27 +02:00
Justin Lei 83f43982c9 Add support for native histograms to concreteSeriesIterator
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:54:15 -07:00
Justin Lei 73ff91d182 Test fixes
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:42:59 -07:00
Justin Lei c770ba8047 Add comment linking to PR
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:19:32 -07:00
Justin Lei 79db04eb12 Adjust samplesPerChunk from 120 to 220
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:19:32 -07:00
Ganesh Vernekar e709b0b36e
Merge pull request #12127 from codesome/ooo-mmap-replay
Update OOO min/max time properly after replaying m-map chunks
2023-04-04 12:05:57 +05:30
Ganesh Vernekar 5588cab8b2
Merge pull request #12173 from bboreham/builder-no-empty-labels
labels: simplify call to get Labels from Builder
2023-04-04 12:02:55 +05:30
Alex Le 1936868e9d
Allow populate block logic in compact to be overriden outside Prometheus (#11711)
Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Alex Le <emoc1989@gmail.com>
2023-04-04 12:01:49 +05:30
Ganesh Vernekar f55ab22179
Merge pull request #12186 from codesome/remove-file
Remove mistakenly added file
2023-03-30 19:24:04 +05:30
Oleg Zaytsev 3ded84e649
Fix TestCancelCompactions on windows
It seems that readOnlyDB was still opened which blocked the temp dir
cleanup.

Also changed the copy dir to be another TempDir instead of manually
creating one.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-03-30 13:38:43 +02:00
Björn Rabenstein ae42dd4c4a
Merge pull request #12179 from colega/fix-block-compaction-failed-when-shutting-down
Fix block compaction failed when shutting down
2023-03-30 13:27:49 +02:00
Oleg Zaytsev 6e2905a4d4
Use zeropool.Pool to workaround SA6002 (#12189)
* Use zeropool.Pool to workaround SA6002

I built a tiny library called https://github.com/colega/zeropool to
workaround the SA6002 staticheck issue.

While searching for the references of that SA6002 staticheck issues on
Github first results was Prometheus itself, with quite a lot of ignores
of it.

This changes the usages of `sync.Pool` to `zeropool.Pool[T]` where a
pointer is not available.

Also added a benchmark for HeadAppender Append/Commit when series
already exist, which is one of the most usual cases IMO, as I didn't find
any.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Improve BenchmarkHeadAppender with more cases

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* A little copying is better than a little dependency

https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Fix imports order

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Add license header

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Copyright should be on one of the first 3 lines

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Use require.Equal for testing

I don't depend on testify in my lib, but here we have it available.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Avoid flaky test

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Also use zeropool for pointsPool in engine.go

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-03-29 20:34:34 +01:00
Ganesh Vernekar b33a382646
Remove mistakenly added file
It got added in https://github.com/prometheus/prometheus/pull/11992

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-27 20:44:11 +05:30
Alan Protasio 6ddadd98b4
Optimization on mergedStringIter (#12132)
Optimization on NewMergedStringIter

Signed-off-by: Alan Protasio <alanprot@gmail.com>
2023-03-27 17:10:45 +05:30
Oleg Zaytsev 344c630857
Fix context.Canceled wrapping in compaction
We need to make sure that `tsdb_errors.NewMulti` handles the errors.Is()
calls properly, like it's done in grafana/dskit.

Also we need to check that `errors.Is(err, context.Canceled)`, not that
`err == context.Canceled`.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-03-23 11:10:00 +01:00
Oleg Zaytsev 2f32a9e3c3
Test compaction not failed during shutdown
Test that blocks are not marked as "compaction failed" during shutdown.
This shouldn't happen but this test currently fails.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-03-23 11:08:56 +01:00
Bryan Boreham b987afa7ef labels: simplify call to get Labels from Builder
It took a `Labels` where the memory could be re-used, but in practice
this hardly ever benefitted. Especially after converting `relabel.Process`
to `relabel.ProcessBuilder`.

Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil`
so the slice was reallocated multiple times by `append`.

Lastly `Builder.Labels()` now estimates that the final size will depend
on labels added and deleted.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-22 17:05:20 +00:00
Bryan Boreham 90b2f7a540
Merge pull request #12161 from codesome/update-comment
tsdb: Fix a comment in tsdb/head_read.go
2023-03-22 17:02:28 +00:00
Vernon Miller ca0abf26c5
Adds an affirmative log message for successful WAL repair (#12135)
* Adds an affirmative log message for successful WAL repair

Signed-off-by: Vernon Miller <vernon.miller@grafana.com>
Signed-off-by: Vernon Miller <96601789+aldernero@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-21 19:33:43 +05:30
Ganesh Vernekar 1b7d973f14
tsdb: Fix a comment in tsdb/head_read.go
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-21 15:15:36 +05:30
Abhijit Mukherjee 8f6d5dcd45
Fix: getting rid of EncOOOXOR chunk encoding (#12111)
Signed-off-by: mabhi <abhijit.mukherjee@infracloud.io>
2023-03-16 15:53:47 +05:30
Ganesh Vernekar 58a8d526e8
Merge pull request #11992 from codesome/no-reencode-chunk
Do not re-encode head chunk for ChunkQuerier
2023-03-15 18:30:38 +05:30
Ganesh Vernekar 0a3f203c63
Update tests to not assume the chunk implementation
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-15 17:58:37 +05:30
Ganesh Vernekar 45b025898f
Add BenchmarkHeadChunkQuerier and BenchmarkHeadQuerier
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-15 17:58:31 +05:30
Ganesh Vernekar 0c0c2af7f5
Do not re-encode head chunk in ChunkQuerier
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-15 17:58:01 +05:30
Ganesh Vernekar 2af44f9558
tsdb: Update OOO min/max time properly after replaying m-map chunks
Without this fix, if snapshots were enabled, and wbl goes missing
between restarts, then TSDB does not recognize that there are ooo
mmap chunks on disk and we cannot query them until those chunks
are compacted into blocks.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-13 13:14:00 +05:30
Ganesh Vernekar 1c3f1216b3
tsdb: Test querying after missing wbl with snapshots enabled
If the snapshot was enabled with some ooo mmap chunks on disk,
and wbl was removed between restarts, then we should still be able
to query the ooo mmap chunks after a restart. This test shows that
we are not able to query those ooo mmap chunks after a restart
under this situation.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-13 13:14:00 +05:30
Ganesh Vernekar c9d06f2826
tsdb: Replay m-map chunk only when required
M-map chunks replayed on startup are discarded if there
was no WAL and no snapshot loaded, because there is no
series created in the Head that it can map to. So only
load m-map chunks from disk if there is either a snapshot
loaded or there is WAL on disk.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-13 13:13:42 +05:30
Ganesh Vernekar 6c008ec56a
Merge pull request #11962 from jesusvazquez/jvp/protect-new-compaction-head-from-uninitialized-wbl
TSDB: Protect NewOOOCompactionHead from an uninitialized wbl
2023-03-13 10:52:03 +05:30
Đurica Yuri Nikolić c9b85afd93
Making the number of CPUs used for WAL replay configurable (#12066)
Adds `WALReplayConcurrency` as an option on tsdb `Options` and `HeadOptions`.
If it is not set or set <=0, then `GOMAXPROCS` is used, which matches the previous behaviour.

Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>
2023-03-07 16:41:33 +00:00
ansalamdaniel c1c444504e
Feat: metrics for head_chunks & wal folders (#12013)
Signed-off-by: ansalamdaniel <ansalam.daniel@infracloud.io>
2023-03-02 15:25:56 +05:30
Rens Groothuijsen d33eb3ab17
Automatically remove incorrect snapshot with index that is ahead of WAL (#11859)
Signed-off-by: Rens Groothuijsen <l.groothuijsen@alumni.maastrichtuniversity.nl>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-03-01 17:51:02 +05:30
Bryan Boreham f34b2cede3 Remove microbenchmarks
These benchmarks are all testing things related to what Prometheus does,
so perhaps have some historical interest, but we should not retain them
in the main repo.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-02-22 16:36:45 +00:00
Ganesh Vernekar 66da1d51fd
Merge pull request #12003 from codesome/redundant-chunk-access
Remove unnecessary chunk fetch in Head queries
2023-02-22 12:57:38 +05:30
Ganesh Vernekar d504c950a2
Remove unnecessary chunk fetch in Head queries
`safeChunk` is only obtained from the `headChunkReader.Chunk` call where
the chunk is already fetched and stored with the `safeChunk`. So, when
getting the iterator for the `safeChunk`, we don't need to get the chunk again.

Also removed a couple of unnecessary fields from `safeChunk` as a part of this.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-02-22 12:21:12 +05:30
Vishal N 96ba6831ae Observe delta in seconds prometheus_tsdb_sample_ooo_delta
Signed-off-by: Vishal Nadagouda <vishalmn1996@gmail.com>
2023-02-21 18:55:09 +05:30
Jesus Vazquez 5c3f058755 Add unit test and also protect truncateOOO
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2023-02-10 15:18:17 +01:00
Jesus Vazquez f269077855 Protect NewOOOCompactionHead from an unitialized wbl
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2023-02-10 13:00:29 +01:00
Justin Lei af1d9e01c7
Refactor tsdbutil for tests/native histograms (#11948)
* Add float histograms to ChunkFromSamplesGeneric

Signed-off-by: Justin Lei <justin.lei@grafana.com>

* Add Generate*Samples functions to tsdbutil

Signed-off-by: Justin Lei <justin.lei@grafana.com>

* PR responses

Signed-off-by: Justin Lei <justin.lei@grafana.com>

---------

Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-02-10 17:09:33 +05:30
George Krajcsovits 1f0cc09579
Export single ith test histogram generation functions (#11911)
* Export single ith test histogram generation functions

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Do not set counter reset hint for non-gauge histograms individually

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Apply suggestions from code review

Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-02-01 16:23:38 +05:30
Ganesh Vernekar 8e8b718365
Merge pull request #11858 from fayzal-g/fix-chunks-metrics
tsdb: when reading WAL, correctly update chunksRemoved and chunks metrics
2023-01-30 19:45:12 +05:30
beorn7 1cfc8f65a3 histograms: Return actually useful counter reset hints
This is a bit more conservative than we could be. As long as a chunk
isn't the first in a block, we can be pretty sure that the previous
chunk won't disappear. However, the incremental gain of returning
NotCounterReset in these cases is probably very small and might not be
worth the code complications.

Wwith this, we now also pay attention to an explicitly set counter
reset during ingestion. While the case doesn't show up in practice
yet, there could be scenarios where the metric source knows there was
a counter reset even if it might not be visible from the values in the
histogram. It is also useful for testing.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-25 16:57:21 +01:00
beorn7 57c18420ab histograms: General readability tweaks
- Adjust doc comments to go1.19 style.
- Break down some overly long lines.
- Minor doc comment tweaks and fixes.
- Some renaming.

Some rationales for the last point:

I have renamed “interjections” into “inserts”, mostly because it is
shorter, and the word shows up a lot by now (and the concept is
cryptic enough to not obfuscate it even more with abbreviations).

I have also tried to find more descriptive naming for the “compare
spans” functions.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-19 13:26:42 +01:00
fayzal-g cfa4ea53cc Correctly update chunksRemoved and chunks metrics
Signed-off-by: fayzal-g <fayzal.ghantiwala@grafana.com>
2023-01-18 10:58:48 +00:00
Ganesh Vernekar 6e560fe19b
tsdb: Avoid unnecessary allocation from 11779
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-17 16:53:49 +05:30
Mingjie Shao 78d3c4e823 tsdb: Fixed typo in Histogram
Signed-off-by: Mingjie Shao <com.jerryshao@jerryshao.com>
2023-01-16 18:13:45 +08:00
Ganesh Vernekar cb2be6e62f
Merge pull request #11779 from codesome/memseries-ooo
tsdb: Only initialise out-of-order fields when required
2023-01-16 10:58:05 +05:30
Jesus Vazquez 136956cca4
Attempt to append ooo sample at the end first (#11615)
This is an optimization on the existing append in OOOChunk.

What we've been doing so far is find the place inside the out-of-order
slice where the new sample should go in and then place it there and move
any samples to the right if necessary. This is OK but requires a binary
search every time the slice is bigger than 0.

The optimization is opinionated and suggests that although out-of-order
samples can be out-of-order amongst themselves they'll probably be in
order thus we can probably optimistically append at the end and if not
do the binary search.

OOOChunks are capped to 30 samples by default so this is a small
optimization but everything adds up, specially if you handle many active
timeseries with out-of-order samples.

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-13 19:00:50 +05:30
Marc Tudurí 721f33dbb0
histograms: Add remote-write support for Float Histograms (#11817)
* adapt code.go and write_handler.go to support float histograms
* adapt watcher.go to support float histograms
* wip adapt queue_manager.go to support float histograms
* address comments for metrics in queue_manager.go
* set test cases for queue manager
* use same counts for histograms and float histograms
* refactor createHistograms tests
* fix float histograms ref in watcher_test.go
* address PR comments

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-01-13 16:39:20 +05:30
Sebastian Rabenhorst c057318578
agent: native histogram support (#11842)
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
2023-01-12 11:13:44 -05:00
Ganesh Vernekar 38fa151a7c
tsdb: Only initialise out-of-order fields when required
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-12 20:29:16 +05:30
beorn7 6dcd03dbf3 tsdb: Add integer gauge histogram support
This follows what #11783 has done for float gauge histograms.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-11 13:28:43 +01:00
Ganesh Vernekar 57bcbf1888
Merge pull request #11783 from codesome/gauge-histogram
tsdb: Add gauge histogram support
2023-01-10 19:06:08 +05:30
Ganesh Vernekar 3c2ea91a83
tsdb: Test gauge float histograms
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 18:35:37 +05:30
Ganesh Vernekar 609b12d719
tsdb: Support gauge float histogram with recoding of chunk
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:48:09 +05:30
Ganesh Vernekar 8ad0d2d5d7
tsdb: Find union of two sets of histogram spans
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:43:33 +05:30
Ganesh Vernekar d7f5129042
tsdb: Add logic to determine appendable gauge float histograms
This is to check if a gauge histogram can be appended to the given chunk.
If not, it tells what changes to make to the chunk and the histogram
if possible.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:43:33 +05:30
Ganesh Vernekar a87e7e9e33
tsdb: Add counter reset hint to histograms and support in WAL
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:41:53 +05:30
Oleg Zaytsev de93a279a0
Shortcut postings for matchers when empty postings are selected (#11813)
* Add more benchmark cases
* Add shortcuts for empty postings

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-01-10 15:21:49 +05:30
Ganesh Vernekar fd89d7892c
Merge pull request #11809 from bboreham/dont-sort-postings-values
tsdb: sort values for Postings only when required
2023-01-10 15:02:21 +05:30
Ganesh Vernekar c94a41c4b2
Merge pull request #11785 from Fish-pro/erroris
Use errors.Is to check for a specific error
2023-01-10 14:56:14 +05:30
György Krajcsovits 97626c9583 Fix comment
Comment was not updated when code changed from labels to builder
in #11717

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-01-08 16:29:02 +01:00
Björn Rabenstein c49a28bb97
Merge pull request #11782 from codesome/floatappendabletest
tsdb: Improve TestFloatHistogramChunkAppendable and TestHistogramChunkAppendable
2023-01-05 17:15:10 +01:00
Bryan Boreham e61348d9f3 tsdb/index: fast-track postings for label=""
We need to special-case ""="" too, which is used in some tests to mean "everything".

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-05 14:05:54 +00:00
Bryan Boreham cf92cd2688 tsdb: sort values for Postings only when required
In the head and in v1 postings on disk, it makes no difference whether
postings are sorted. Only for v2 does the code step through in order.
So, move the sorting to where it is required, and thus skip it entirely
in the head.

Label values in on-disk blocks are already sorted, but `slices.Sort` is
very fast on already-sorted data so we don't bother checking.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-05 14:05:54 +00:00
Ganesh Vernekar 7ed1ddb338
tsdb: Improve TestHistogramChunkAppendable and add new cases
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-05 14:44:24 +05:30
Ganesh Vernekar fa0f04bbc6
Merge pull request #11805 from bboreham/fix-benchmark-intersect
tsdb/index: fix BenchmarkIntersect to do work on each loop
2023-01-04 18:19:14 +05:30
Bryan Boreham 3da2c99ffd tsdb/index: don't call ExpandPostings in a benchmark
This allocates memory for all the returned values, which skews the
result. We aren't trying to benchmark `ExpandPostings`, so just step
through all the values without storing them to consume them.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-03 15:26:29 +00:00
Bryan Boreham 4931983ca9 tsdb/index: make BenchmarkIntersect do work on each loop
Previously all the postings constructed were consumed on the first
iteration, so subsequent iterations did no work.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-03 15:25:38 +00:00
Fish-pro 6ed71a229e Use errors.Is to check for a specific error
Signed-off-by: Fish-pro <zechun.chen@daocloud.io>
2022-12-29 23:23:07 +08:00
Ganesh Vernekar b42802af9a
tsdb: Improve TestFloatHistogramChunkAppendable and add new cases
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 21:07:47 +05:30
Ganesh Vernekar c155c0e312
tsdb: Test staleness handling of FloatHistogram
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:48:56 +05:30
Ganesh Vernekar 2820e327db
tsdb: Add staleness handling for FloatHistogram
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:48:39 +05:30
Ganesh Vernekar e555469ba1
tsdb: Remove isHistogramSeries from memSeries
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:31:55 +05:30
Marc Tudurí 9474610baf
Support FloatHistogram in TSDB (#11522)
Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram.

Signed-off-by: Marc Tudurí <marctc@protonmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:25:07 +05:30
Bryan Boreham 1848623c77 tsdb: re-use iterator when stepping through chunks
Saves memory allocations, hence reduces garbage-collection overheads.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-22 17:01:47 +00:00
Bryan Boreham ccea61c7bf
Merge pull request #11717 from bboreham/labels-abstraction
Add and use abstractions over labels.Labels
2022-12-20 17:23:39 +00:00
Ganesh Vernekar 6fd89a6fd2
Add chunk encoding for float histogram (#11716)
Signed-off-by: Marc Tudurí <marctc@protonmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Marc Tudurí <marctc@protonmail.com>
2022-12-20 15:33:32 +05:30
Bryan Boreham 10b27dfb84 Simplify IndexReader.Series interface
Instead of passing in a `ScratchBuilder` and `Labels`, just pass the
builder and the caller can extract labels from it. In many cases the
caller didn't use the Labels value anyway.

Now in `Labels.ScratchBuilder` we need a slightly different API: one
to assign what will be the result, instead of overwriting some other
`Labels`. This is safer and easier to reason about.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 4b6a4d1425 Update package tsdb tests for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham ce2cfad0cb Update package tsdb/record for new labels.Labels type
Implement decoding via labels.ScratchBuilder, which we retain and re-use
to reduce memory allocations.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 543c318ec2 Update package tsdb for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham f0ec81badd Update package tsdb/test for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 14ad2e780b Update package tsdb/agent for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham a5bdff414b Update package tsdb/index tests for new labels.Labels type
Note in one cases we needed an extra copy of labels in case they change.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham d3d96ec887 tsdb/index: use ScratchBuilder to create Labels
This necessitates a change to the `tsdb.IndexReader` interface:
`index.Reader` is used from multiple goroutines concurrently, so we
can't have state in it.

We do retain a `ScratchBuilder` in `blockBaseSeriesSet` which is
iterator-like.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 927a14b0e9 Update package tsdb/index for new labels.Labels type
Incomplete - needs further changes to `Decoder.Series()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 89bf6e1df9 tsdb: Tidy up some test code
Use simpler utility function to create Labels objects, making fewer
assumptions about the data structure.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 19:39:46 +00:00
Bryan Boreham 0853250695 Review feedback
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 18:32:45 +00:00
Bryan Boreham 463f5cafdd storage: re-use iterators to save garbage
Re-use previous memory if it is already of the correct type.

In `NewListSeries` we hoist the conversion to an interface value out
so it only allocates once.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 18:32:45 +00:00
Bryan Boreham f0866c0774 tsdb: optimise block series iterators
Re-use previous memory if it is already of the correct type.

Also turn two levels of function closure into a single object that
holds the required data.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 18:32:45 +00:00
Bryan Boreham 3c7de69059 storage: allow re-use of iterators
Patterned after `Chunk.Iterator()`: pass the old iterator in so it
can be re-used to avoid allocating a new object.

(This commit does not do any re-use; it is just changing all the method
signatures so re-use is possible in later commits.)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 18:32:45 +00:00
Julien Pivotto 475cfe8a6b Merge remote-tracking branch 'origin/release-2.40'
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-12-14 11:22:01 +01:00
Ganesh Vernekar db99fc43e4
Merge pull request #11632 from bboreham/improve-bbss
tsdb: improve blockBaseSeriesSet scan
2022-12-14 15:05:27 +05:30
Ganesh Vernekar 54739a1465
Merge pull request #11674 from bboreham/fix-tsdb-test-mem
tsdb tests: allocate more reasonable sample slice
2022-12-14 15:01:04 +05:30
beorn7 5f366e9b62 histograms: Improve tests and fix exposed bugs
This adds negative buckets and access of float histograms to
TestHistogramChunkSameBuckets and TestHistogramChunkBucketChanges.

It also exercises a specific pattern of reusing an iterator (one where
no access has happened).

This exposes two bugs (where entries for positive buckets where used
where the corresponding entries for negative buckets should have been
used). One was fixed in #11627 (not merged), which triggered the work
in this commit.

This commit fixes both issues, so #11627 can be closed.

It also simplifies the code in the histogramIterator.Next method that
aims to recycle existing slice capacity.

Furthermore, this is on top of the release-2.40 branch because we
should probably cut a bugfix release for this.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-12-12 00:08:23 +01:00
Julien Pivotto 0b302f8a39
Merge pull request #11662 from prometheus/release-2.40
Merge back release-2.40 branch again
2022-12-06 17:30:51 +01:00
Bryan Boreham 9853888f9b tsdb tests: allocate more reasonable sample slice
Typical parameters are one hour  by 1 minute step, where the
function would allocate a slice of 3.6 million samples instead of 60.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-05 17:15:02 +00:00
Ganesh Vernekar 72a48321da
Merge pull request #11633 from pstibrany/populate-error
Enhance "cannot populate chunk" error message to include source block ID
2022-12-02 16:28:52 +05:30
Ganesh Vernekar b8b0d45d69
Fix reset of a histogram chunk iterator
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-11-30 17:50:05 +05:30
Julien Pivotto 0372e259ba
Merge pull request #11634 from prometheus/release-2.40
Merge release-2.40 branch into main
2022-11-29 15:54:58 +01:00
Bryan Boreham 6bdecf377c
Switch from 'sanity' to more inclusive lanuage (#9376)
* Switch from 'sanity' to more inclusive lanuage

"Removing ableist language in code is important; it helps to create and
maintain an environment that welcomes all developers of all backgrounds,
while emphasizing that we as developers select the most articulate,
precise, descriptive language we can rather than relying on metaphors.

The phrase sanity check is ableist, and unnecessarily references mental
health in our code bases. It denotes that people with mental illnesses
are inferior, wrong, or incorrect, and the phrase sanity continues to be
used by employers and other individuals to discriminate against these
people."

From https://gist.github.com/seanmhanson/fe370c2d8bd2b3228680e38899baf5cc

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-11-28 17:09:18 +00:00
Peter Štibraný af838ccf83 Include source block in error message when loading chunk fails.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
2022-11-28 09:12:54 +01:00
Bryan Boreham 1226922ff5 tsdb: improve blockBaseSeriesSet scan
Inverting the test for chunks deleted by tombstones makes all three
rejections consistent, and also avoids the case where a chunk is
excluded but still causes `trimFront` or `trimBack` to be set.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-11-26 15:23:02 +00:00
Bryan Boreham 0c05f95e92 tsdb: use smaller allocation in blockBaseSeriesSet
This reduces garbage, hence goes faster, when a short time range is
required compared to the amount of chunks in the block. For example
recording rules and alerts often look only at the last few minutes.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-11-26 14:56:22 +00:00
Ganesh Vernekar ad79fb9f25
Do not error on empty chunk during iteration in populateWithDelChunkSeriesIterator
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-11-23 17:32:28 +05:30
Ganesh Vernekar d0e683e26d
Add TestCompactHeadWithDeletion to test compaction failure after deletion
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-11-23 17:31:18 +05:30
Ganesh Vernekar 42633bd05c
Merge pull request #11485 from t00350320/prometheus-office
GetRefByhash() will query a label's ref with hash value rather than lset.Hash().
2022-11-16 15:09:49 +01:00
tanghengjian 982007ecab
GetRefByhash will query a label's ref with hash value rather than lset.Hash().
Signed-off-by: tanghengjian <1040104807@qq.com>
2022-11-16 14:13:59 +01:00
Oleg Zaytsev 8553a98267
Optimize postings offset table reading (#11535)
* Add BenchmarkOpenBlock

* Use specific types when reading offset table

Instead of reading a generic-ish []string, we can read a generic type
which would be specifically labels.Label.

This avoid allocating a slice that escapes to the heap, making it both
faster and more efficient in terms of memory management.

* Update error message for unexpected number of keys

* s/posting offset table/postings offset table/

* Remove useless lastKey assignment

* Use two []bytes vars, simplify

Applied PR feedback: removed generics, moved the label indices reading
to that specific test as we're not using it in production anyway, we're
just testing what we've just built.

Also using two []bytes variables for name and value that use the backing
buffer instead of using strings, this reduces allocations a lot as we
only copy them when we store them (this is optimized by the compiler).

* Fix the dumb bug

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
2022-11-14 17:48:16 +01:00
Julien Pivotto 739494d81b
Fix alignment of atomic int64 (#11547)
* Fix atomix int64 placement
* Test main for 386

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-11-09 11:18:49 +01:00
Ganesh Vernekar fa6e05903f
Merge pull request #11447 from prometheus/sparsehistogram
Add Support for Native Histograms

This PR merges all the coding work that has been done in sparsehistogram branch over the last 1 year into main branch.

Design doc on native histograms: https://docs.google.com/document/d/1cLNv3aufPZb3fNfaJgdaRBZsInZKKIHo9E6HinJVbpM/edit
Some sneak peak: https://www.youtube.com/watch?v=T2GvcYNth9U
2022-10-26 17:10:46 -04:00
Viacheslav Panasovets 3d2e18bad5
Fix time.Since() in defer. Wrap in anonymous function (#11489)
Function arguments in defer evaluated during definition of defer, not
during execution

Signed-off-by: Slavik Panasovets <slavik@google.com>

Signed-off-by: Slavik Panasovets <slavik@google.com>
2022-10-26 00:26:12 +02:00
Björn Rabenstein 503ffba49a
chunkenc: Slightly optimize xorWrite/xoRead (#11476)
With these changes, the "happy path" when the leading and trailing
number of bits don't need an update, fewer operations are needed.

The change is probably very marginal (no change in the benchmark added
here, but the benchmark also doesn't cover non-changing values), and
an argument could me made that avoiding pointers also has its
benefits.

However, I think that reducing the number of return values improves
readability. Which convinced me that I should at least propose this.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-10-20 15:08:01 +05:30
Ganesh Vernekar 8ee4dfd40c
Fix the build after conflict resolution
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 17:59:42 +05:30
Ganesh Vernekar 648be89822
Merge remote-tracking branch 'upstream/main' into fix-conflict
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 14:20:02 +05:30
Ganesh Vernekar 8e29110949
Add/Improve unit tests for compaction with histogram (#11342)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 13:31:12 +05:30
Ganesh Vernekar 507bfa46fd
Fix HistogramChunk's AtFloatHistogram()
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 10:38:13 +05:30
Signed-off-by: Jesus Vazquez 3362bf6d79
Fix merge conflicts
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-11 22:53:37 +05:30
Jesus Vazquez 775d90d5f8
TSDB: Rename wal package to wlog (#11352)
The wlog.WL type can now be used to create a Write Ahead Log or a Write
Behind Log.

Before the prefix for wbl metrics was
'prometheus_tsdb_out_of_order_wal_' and has been replaced with
'prometheus_tsdb_out_of_order_wbl_'.

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2022-10-10 20:38:46 +05:30
Sonali Rajput 9165aedb49 Fixed broken link in tsdb README.md
Signed-off-by: Sonali Rajput <sonalirajput1088@gmail.com>
2022-10-07 16:20:20 +00:00
Jesus Vazquez e934d0f011 Merge 'main' into sparsehistogram
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2022-10-05 22:14:49 +02:00
Ganesh Vernekar d0a6488c74
Update metrics for histograms
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-03 13:48:59 +05:30
Bryan Boreham 9b31adc4e8
tsdb: fix up sort call with faster slices.Sort (#11380)
This call was added by PR #11075 merged before #11318 which changed all
similar calls to `sort.Sort` into a faster one.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-10-01 12:55:40 -04:00
Bryan Boreham 3330d85ba8
Replace sort.Strings and sort.Ints with faster slices.Sort (#11318)
Use new experimental package `golang.org/x/exp/slices`.

slices.Sort works on values that are directly comparable, like ints,
so avoids the overhad of an interface call to `.Less()`.

Left tests unchanged, because they don't need the speed and it may be
a cross-check that slices.Sort gives the same answer.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-30 20:03:56 +05:30
Bryan Boreham 7f2374b703
tsdb: faster postings sort with generic slices.Sort (#11054)
Use new experimental package `golang.org/x/exp/slices`.

Some of the speedup comes from comparing SeriesRef (which is an int64)
directly rather than through an interface `.Less()` call; some comes
from exp/slices using "pattern-defeating quicksort(pdqsort)".

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-30 20:01:32 +05:30
Ganesh Vernekar 83d738e263
Fix 'invalid magic number 0' bug (#11338)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-28 21:43:58 +05:30
Ganesh Vernekar f34aeefe6e
Allow overlapping blocks by default (#11331)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-28 19:17:54 +05:30
Robert Fratto 448cfda6c1
tsdb/agent: fix validation of default options (#9876)
* tsdb/agent: fix application of defaults

MaxTS was being incorrectly constrained to the truncation interval

* add more tests to check validation

* force MaxWALTime = MinWALTime if min > max

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
2022-09-27 19:41:43 +05:30
Bryan Boreham d166da7b59
tsdb: stop saving a copy of last 4 samples in memSeries (#11296)
* TSDB chunks: remove race between writing and reading

Because the data is stored as a bit-stream, the last byte in the stream
could change if the stream is appended to after an Iterator is obtained.
Copy the last byte when the Iterator is created, so we don't have to
read it later.

Clarify in comments that concurrent Iterator and Appender are allowed,
but the chunk must not be modified while an Iterator is created.
(This was already the case, in order to copy the bstream slice header.)

* TSDB: stop saving last 4 samples in memSeries

This extra copy of the last 4 samples was introduced to avoid a race
condition between reading the last byte of the chunk and writing to it.

But now we have fixed that by having `bstreamReader` copy the last byte,
we don't need to copy the last 4 samples.

This change saves 56 bytes per series, which is very worthwhile when
you have millions or tens of millions of series.

* TSDB: tidy up stopIterator re-use

Previous changes have left this code duplicating some lines; pull
them out to a separate function and tidy up.

* TSDB head_test: stop checking when iterators are wrapped

The behaviour has changed so chunk iterators are only wrapped when
transaction isolation requires them to stop short of the end.
This makes tests fail which are checking the type.

Tests should check the observable behaviour, not the type.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-27 19:32:05 +05:30
Bryan Boreham ff00dee262
tsdb: turn off transaction isolation for head compaction (#11317)
* tsdb: add a basic test for read/write isolation

* tsdb: store the min time with isolationAppender
So that we can see when appending has moved past a certain point in time.

* tsdb: allow RangeHead to have isolation disabled
This will be used when for head compaction.

* tsdb: do head compaction with isolation disabled
This saves a lot of work tracking appends done while compaction is ongoing.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-27 19:31:23 +05:30
Bryan Boreham d0607435a2
tsdb: remove chunkRange and oooCapMax from memSeries (#11288)
* tsdb: remove chunkRange from memSeries

chunkRange is the (oddly-named) configured duration for the head block.

We don't need a copy of this value per series. Pass it down where
required, and remove the copy.

The value in `Head` is only updated in `resetInMemoryState()`, which
also discards all `memSeries`.

* tsdb: remove oooCapMax from memSeries

oooCapMax is the configured maximum capacity for an out-of-order chunk.

Storing it per-series uses extra memory, and has surprising behaviour
if users change the value in config - series created before the change
will keep their old value.

Instead, pass it down where required, and remove the per-series value.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-27 13:52:22 +05:30
Ganesh Vernekar 758e29258b
Add/Improve unit tests for compaction with histogram Part 2 (#11343)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-23 14:01:10 +05:30
Jesus Vazquez c1b669bf9b
Add out-of-order sample support to the TSDB (#11075)
* Introduce out-of-order TSDB support

This implementation is based on this design doc:
https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing

This commit adds support to accept out-of-order ("OOO") sample into the TSDB
up to a configurable time allowance. If OOO is enabled, overlapping querying
are automatically enabled.

Most of the additions have been borrowed from
https://github.com/grafana/mimir-prometheus/
Here is the list ist of the original commits cherry picked
from mimir-prometheus into this branch:
- 4b2198d7ec
- 2836e5513f
- 00b379c3a5
- ff0dc75758
- a632c73352
- c6f3d4ab33
- 5e8406a1d4
- abde1e0ba1
- e70e769889
- df59320886

Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Dieter Plaetinck <dieter@grafana.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* gofumpt files

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Add license header to missing files

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix OOO tests due to existing chunk disk mapper implementation

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix truncate int overflow

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Add Sync method to the WAL and update tests

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* remove useless sync

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Update minOOOTime after truncating Head

* Update minOOOTime after truncating Head

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Add a unit test

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Load OutOfOrderTimeWindow only once per appender

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix OOO Head LabelValues and PostingsForMatchers

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix replay of OOO mmap chunks

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Remove unnecessary err check

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Prevent panic with ApplyConfig

Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Run OOO compaction after restart if there is OOO data from WBL

Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Apply Bartek's suggestions

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Refactor OOO compaction

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Address comments and TODOs

- Added a comment explaining why we need the allow overlapping
  compaction toggle
- Clarified TSDBConfig OutOfOrderTimeWindow doc
- Added an owner to all the TODOs in the code

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Run go format

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix remaining review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Change wbl reference when truncating ooo in TestHeadMinOOOTimeUpdate

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix TestWBLAndMmapReplay test failure on windows

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Address most of the feedback

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Refactor the block meta for out of order

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix windows error

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Dieter Plaetinck <dieter@grafana.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2022-09-20 22:35:50 +05:30
Bryan Boreham af6167df58
WAL loading: don't send empty buffers over chan (#11319)
If some shards did not get any samples mapped, the buffer will be empty
so sending it over the chan to `processWALSamples()` is a waste of time.

This is especially likely now we are checking `minValidTime` before
sending.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-20 19:43:30 +05:30
Ganesh Vernekar 2474c6fb2c
Error on amending histograms on append (#11308)
* Error on amending histograms on append

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Rename Matches to Equals

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-19 13:10:30 +05:30
Bryan Boreham d2701be53a
tsdb: remove chunk pool from memSeries (#11280)
The chunk pool belongs to the head not to the series. Pass it down where
required, and remove the copy of the pointer that `memSeries` was
holding.

`safeChunk` also needs to hold it, because in scenarios where it is used
we don't have a reference to the head. However it was already holding
`chunkDiskMapper` for the same reason, so no big change.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-15 13:22:09 +05:30
Björn Rabenstein 7ad36505d5
tsdb: Update comment about a possible space optimization (#11303)
See also #11195 for the detailed reasoning.

Signed-off-by: beorn7 <beorn@grafana.com>

Signed-off-by: beorn7 <beorn@grafana.com>
2022-09-15 13:11:57 +05:30
Bryan Boreham e49d596fb1
WAL loading: check sample time is valid earlier (#11307)
There's no point splitting a sample onto the right shard and checking
if the series needs to be re-mapped, if we're only going to discard it
once it arrives at `ProcessWALSamples()`. Simply discard it earlier.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-15 12:36:57 +05:30
Ganesh Vernekar d354f20c2a
Add a feature flag to control native histogram ingestion (#11253)
* Add runtime config to control native histogram ingestion

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Make the config into a CLI flag

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-14 17:38:34 +05:30
Ganesh Vernekar 83e11014dd
Remove unnecessary tsdb/tsdbutil/buffer.go (#11302)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-13 19:36:32 +05:30
Ganesh Vernekar b2d01cbc57
Remove unnecessary code in encoding/decoding histograms (#11252)
* Remove unnecessary code in encoding/decoding histograms

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-13 19:30:54 +05:30
Bryan Boreham 136f8b0ebb
tsdb: comment reason for isolation tracking reads (#11301)
I find it useful to know why a restriction exists, to check whether that
reason still applies, or in which other places it might apply.

This is based on the note here:
https://github.com/prometheus/prometheus/pull/9270#pullrequestreview-743820956
on the PR where the original comment was added.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-13 11:11:03 +02:00
Bryan Boreham 176fa38e76 tsdb: in tests use labels.FromStrings
Replacing code which assumes the internal structure of `Labels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-09 13:34:49 +02:00
Bryan Boreham 0437dd7cee tsdb/wal: in tests use labels.FromStrings
Replacing code which assumes the internal structure of `Labels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-09 13:34:49 +02:00
Julien Pivotto ec6c1f17d1
Update dependencies (#11287)
Updating dependencies following CI changes and move to go 1.19

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-09-09 13:28:55 +02:00
Julien Pivotto 96d5a32659
Update go to 1.19, set min version to 1.18 (#11279)
* Update go to 1.19, set min version to 1.18

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

* Update golangci-lint

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-09-07 11:30:48 +02:00