1250 commits

Author	SHA1	Message	Date
Björn Rabenstein	02d0de9987	Merge pull request #14997 from fionaliao/fl/update-format-docs ... Update chunk format docs with native histograms and OOO	2024-10-09 13:29:01 +02:00
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log ... For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-10-07 15:58:50 -04:00
Bryan Boreham	91de19fbef	[BUGFIX] TSDB: Don't read in-order chunks from before head MinTime ... Because we are reimplementing the `IndexReader` to fetch in-order and out-of-order chunks together, we must reproduce the behaviour of `Head.indexRange()`, which floors the minimum time queried at `head.MinTime()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-10-07 13:50:03 +01:00
Matthieu MOREL	ab64966e9d	fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094) ... * fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-10-06 16:35:29 +00:00
György Krajcsovits	44ebbb8458	Fix missing histogram copy in sampleRing ... The specialized version of sample add to the ring: func addH(s hSample, buf []hSample, r *sampleRing) []hSample func addFH(s fhSample, buf []fhSample, r *sampleRing) []fhSample already correctly copy histogram samples from the reused hReader, fhReader buffers, but the generic version does not. This means that the data is overwritten on the next read if the sample ring has seen histogram and float samples at the same time and switched to generic mode. The `genericAdd` function (which was commented anyway) is by now quite different from the specialized functions so that this commit deletes it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-02 13:57:28 +02:00
Bryan Boreham	54de4fb780	Merge pull request #14975 from colega/process-mempostings-delete-with-gomaxprocs-workers ... Process `MemPostings.Delete()` with `GOMAXPROCS` workers	2024-09-29 07:58:42 +01:00
Fiona Liao	fd62dbc291	Update chunk format docs with native histograms and OOO ... Signed-off-by: Fiona Liao <fiona.liao@grafana.com>	2024-09-27 18:57:58 +01:00
Ayoub Mrini	105ab2e95a	fix(test): adjust defer invocations (#14996) ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-27 17:13:51 +01:00
Oleg Zaytsev	ada8a6ef10	Add some more tests for MemPostings_Delete ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-27 10:14:39 +02:00
Arthur Silva Sens	d5f65cfce0	Merge pull request #14694 from prometheus/ct-histogram ... Histogram CT Zero ingestion	2024-09-26 12:48:46 -03:00
Arthur Silva Sens	95a53ef982	Join tests for appending float and histogram CTs ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-09-26 11:29:31 -03:00
Arthur Silva Sens	6bd9b1a7cc	Histogram CT Zero ingestion ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-09-26 11:29:22 -03:00
Oleg Zaytsev	4fd2556baa	Extract processWithBoundedParallelismAndConsistentWorkers ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-26 15:43:19 +02:00
Oleg Zaytsev	ccd0308abc	Don't do anything if MemPostings are empty ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-25 15:00:10 +02:00
Oleg Zaytsev	9c417aa710	Fix deadlock with empty MemPostings ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-25 14:08:50 +02:00
Bryan Boreham	5d8f0ef0c2	Merge pull request #14721 from bboreham/exp-grow-postings ... [PERF] TSDB: Grow postings by doubling	2024-09-25 10:47:55 +01:00
Oleg Zaytsev	e196b977af	Process MemPostings.Delete() with GOMAXPROCS workers ... We are still seeing lock contention on MemPostings.mtx, and MemPostings.Delete() is by far the most expensive operation on that mutex. This adds parallelism to that method, trying to reduce the amount of time we spend with the mutex held. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-25 10:38:47 +02:00
Bryan Boreham	ca673eb749	Merge remote-tracking branch 'origin/release-2.55' into merge-2.55-into-main ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-22 17:49:34 +01:00
Bryan Boreham	31c5760551	Neater string vs byte-slice conversions (#14425) ... unsafe.Slice and unsafe.StringData were added in Go 1.20 Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-21 12:19:21 +02:00
Bryan Boreham	d42232e178	Merge pull request #14932 from bboreham/chunk-xor-combine-writebits ... [PERF] TSDB: Chunk encoding: shorten some write sequences	2024-09-20 17:53:54 +01:00
Bryan Boreham	6f0d6038b7	[BUGFIX] TSDB: Only query chunks up to truncation time (#14948) ... If the query overlaps the range currently undergoing compaction, we should only fetch chunks up to that time. Need to store that min time in `HeadAndOOOIndexReader`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-20 17:44:04 +01:00
Bryan Boreham	9215252221	[BUGFIX] TSDB: Only query chunks up to truncation time (#14948) ... If the query overlaps the range currently undergoing compaction, we should only fetch chunks up to that time. Need to store that min time in `HeadAndOOOIndexReader`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-20 18:40:17 +02:00
Ganesh Vernekar	5ccb069414	Backward compatibility with upcoming index v3 ... Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2024-09-19 10:27:52 +01:00
George Krajcsovits	0d22a91267	Merge pull request #14874 from krajorama/fix-panic-in-ooo-query2 ... BUGFIX: TSDB: panic in chunk querier	2024-09-19 10:03:53 +01:00
Bryan Boreham	e8c2d916ec	lint ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-18 15:23:46 +01:00
Bryan Boreham	648a668835	[PERF] Chunk encoding: combine timestamp writes ... Instead of a 2-bit write followed by a 14-bit write, do two 8-bit writes, which goes much faster since it avoids looping. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-18 13:19:21 +01:00
Bryan Boreham	b9a9689aae	[PERF] Chunk encoding: simplify writeByte ... Rather than append a zero then set the value at that position, append the value. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-18 13:19:04 +01:00
Bryan Boreham	b65f1b6560	TSDB: Improve xor-chunk benchmarks ... Benchmarks must do the same work N times. Run 3 cases, where the values are constant, vary a bit, and vary a lot. Also aim for 120 samples same as TSDB default. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-18 13:14:49 +01:00
Bryan Boreham	bb47f78929	Merge pull request #14505 from marioferh/improve_performance_regex ... [CHANGE] regexp . to match \n and optimize performance	2024-09-18 09:54:16 +01:00
Antoine Pultier	d90d0976b5	fix(bstream/writeByte): ensure it appends only one byte (#14854) ... fix(bstream/writeByte): ensure it appends only one byte Signed-off-by: Antoine Pultier <antoine.pultier@sintef.no>	2024-09-17 16:28:33 +02:00
machine424	d1b4312f0a	fix(wlog/watcher_test.go): make TestRun_AvoidNotifyWhenBehind more resilient ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-17 13:11:04 +02:00
Mario Fernandez	5814920601	Fix: optimize .* regexp performance ... Shortcut for `.*` matches newlines as well. Add preamble change ^(?s: Add test dotAll flag por al regex Add and fix regex tests Signed-off-by: Mario Fernandez <mariofer@redhat.com>	2024-09-17 12:18:31 +02:00
Bryan Boreham	d5f4fabd12	Merge pull request #14911 from bboreham/clarify-postings-benchmark ... TSDB: Simplify benchmark regexps	2024-09-17 11:52:13 +02:00
Carrie Edwards	14e3c05ce8	tsdb: Add support for ingestion of out-of-order native histogram samples (#14546) ... Add support for ingesting OOO native histograms * Add flag for enabling and disabling OOO native histogram ingestion * Update OOO querying tests to include native histogram samples * Add OOO head tests * Add test for OOO native histogram counter reset headers Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored by: Carrie Edwards <edwrdscarrie@gmail.com> Co-authored by: Jeanette Tan <jeanette.tan@grafana.com> Co-authored by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored by: Fiona Liao <fiona.liao@grafana.com>	2024-09-17 11:19:06 +02:00
Harry John	919dc0cbc6	storage: Update LabelQuerier interface to return sorted label values (#14849) ... * Change LabelQuerier.LabelValues() to return sorted values --------- Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>	2024-09-17 08:55:02 +02:00
Bryan Boreham	a8133f3e87	TSDB: Simplify benchmark regexps ... Several regexps were coded like `"^.*$"`, which is an unnatural formulation nobody is likely to use. Inside `NewMatcher`, `^` and `$` are added anyway, which makes the form in the benchmark redundant. It even printed it out in the expected way. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-16 17:48:05 +01:00
George Krajcsovits	5aa3d8260a	TSDB: OOO native histograms: prep for multiple ooo head chunks (#14850) ... * tsdb: mmapCurrentOOOHeadChunk prepare for multiple ooo chunks Currently float samples can only create a single ooo head chunk, but native histograms can result in multiple due to counter resets, etc. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * tsdb: getOOOSeriesChunks prepare for multiple ooo chunks Currently float samples can only create a single ooo head chunk, but native histograms can result in multiple due to counter resets, etc. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-09-11 23:55:39 +01:00
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868) ... * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2024-09-10 22:32:03 +02:00
Bryan Boreham	16e5e99546	Merge pull request #14767 from bboreham/fix-encoding-comment ... [Comment] Correct the comment on Decbuf.UvarintBytes	2024-09-09 12:52:36 +01:00
György Krajcsovits	d3f4e7c223	Remove unnecessary conversion ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-09-09 12:51:02 +02:00
György Krajcsovits	60ab1cc5a5	BUGFIX: TSDB: panic in chunk querier ... Followup to #14831 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-09-09 12:43:02 +02:00
George Krajcsovits	536d9f9ce9	BUGFIX: TSDB: panic in query during truncation with OOO head (#14831) ... Check if headQuerier is nil before trying to use it. * TestQueryOOOHeadDuringTruncate: unit test to check query during truncate Regression test for #14822 * Simulate race between query and Compact() Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-09-05 17:17:42 +01:00
Antoine Pultier	f5971bf292	tsdb documenation: More details about chunks ... Signed-off-by: Antoine Pultier <45740+fungiboletus@users.noreply.github.com>	2024-09-04 14:57:30 +02:00
Joshua Hesketh	f2064c7987	NH: Do not re-use spans between histograms (#14771) ... promql, tsdb (histograms): Do not re-use spans between histograms When multiple points exist with the same native histogram schemas they share their spans. This causes a problem when a native histogram (NH) schema is modified (for example, during a Sum) then the other NH's with the same spans are also modified. As such, we should create a new Span for each NH. This will ensure NH's interfaces are safe to use without considering the effect on other histograms. At the moment this doesn't present itself as a problem because in all aggregations and functions operating on native histograms they are copied by the promql query engine first. Signed-off-by: Joshua Hesketh <josh@nitrotech.org> --------- Signed-off-by: Joshua Hesketh <josh@nitrotech.org>	2024-09-04 12:07:16 +02:00
George Krajcsovits	282fb1632a	Merge pull request #14772 from krajorama/fix-mockseriesiterator ... Fix: chunkenc.MockSeriesIterator	2024-09-03 16:55:26 +02:00
Arthur Silva Sens	442f24e099	chore: Simplify TestHeadAppender_AppendCTZeroSample (#14812) ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-09-02 21:30:37 +01:00
Arve Knudsen	2cfc7b244a	Merge pull request #14700 from shandongzhejiang/main ... Comments: fix some function names	2024-09-02 18:59:28 +02:00
Oleg Zaytsev	ce7d830f1f	Bring back BenchmarkLoadRealWLs (#14757) ... This was part of #14525 which was reverted. I still think that having this benchmark committed in to the repo is useful. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-09-02 17:20:10 +01:00
György Krajcsovits	a693dd19f2	Fix: chunkenc.MockSeriesIterator ... Starts its index from 0 , but users call Next() before first sample so it needs to start from -1 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-30 16:44:36 +02:00
Bryan Boreham	0a4f130b39	[Comment] Correct the comment on Decbuf.UvarintBytes ... The value is valid when returned, but can become invalid later. Return to previous wording. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-30 09:40:18 +01:00
Callum Styan	a77f5007f9	fix bug with metadata for rw2 (#14766) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	2024-08-30 08:14:20 +01:00
Bryan Boreham	1f38ae7bca	[TESTS] TSDB: fix up OOO tests for new Series behaviour ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-29 10:59:09 +01:00
Bryan Boreham	cde42f30e9	TSDB: streamline reading of overlapping head chunks ... `getOOOSeriesChunks` was already finding sets of overlapping chunks; we store those in a `multiMeta` struct so that `ChunkOrIterable` can reconstruct an `Iterable` easily and predictably. We no longer need a `MergeOOO` flag to indicate that this Meta should be merged with other ones; this is explicit in the `multiMeta` structure. We also no longer need `chunkMetaAndChunkDiskMapperRef`. Add `wrapOOOHeadChunk` to defeat `chunkenc.Pool` - chunks are reset during compaction, but if we wrap them (like `safeHeadChunk` was doing then this is skipped) . Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-29 10:57:29 +01:00
Bryan Boreham	838e49e7b8	[REFACTOR] TSDB: move chunkFromSeries from headChunkReader to head ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-29 10:51:48 +01:00
Björn Rabenstein	1d6e0071b7	Merge pull request #14751 from riskrole/main ... chore: fix some comments	2024-08-28 16:38:39 +02:00
riskrole	406bf775aa	chore: fix some comments ... Signed-off-by: riskrole <yuhang@before.tech>	2024-08-28 11:26:57 +08:00
Marco Pracucci	ef649d5968	Revert " Store mmMaxTime in same field as seriesShard" ... Signed-off-by: Marco Pracucci <marco@pracucci.com>	2024-08-26 08:56:16 +02:00
Bryan Boreham	33adbe47b1	[PERF] TSDB: Grow postings by doubling ... Go's built-in append() grows larger slices with factor 1.3, which means we do a lot more allocating and copying for larger postings. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-24 11:16:58 +01:00
György Krajcsovits	183bbc39a2	Make requesting merge with OOO head explicit in chunk.Meta ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-23 15:50:53 +02:00
György Krajcsovits	41c076196e	New cases in Test_ChunkQuerier_OOOQuery and Test_Querier_OOOQuery ... Case 1: OOO in-memory head chunk overlaps with first mmaped in-order chunk. Query: |----------------------------------------------------------------| InO: |------mmap---------------||---------mem----------------------| OOO: |-----mem-----------| This triggers ChunkOrIterableWithCopy not including OOO head chunks bug. Similar to #14693 however testing the end of the interval doesn't trigger the problem because there the in-order head chunk will be trimmed with a tombstone, causing the code to switch to ChunkOrIterable which was fixed. See a36d1a8a92/tsdb/querier.go (L646) where len(p.bufIter.Intervals) will be non zero, because it includes the tombstone to trim the result to the query max time. Thus a new test is added to check the overlap at the beginning of the interval that has a separate chunk, which does not need trimming. Note: same test doesn't fail for sample querier in Test_Querier_OOOQuery as that doesn't use copy, that is copyHeadChunk is false in the if condition above. Case 2: OOO mmaped head chunk overlaps with first mmaped in-order chunk. Query: |----------------------------------------------------------------| InO: |------mmap---------------||---------mem----------------------| OOO: |-----mmap-----------| |--mem--| In this case the meta contains the reference of the in-order chunk and no indication that a merge is needed with the OOO mmaped chunk. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-23 15:50:47 +02:00
Arve Knudsen	b0aba26ed5	tsdb: Fix ValNone typo in comment ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-23 08:20:20 +02:00
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments ... Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2024-08-22 17:36:11 +02:00
Bryan Boreham	9a74d53935	[BUGFIX] TSDB: Fix query overlapping in-order and ooo head (#14693) ... * tsdb: Unit test query overlapping in order and ooo head Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * TSDB: Merge overlapping head chunk The basic idea is that getOOOSeriesChunks can populate Meta.Chunk, but since it only returns one Meta per overlapping time-slot, that pointer may end up in a Meta with a head-chunk ID. So we need HeadAndOOOChunkReader.ChunkOrIterable() to call mergedChunks in that case. Previously, mergedChunks was checking that meta.Ref was a valid OOO chunk reference, but it never actually uses that reference; it just finds all chunks overlapping in time. So we can delete that code. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-21 14:24:20 +01:00
shandongzhejiang	b2712ff284	chore: fix some function names ... Signed-off-by: shandongzhejiang <shandongzhejiang@icloud.com>	2024-08-21 11:09:37 +08:00
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-18 12:13:25 +02:00
Bryan Boreham	87dccb1d1b	Merge pull request #14649 from machine424/ftest ... fix(tsdb/db_test.go): close the corrupted chunk after creating it to satisfy Windows FS	2024-08-16 11:57:54 +01:00
Arve Knudsen	66388f706a	Merge pull request #14042 from aknuds1/arve/wlog-histograms ... tsdb/wlog: Only treat unknown record types as failure	2024-08-16 12:00:49 +02:00
Björn Rabenstein	1daf7cdd62	Merge pull request #14626 from cuiweiyuan/main ... chore: fix some function names	2024-08-15 11:46:21 +02:00
cuiweiyuan	1800af54f0	chore: fix some function names ... Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2024-08-15 13:57:21 +08:00
Arve Knudsen	b5d13a1ab5	Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-14 19:04:53 +01:00
Bryan Boreham	512c67ec26	TSDB: Never go over maximum number of OOO chunks ... In `mmapCurrentOOOHeadChunk`, check if the number is at the maximum and drop the data with an error log. This is not expected to happen as the maximum is over 8 million; that's 8 years of 1 sample every second. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:59 +01:00
Bryan Boreham	9135da1e4f	TSDB: Review feedback ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Re-enable check in `createHeadWithOOOSamples` which wasn't really broken. * Move code making `Block` into a `Queryable` into test file. * Make `getSeriesChunks` return a slice (renamed `appendSeriesChunks`). * Rename `oooMergedChunks` to `mergedChunks`. * Improve comment on `ChunkOrIterableWithCopy`. * Name return values from unpackHeadChunkRef. Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:44 +01:00
Bryan Boreham	7ffd3ca280	TSDB: Cosmetic: move HeadAndOOO implementations where old code was ... This makes the diffs easier to follow. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	e95607b276	TSDB: Lock round access to labels, where necessary ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	26b3de0438	TSDB: Remove OOOHeadIndexReader ... Use headIndexReader instead. OOOCompactionHeadIndexReader needs to be expanded slightly, because it previously delegated to OOOHeadIndexReader. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	a299c7b6d6	TSDB: Remove OOOHeadChunkReader ... Use HeadAndOOOChunkReader instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	e7e50a3afd	TSDB: Remove code for querying OOO-head only ... Just query via `HeadAndOOOQuerier`, which will skip series where no in-order chunks are in range. Now we don't need `OOORangeHead`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	0a2ff76881	TSDB tests: Fix up BenchmarkQueries ... Was not working even on main. Some cases still error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:04 +01:00
Bryan Boreham	f261597944	TSDB: Fix up LabelValues to work for OOO-only head ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	6529d6336c	TSDB: NewHeadAndOOOChunkReader takes headChunkReader ... So we can pass nil and have it read just OOO chunks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	e04d137649	[PERF] TSDB: Query head and ooo-head together ... Add `HeadAndOOOQuerier` which iterates just once over series, then where necessary merges chunks from in-order and out-of-order lists. Add a ChunkQuerier for in-order and ooo together Add copy-last-chunk behaviour to HeadAndOOOChunkReader Out-of-order chunk IDs are distinguished from in-order by setting bit 23. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	da31da3ea6	Refactor: extract selectSeriesSet and selectChunkSeriesSet ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	7e24844d08	Refactor: extract headChunkReader.chunkFromSeries() ... For when you have a series locked already. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	a32aca0cd7	Refactoring: extract getOOOSeriesChunks ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	c75c8f8329	Refactoring: extract getSeriesChunks ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	0c852680bf	[Benchmark] TSDB: Add BenchmarkQuerierSelectWithOutOfOrder ... Refactor existing BenchmarkQuerierSelect to provide the set-up. Note that Head queries now run faster because they use a RangeHead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
György Krajcsovits	41656162fc	tsdb: prepare inserting native histograms into OOO head ... Rename a variable. Add parameters to memSeries.insert function. No effect on how float samples are handled. Related to #14546 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-14 11:13:47 +02:00
Bryan Boreham	aa4b056ad0	Merge pull request #13200 from bboreham/wlog-defer ... tsdb/wlog: close segment files sooner	2024-08-13 14:11:38 +01:00
machine424	82f38d3e9a	fix(tsdb/db_test.go): close the corrupted chunk after creating it to satisfy Windows FS ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-08-09 14:53:57 +02:00
George Krajcsovits	cf62fb5c44	Merge pull request #14629 from krajorama/fix-to-encoded-chunks ... Fix ToEncodedChunks minT for recoded chunks	2024-08-08 20:00:31 +02:00
György Krajcsovits	1ea3781699	Fix ToEncodedChunks minT for recoded chunks ... Discovered while working on #14546 OOO native histograms. Not triggered on main before #14546 as the code path is unused. There was a bug where the min time of a chunk was adjusted even if it was only recoded and not completely new. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-08 15:22:46 +02:00
Ben Ye	b7a58dcf3d	Add hidden flag to disable overlapping compaction (#14581) ... TSDB: add hidden flag to disable overlapping compaction Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com>	2024-08-08 12:09:39 +02:00
George Krajcsovits	3a673cd0bc	Merge pull request #14598 from krajorama/fix-compaction-panic ... Fix: panic: runtime error: index out of range [4] with length 4	2024-08-07 17:14:14 +02:00
machine424	92873d3009	feat: allow to delay head compaction start time helping Prometheus instances to ... avoid simultaneous compactions and reduce stress on shared resources. This is enabled via `--enable-feature=delayed-compaction`. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-08-07 17:10:27 +02:00
Oleg Zaytsev	0833d2a230	Fix appendable: check whether last val was a histogram (#14613) ... * Fix appendable: check whether last val was a histogram When appending a float, we were checking whether lastValue was equal to current value, but we didn't check whether last value was a float value. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-08-07 15:02:59 +02:00
György Krajcsovits	98ecdf3589	Fix corrupting spans via iterator sharing ... Iterator may share spans without copy, so we always have to make a copy before modification - copy-on-write. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 16:51:20 +02:00
György Krajcsovits	d2f6fa7289	Fix lint error ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 13:24:46 +02:00
György Krajcsovits	1b6d1366d8	Fix re-code histogram and chunk re-code conflict ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 13:09:17 +02:00
György Krajcsovits	aff089a014	Reproduce recoding bug with new and missing buckets ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 10:51:44 +02:00
Bryan Boreham	80adc5baf4	Merge remote-tracking branch 'origin/main' into merge-2.54-to-main	2024-08-06 09:19:55 +01:00