1122 commits

Author	SHA1	Message	Date
Björn Rabenstein	1daf7cdd62	Merge pull request #14626 from cuiweiyuan/main ... chore: fix some function names	2024-08-15 11:46:21 +02:00
cuiweiyuan	1800af54f0	chore: fix some function names ... Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	2024-08-15 13:57:21 +08:00
Bryan Boreham	512c67ec26	TSDB: Never go over maximum number of OOO chunks ... In `mmapCurrentOOOHeadChunk`, check if the number is at the maximum and drop the data with an error log. This is not expected to happen as the maximum is over 8 million; that's 8 years of 1 sample every second. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:59 +01:00
Bryan Boreham	9135da1e4f	TSDB: Review feedback ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Re-enable check in `createHeadWithOOOSamples` which wasn't really broken. * Move code making `Block` into a `Queryable` into test file. * Make `getSeriesChunks` return a slice (renamed `appendSeriesChunks`). * Rename `oooMergedChunks` to `mergedChunks`. * Improve comment on `ChunkOrIterableWithCopy`. * Name return values from unpackHeadChunkRef. Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:44 +01:00
Bryan Boreham	7ffd3ca280	TSDB: Cosmetic: move HeadAndOOO implementations where old code was ... This makes the diffs easier to follow. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	e95607b276	TSDB: Lock round access to labels, where necessary ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	26b3de0438	TSDB: Remove OOOHeadIndexReader ... Use headIndexReader instead. OOOCompactionHeadIndexReader needs to be expanded slightly, because it previously delegated to OOOHeadIndexReader. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	a299c7b6d6	TSDB: Remove OOOHeadChunkReader ... Use HeadAndOOOChunkReader instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	e7e50a3afd	TSDB: Remove code for querying OOO-head only ... Just query via `HeadAndOOOQuerier`, which will skip series where no in-order chunks are in range. Now we don't need `OOORangeHead`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:13 +01:00
Bryan Boreham	0a2ff76881	TSDB tests: Fix up BenchmarkQueries ... Was not working even on main. Some cases still error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 13:41:04 +01:00
Bryan Boreham	f261597944	TSDB: Fix up LabelValues to work for OOO-only head ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	6529d6336c	TSDB: NewHeadAndOOOChunkReader takes headChunkReader ... So we can pass nil and have it read just OOO chunks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	e04d137649	[PERF] TSDB: Query head and ooo-head together ... Add `HeadAndOOOQuerier` which iterates just once over series, then where necessary merges chunks from in-order and out-of-order lists. Add a ChunkQuerier for in-order and ooo together Add copy-last-chunk behaviour to HeadAndOOOChunkReader Out-of-order chunk IDs are distinguished from in-order by setting bit 23. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	da31da3ea6	Refactor: extract selectSeriesSet and selectChunkSeriesSet ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	7e24844d08	Refactor: extract headChunkReader.chunkFromSeries() ... For when you have a series locked already. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	a32aca0cd7	Refactoring: extract getOOOSeriesChunks ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	c75c8f8329	Refactoring: extract getSeriesChunks ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Bryan Boreham	0c852680bf	[Benchmark] TSDB: Add BenchmarkQuerierSelectWithOutOfOrder ... Refactor existing BenchmarkQuerierSelect to provide the set-up. Note that Head queries now run faster because they use a RangeHead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
György Krajcsovits	41656162fc	tsdb: prepare inserting native histograms into OOO head ... Rename a variable. Add parameters to memSeries.insert function. No effect on how float samples are handled. Related to #14546 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-14 11:13:47 +02:00
Bryan Boreham	aa4b056ad0	Merge pull request #13200 from bboreham/wlog-defer ... tsdb/wlog: close segment files sooner	2024-08-13 14:11:38 +01:00
George Krajcsovits	cf62fb5c44	Merge pull request #14629 from krajorama/fix-to-encoded-chunks ... Fix ToEncodedChunks minT for recoded chunks	2024-08-08 20:00:31 +02:00
György Krajcsovits	1ea3781699	Fix ToEncodedChunks minT for recoded chunks ... Discovered while working on #14546 OOO native histograms. Not triggered on main before #14546 as the code path is unused. There was a bug where the min time of a chunk was adjusted even if it was only recoded and not completely new. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-08 15:22:46 +02:00
Ben Ye	b7a58dcf3d	Add hidden flag to disable overlapping compaction (#14581) ... TSDB: add hidden flag to disable overlapping compaction Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com>	2024-08-08 12:09:39 +02:00
George Krajcsovits	3a673cd0bc	Merge pull request #14598 from krajorama/fix-compaction-panic ... Fix: panic: runtime error: index out of range [4] with length 4	2024-08-07 17:14:14 +02:00
machine424	92873d3009	feat: allow to delay head compaction start time helping Prometheus instances to ... avoid simultaneous compactions and reduce stress on shared resources. This is enabled via `--enable-feature=delayed-compaction`. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-08-07 17:10:27 +02:00
Oleg Zaytsev	0833d2a230	Fix appendable: check whether last val was a histogram (#14613) ... * Fix appendable: check whether last val was a histogram When appending a float, we were checking whether lastValue was equal to current value, but we didn't check whether last value was a float value. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-08-07 15:02:59 +02:00
György Krajcsovits	98ecdf3589	Fix corrupting spans via iterator sharing ... Iterator may share spans without copy, so we always have to make a copy before modification - copy-on-write. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 16:51:20 +02:00
György Krajcsovits	d2f6fa7289	Fix lint error ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 13:24:46 +02:00
György Krajcsovits	1b6d1366d8	Fix re-code histogram and chunk re-code conflict ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 13:09:17 +02:00
György Krajcsovits	aff089a014	Reproduce recoding bug with new and missing buckets ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-06 10:51:44 +02:00
Bryan Boreham	80adc5baf4	Merge remote-tracking branch 'origin/main' into merge-2.54-to-main	2024-08-06 09:19:55 +01:00
machine424	9e43ad2e37	chore(remote_write): clean up as watcher.go is part of wlog now ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-08-05 13:40:23 +02:00
Bryan Boreham	015638c4b6	[BUGFIX] TSDB: Exclude OOO chunks mapped after compaction starts ... Otherwise the writer can end up with invalid chunks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-05 10:35:34 +01:00
Bryan Boreham	bded853035	[Test] TSDB: TestOOOCompaction with samples added after compaction starts ... Test fails due to bug. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-05 10:35:34 +01:00
George Krajcsovits	00ab05c3b9	Native histograms: fix spurios counter reset when merging recoded chunk to normal chunk (#14513) ... * chunkenc: allow missing empty buckets on histogram append Allow appending to chunks when the histogram to be added is missing some buckets, but the missing buckets are empty in the chunk. For example bucket at index 5 is present in the chunk, but its value is 0 and the new histogram doesn't have a bucket at index 5. This fixes an issue of merging chunks where one chunk was recoded to retroactively have some empty buckets in all the histograms and we are merging in a histogram that doesn't have the empty bucket (because it was not recoded yet). The operation alters the histogram that is being added, however this has already been the case when appending gauge histograms. Thus the test TestHistogramSeriesToChunks in storage package is changed to explicitly test what happened to the appended histogram - Compact(0) call is removed. The new expandIntSpansAndBuckets and expandFloatSpansAndBuckets functions are a merge of expandSpansForward and counterResetInAnyBucket and counterResetInAnyFloatBucket. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-08-01 09:22:32 +02:00
Bartlomiej Plotka	6816149852	Merge pull request #14525 from colega/merge-mmmaxtime-into-shardhash ... Store `mmMaxTime` in same field as `seriesShard`	2024-07-31 08:39:38 +02:00
Max Amin	84b819a69f	feat: add Google cloud roundtripper for remote write (#14346) ... * feat: Google Auth for remote write Signed-off-by: Max Amin <maxamin@google.com> --------- Signed-off-by: Max Amin <maxamin@google.com>	2024-07-30 16:25:19 +01:00
Oleg Zaytsev	0300ad58a9	Revert the option regardless of error ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-07-30 11:31:31 +02:00
Oleg Zaytsev	d8e1b6bdfd	Store mmMaxTime in same field as seriesShard ... We don't use seriesShard during DB initialization, so we can use the same 8 bytes to store mmMaxTime, and save those during the rest of the lifetime of the database. This doesn't affect CPU performance. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-07-30 10:20:29 +02:00
Oleg Zaytsev	b7f2f3c3ac	Add BenchmarkLoadRealWLs ... This benchmark runs on real WLs rather than fake generated ones. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-07-30 10:19:56 +02:00
Filip Petkovski	be7a4c9b83	Ignore stale histograms for counter reset detection ... The histogram stats decoder keeps track of the last seen histogram sample in order to properly detect counter resets. We are seeing an issue where a histogram with UnknownResetHint gets treated as a counter reset when it follows a stale histogram sample. I believe that this is incorrect since stale samples should be completely ignored in PromQL. As a result, they should not be stored in the histogram stats iterator and the counter reset detection needs to be done against the last non-stale sample. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2024-07-26 10:08:31 +02:00
Bryan Boreham	bdfeae6489	Merge pull request #14474 from bboreham/shrink-memseries ... TSDB: shrink memSeries by moving bools together	2024-07-18 12:15:57 +01:00
Carrie Edwards	a28d1974b4	Merge pull request #14438 from prometheus/cedwards/ooo-chunk-encoding ... tsdb: Add support for handling multiple chunks in OOO head	2024-07-16 11:12:23 -07:00
Carrie Edwards	02f05cbf2c	Ensure lastMmapRef set to 0 if mmapRefs is empty ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	2024-07-16 10:54:06 -07:00
Carrie Edwards	79b53bd3de	Refactor TestWBLReplay to use scenarios ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com> Co-authored by: Fiona Liao <fiona.liao@grafana.com>:	2024-07-16 10:53:28 -07:00
Carrie Edwards	7e020bb4e9	Refactor ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	2024-07-16 10:53:23 -07:00
Carrie Edwards	e0d7cef545	Rename some variables in Commit() ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	2024-07-16 10:53:23 -07:00
Carrie Edwards	fb2ebe1c3d	Fix linting ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	2024-07-16 10:53:23 -07:00
Carrie Edwards	2e0e4e9ce9	Add support for handling multiple chunks in OOO head ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com> Co-authored by: Jeanette Tan <jeanette.tan@grafana.com>: Co-authored by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>: Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com> Co-authored by: Fiona Liao <fiona.liao@grafana.com>:	2024-07-16 10:53:09 -07:00
Bryan Boreham	d878146c70	TSDB: shrink memSeries by moving bools together ... In each case the following member requires 8-byte alignment, so moving one beside the other shrinks memSeries from 176 to 168 bytes, when compiled with `-tags stringlabels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-07-15 16:09:48 +01:00