1243 commits

Author	SHA1	Message	Date
Oleg Zaytsev	9ad93ba8df	Optimize l=~".+" matcher (#15474) ... Since dot is matching newline now, `l=~".+"` is "any non empty label value", and #14144 added a specific method in the index for that so we don't need to run the matcher on each one of the label values. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-11-27 12:33:20 +01:00
Bryan Boreham	ca3119bd24	TSDB: eliminate one yolostring ... When the only use of a []byte->string conversion is as a map key, Go doesn't allocate. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-26 17:21:55 +00:00
Bryan Boreham	e98c19c1ce	[PERF] TSDB: Cache all symbols for compaction ... Trade a bit more memory for a lot less CPU spent looking up symbols. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-26 17:21:55 +00:00
Oleg Zaytsev	9aa6e041d3	MemPostings: allocate ListPostings once in PFALV (#15465) ... Same as #15427 but for the new method added in #14144 Instead of allocating each ListPostings one by one, allocate them all in one go. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-11-26 16:03:45 +01:00
DC	d535d501d1	[DOCS] Improve description of WAL record format (#14936) ... Signed-off-by: DC <413331538@qq.com>	2024-11-26 11:48:17 +00:00
Bryan Boreham	dd0252a774	Merge pull request #15380 from bboreham/improve-loadwbl ... [BUGFIX] TSDB: Apply fixes from loadWAL to loadWBL	2024-11-25 17:31:49 +00:00
Bryan Boreham	7996a13fdd	Merge pull request #15403 from bboreham/fix-rw-benchmark-startup ... [TESTS] Remote-Write: Fix BenchmarkStartup	2024-11-25 17:31:24 +00:00
Oleg Zaytsev	cc390aab64	MemPostings: allocate ListPostings once in PFLM (#15427) ... Instead of allocating ListPostings pointers one by one, allocate a slice and take pointers from that. It's faster, and also generates less garbage (NewListPostings is one of the top offenders in number of allocations). Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-11-20 17:52:20 +01:00
Arve Knudsen	89bbb885e5	Upgrade to golangci-lint v1.62.0 (#15424) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-11-20 17:22:20 +01:00
Björn Rabenstein	384c5951ef	Merge pull request #14489 from harry671003/implement_metadata_limit ... storage: Implement limit in mergeGenericQuerier	2024-11-19 17:32:16 +01:00
Arve Knudsen	06d54fcc6c	[PERF] TSDB: Optimize inverse matching (#14144) ... Simple follow-up to #13620. Modify `tsdb.PostingsForMatchers` to use the optimized tsdb.IndexReader.PostingsForLabelMatching method also for inverse matching. Introduce method `PostingsForAllLabelValues`, to avoid changing the existing method. The performance is much improved for a subset of the cases; there are up to ~60% CPU gains and ~12.5% reduction in memory usage. Remove `TestReader_InversePostingsForMatcherHonorsContextCancel` since `inversePostingsForMatcher` only passes `ctx` to `IndexReader` implementations now. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-11-19 15:49:01 +00:00
Bryan Boreham	0ef0b75a4f	[TESTS] Remote-Write: Fix BenchmarkStartup ... It was crashing due to uninitialized metrics, and not terminating due to incorrectly reading segment names. We need to export `SetMetrics` to avoid the first problem. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-15 11:22:07 +00:00
Fiona Liao	c599d37668	Always return unknown hint for first sample in non-gauge histogram chunk (#15343) ... Always return unknown hint for first sample in non-gauge histogram chunk --------- Signed-off-by: Fiona Liao <fiona.liao@grafana.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-11-12 15:14:06 +01:00
Bryan Boreham	5450e6d368	[BUGFIX] TSDB: Apply fixes from loadWAL to loadWBL ... Move a couple of variables inside the scope of a goroutine, to avoid data races. Use `zeropool` to reduce garbage and avoid some lint warnings. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-11 18:41:33 +00:00
Ben Ye	140f4aa9ae	feat: Allow customizing TSDB postings decoder (#13567) ... * allow customizing TSDB postings decoder --------- Signed-off-by: Ben Ye <benye@amazon.com>	2024-11-11 07:59:24 +01:00
Ben Ye	f9057544cb	Fix AllPostings added twice (#13893) ... * handle all postings added twice --------- Signed-off-by: Ben Ye <benye@amazon.com>	2024-11-10 18:17:21 +01:00
🌲 Harry 🌊 John 🏔	f9bc50b247	storage: Implement limit in mergeGenericQuerier ... Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>	2024-11-07 09:08:23 -08:00
Bryan Boreham	f42b37ff2f	[BUGFIX] TSDB: Fix race on stale values in headAppender (#15322) ... * [BUGFIX] TSDB: Fix race on stale values in headAppender Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Simplify Signed-off-by: Bryan Boreham <bjboreham@gmail.com> --------- Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-06 16:51:39 +01:00
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-11-06 16:50:36 +01:00
Bryan Boreham	02aa6d1de6	Merge pull request #15338 from bboreham/cosmetic-tsdb ... [COMMENT] Remove duplicate line	2024-11-05 12:03:04 +00:00
Oleg Zaytsev	b1e4052682	MemPostings.Delete(): make pauses to unlock and let the readers read (#15242) ... This introduces back some unlocking that was removed in #13286 but in a more balanced way, as suggested by @pracucci. For TSDBs with a lot of churn, Delete() can take a couple of seconds, and while it's holding the mutex, reads and writes are blocked waiting for that mutex, increasing the number of connections handled and memory usage. This implementation pauses every 4K labels processed (note that also compared to #13286 we're not processing all the label-values anymore, but only the affected ones, because of #14307), makes sure that it's possible to get the read lock, and waits for a few milliseconds more. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>	2024-11-05 12:59:57 +01:00
Bryan Boreham	541c7fd9fe	[COMMENT] Remove duplicate line ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-05 11:03:40 +00:00
Alban Hurtaud	4b56af7eb8	Add hidden flag for the delayed compaction random time window (#14919) ... * Add hidden flag for the delayed compaction random time window Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Update cmd/prometheus/main.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com> * Update cmd/prometheus/main.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com> * Update tsdb/db.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com> * Fix flag name according to review - add test for delay Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Fix afer main rebase Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Implement review comments Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Update generatedelaytest to try with limit values Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> --------- Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>	2024-11-04 08:26:26 +01:00
Bryan Boreham	2fbbfc3da8	Revert "Fix MemPostings.Add and MemPostings.Get data race (#15141)" ... This reverts commit 50ef0dc954. Memory allocation goes so high in Prombench that the system is unusable. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-03 12:30:34 +00:00
Bryan Boreham	e2e01c1cff	Merge pull request #15216 from yeya24/log-last-series-labels ... log last series labelset when hitting OOO series labels	2024-11-01 14:15:39 +00:00
Oleg Zaytsev	ba11a55df4	Revert "Process MemPostings.Delete() with GOMAXPROCS workers" ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-10-29 17:13:40 +01:00
Nicolas Takashi	b6c538972c	[REFACTORY] simplify appender commit (#15112) ... * [REFACTOR] simplify appender commit Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com> Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-10-29 12:34:02 +00:00
Arve Knudsen	706dcfeecf	tsdb.CircularExemplarStorage: Avoid racing (#15231) ... * tsdb.CircularExemplarStorage: Avoid racing --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-10-29 10:40:46 +01:00
Pedro Tanaka	bab587b9dc	Agent: allow for ingestion of CT samples (#15124) ... * Remove unused option from HeadOptions Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Improve docs for appendable() method in head appender Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Ingest CT (float) samples in Agent DB Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * allow for ingestion of CT native histogram Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * adding some verification for ct ts Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Validating CT histogram before append and add newly created series to pending series Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * checking the wal for written samples Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Checking for samples in test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * adding case for validations Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fixing comparison when dedupelabels is enabled Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * unite tests, use table testing Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Implement CT related methods in timestampTracker for write storage Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * adding error case to test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * removing unused fields Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Updating lastTs for series when adding CT to invalidate duplicates Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * making sure that updating the lastTS wont cause OOO later on in Commit(); Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> --------- Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>	2024-10-27 01:06:34 +01:00
Ayoub Mrini	93db81dd3d	Merge pull request #14983 from machine424/dopp ... fix(storage/mergeQuerier): fix a data race	2024-10-25 18:34:51 +02:00
Ben Ye	99882eec3b	log last series labelset when hitting OOO series labels during compaction ... Signed-off-by: Ben Ye <benye@amazon.com>	2024-10-24 09:27:15 -07:00
Vanshika	cccbe72514	TSDB: Fix some edge cases when OOO is enabled (#14710) ... Fix some edge cases when OOO is enabled Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	2024-10-23 17:34:28 +02:00
machine424	cebcdce78a	fix(storage/mergeQuerier): copy the matcjers slice before passing it to queriers as ... some of them may alter it. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-10-22 14:08:47 +02:00
machine424	eb523a6b29	fix(storage/mergeQuerier): add a reproducer for data race that occurs when one of the queriers alters the passed matchers and propose a fix ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-10-22 14:08:46 +02:00
György Krajcsovits	a4083f14e8	Fix populateWithDelChunkSeriesIterator corrupting chunk meta ... When handling recoded histogram chunks the min time of the chunk is updated by mistake. It should only update when the chunk is completely new. Otherwise the ongoing chunk's meta will be later than the previously written samples in it. Same bug as https://github.com/prometheus/prometheus/pull/14629 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-18 10:34:22 +02:00
György Krajcsovits	e6a682f046	Reproduce populateWithDelChunkSeriesIterator corrupting chunk meta ... When handling recoded histogram chunks the min time of the chunk is updated by mistake. It should only update when the chunk is completely new. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-18 10:34:22 +02:00
machine424	ab2475c426	test(tsdb): add a reproducer for https://github.com/prometheus/prometheus/issues/14422 ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-10-15 20:39:25 +02:00
Bryan Boreham	1e1f6ab9df	Merge pull request #15120 from bboreham/floor-ino-mint ... [BUGFIX] TSDB: Don't read in-order chunks from before head MinTime	2024-10-15 10:27:38 +01:00
George Krajcsovits	b8867f8ead	Merge pull request #15142 from krajorama/fix-appendhistogram-race ... bugfix: data race in head.Appender.AppendHistogram and Commit	2024-10-14 08:13:39 +02:00
Oleg Zaytsev	50ef0dc954	Fix MemPostings.Add and MemPostings.Get data race (#15141) ... * Tests for Mempostings.{Add,Get} data race * Fix MemPostings.{Add,Get} data race We can't modify the postings list that are held in MemPostings as they might already be in use by some readers. * Modify BenchmarkHeadStripeSeriesCreate to have common labels If there are no common labels on the series, we don't excercise the ordering part of MemSeries, as we're just creating slices of one element for each label value. --------- Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2024-10-11 15:21:15 +02:00
György Krajcsovits	bb70370d72	TSDB head: fix race between AppendHistogram and Commit ... Move writing memSeries lastHistogramValue and lastFloatHistogramValue after series creation under lock. The resulting code isn't totally correct in the sense that we're setting these values before Commit() , so they might be overwritten/rolled back later. Also Append of stale sample checks the values without lock, so there's still a potential race. The correct solution would be to set these only in Commit() which we actually do, but then Commit() would also need to process samples in order and not floats first, then histograms, then float histograms - which leads to not knowing what stale marker to write for histograms. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-10 16:59:15 +02:00
György Krajcsovits	631fadc4ca	Unit test for data race in head.Appender.AppendHistogram ... Two Appenders race when creating a series with a native histogram as the memSeries will be common and the lastHistogram field is written without lock. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-10 14:10:07 +02:00
beorn7	12c39d5421	docs: Some nitpicking in chunks.md ... - `float histogram` → `floathistogram`, as it is used in the code. - Actual link encodings to the code (to find the actual numerical values). - `<bytes>` → `<data>` for consistency. Signed-off-by: beorn7 <beorn@grafana.com>	2024-10-09 14:32:12 +02:00
beorn7	a4cb52ff15	docs: Update chunk layot for NHCB ... Signed-off-by: beorn7 <beorn@grafana.com>	2024-10-09 14:19:20 +02:00
Björn Rabenstein	02d0de9987	Merge pull request #14997 from fionaliao/fl/update-format-docs ... Update chunk format docs with native histograms and OOO	2024-10-09 13:29:01 +02:00
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log ... For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-10-07 15:58:50 -04:00
Bryan Boreham	91de19fbef	[BUGFIX] TSDB: Don't read in-order chunks from before head MinTime ... Because we are reimplementing the `IndexReader` to fetch in-order and out-of-order chunks together, we must reproduce the behaviour of `Head.indexRange()`, which floors the minimum time queried at `head.MinTime()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-10-07 13:50:03 +01:00
Matthieu MOREL	ab64966e9d	fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094) ... * fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-10-06 16:35:29 +00:00
György Krajcsovits	44ebbb8458	Fix missing histogram copy in sampleRing ... The specialized version of sample add to the ring: func addH(s hSample, buf []hSample, r *sampleRing) []hSample func addFH(s fhSample, buf []fhSample, r *sampleRing) []fhSample already correctly copy histogram samples from the reused hReader, fhReader buffers, but the generic version does not. This means that the data is overwritten on the next read if the sample ring has seen histogram and float samples at the same time and switched to generic mode. The `genericAdd` function (which was commented anyway) is by now quite different from the specialized functions so that this commit deletes it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2024-10-02 13:57:28 +02:00
Bryan Boreham	54de4fb780	Merge pull request #14975 from colega/process-mempostings-delete-with-gomaxprocs-workers ... Process `MemPostings.Delete()` with `GOMAXPROCS` workers	2024-09-29 07:58:42 +01:00