Commit graph

1138 commits

Author SHA1 Message Date
Arve Knudsen f047647bb0 Merge remote-tracking branch 'prometheus/main' into arve/sync-upstream
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-24 14:36:13 +01:00
Charles Korn 59844498f7
Fix issue where queries can fail or omit OOO samples if OOO head compaction occurs between creating a querier and reading chunks (#13115)
* Add failing test.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Don't run OOO head garbage collection while reads are running.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add further test cases for different order of operations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.blockChunkQuerierForRange()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.Querier()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Invert error handling in `DB.Querier()` and `DB.blockChunkQuerierForRange()` to make it clearer

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure that queries that touch OOO data can't block OOO head garbage collection forever.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: fix parameter name in comment

Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>

* Address PR feedback: use `lastGarbageCollectedMmapRef`

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: ensure pending reads are cleaned up if creating an OOO querier fails

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
2023-11-24 12:38:38 +01:00
Peter Štibraný cdc4e7fd4b
Support for computing and keeping secondary hash per series (#568)
* Add secondary hash function to DB options, and use it calculate hashed for memSeries

* Add function to count secondary hashes that fall within a set of ranges

* Simplify SecondaryHash function changes to Head.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Fix TODOs.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Added test for Secondary hash function and ForEachSecondaryHash.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Make sure we don't hold any lock when calling function.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

---------

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Co-authored-by: Patryk Prus <patryk.prus@grafana.com>
2023-11-21 16:26:30 +01:00
Bryan Boreham f13bc1a5c9
Merge pull request #13040 from bboreham/smaller-stripeseries
TSDB: make the global hash lookup table smaller
2023-11-20 12:12:09 +00:00
Oleg Zaytsev f997c72f29
Make head block ULIDs descriptive (#13100)
* Make head block ULIDs descriptive

As far as I understand, these ULIDs aren't persisted anywhere, so it
should be safe to change them.

When debugging an issue, seeing an ULID like
`2ZBXFNYVVFDXFPGSB1CHFNYQTZ` or `33DXR7JA39CHDKMQ9C40H6YVVF` isn't very
helpful, so I propose to make them readable in their ULID string
version.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Set a different ULID for RangeHead

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-11-17 12:29:36 +01:00
Julien Pivotto 1b84c01b76
Merge pull request #13143 from mmorel-35/patch-3
tsdb/tsdbutil: use Go standard errors
2023-11-17 10:23:04 +01:00
Julien Pivotto 9cb96ad2ea
Merge pull request #13142 from mmorel-35/patch-2
tsdb/fileutil: use Go standard errors
2023-11-17 10:20:40 +01:00
Julien Pivotto 58eca19ac0
Merge pull request #13141 from mmorel-35/patch-1
tsdb/errors: fix errorlint linter
2023-11-17 10:20:00 +01:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Matthieu MOREL d7c3bc4cb0
tsdb/tsdbutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 20:46:36 +01:00
Matthieu MOREL e60a508dd8 tsdb/errors: fix errorlint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:16:12 +00:00
Matthieu MOREL e3041740e4 tsdb/fileutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:05:21 +00:00
Matthieu MOREL dd8871379a remplace errors.Errorf by fmt.Errorf
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 13:04:31 +00:00
Bryan Boreham 1bfb3ed062
Labels: reduce allocations when creating from TSDB WAL (#13044)
* Labels: reduce allocations when creating from TSDB

When reading the WAL, by passing references into the buffer we can avoid
copying strings under `-tags stringlabels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-14 11:36:35 +00:00
Julien Pivotto b718dca70b
Merge pull request #13108 from mmorel-35/patch-2
tsdb/chunkenc: use Go standard errors package
2023-11-14 00:54:35 +01:00
Julien Pivotto 3173feb5ef
Merge pull request #13107 from mmorel-35/patch-1
tsdb/agent: use Go standard errors package
2023-11-14 00:54:11 +01:00
Julien Pivotto 90ed7b08dc
Merge pull request #13124 from mmorel-35/patch-5
tsdb/index: use Go standard errors package
2023-11-14 00:53:49 +01:00
Julien Pivotto afc57eb306
Merge pull request #13109 from mmorel-35/patch-3
tsdb/chunks: use Go standard errors package
2023-11-14 00:52:59 +01:00
Julien Pivotto b6274ee747
Merge pull request #13114 from mmorel-35/patch-4
tsdb/wlog: use Go standard errors package
2023-11-14 00:52:21 +01:00
Julien Pivotto 7bcae56b6e
Merge pull request #13130 from mmorel-35/patch-6
tsdb/encoding: use Go standard errors package
2023-11-14 00:51:47 +01:00
Julien Pivotto f53af5c849
Merge pull request #13131 from mmorel-35/patch-7
tsdb/tombstones: use Go standard errors package
2023-11-14 00:51:31 +01:00
Julien Pivotto 17fe6bc296
Merge pull request #13133 from mmorel-35/patch-8
tsdb/record: use Go standard errors package
2023-11-14 00:51:16 +01:00
Bryan Boreham 65a443e6e3 TSDB: initialize conflicts map only when we need it.
Suggested by @songjiayang.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-13 16:20:31 +00:00
Bryan Boreham 37573d083b
Merge pull request #13056 from songjiayang/symbolCacheEntry-filed-alignment
TSDB block index writer: reduce memory used by symbol cache

Improve struct field alignment for symbolCacheEntry
2023-11-13 14:37:43 +00:00
George Krajcsovits acc114fe55
Fix panic during tsdb Commit (#13092)
* Fix panic during tsdb Commit

Fixes the following

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x19deb45]

goroutine 651118930 [running]:
github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0xc19100f7c0)
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:855 +0x245
github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x35bd6f0?, 0xc19100f7c0?}, 0xc000fa4c00?})
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/db.go:1159 +0x2f

We theorize that the panic happened due the the series referenced by the
exemplar being removed between AppendExemplar and Commit due to being idle.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-12 14:51:37 +00:00
Matthieu MOREL 469e415d09
Update record.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 21:01:24 +01:00
Matthieu MOREL 69c07ec6ae
Update record_test.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:57:42 +01:00
Matthieu MOREL 63691d82a5
tsdb/record: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:52:49 +01:00
Matthieu MOREL c74b7ad4fb
Update tombstones.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:22:06 +01:00
Matthieu MOREL 118460a64f
tsdb/tombstones: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:19:50 +01:00
Matthieu MOREL 4d6d3c1715
tsdb/encoding: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:01:11 +01:00
George Krajcsovits 39a35d92bc
tsdb/head: wlog exemplars after samples (#13113)
When samples are committed in the head, they are also written to the WAL.
The order of WAL records should be sample then exemplar, but this was
not the case for native histogram samples. This PR fixes that.

The problem with the wrong order is that remote write reads the WAL and
sends the recorded timeseries in the WAL order, which means exemplars
arrived before histogram samples. If the receiving side is Prometheus
TSDB and the series has not existed before then the exemplar does not
currently create the series. Which means the exemplar is rejected and lost.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-11 17:30:16 +01:00
Arve Knudsen d1c7983f22 Don't end non-owned span
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-10 08:11:06 +01:00
Arve Knudsen e1e64f36c2
Apply suggestions from code review
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
2023-11-10 08:02:13 +01:00
Matthieu MOREL 2972cc5e8f tsdb/index: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-09 21:37:41 +00:00
Arve Knudsen 92606f3b52 Fix trace event
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 17:08:59 +01:00
Arve Knudsen 476ebcb0a6 Adjust benchmark
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 16:58:06 +01:00
Arve Knudsen 83b0ed7bdb Clean up
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 16:58:06 +01:00
Arve Knudsen e3b8667fb1 Increase number of postings in benchmark
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 16:58:06 +01:00
Arve Knudsen ef70a6e6cf Allocate tracing attributes up front
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 16:57:56 +01:00
Arve Knudsen 46bf26e841 Create Tracer once per PostingsForMatchersCache
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 403abaee79 Set span status on error
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 7c313be3b5 Remove unnecessary event attributes
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 2aabf7cc2c Improve tracing
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 25e2c05a47 Implement review feedback
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 6c0910a289 Update tsdb/postings_for_matchers_cache.go
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen ca4cdeb1af Apply suggestions from code review
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
2023-11-09 15:31:31 +01:00
Arve Knudsen 8ece24ddb0 PostingsForMatchersCache: Add tracing
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-09 15:31:31 +01:00
Bryan Boreham e6c0f69f98 TSDB: Only pay for hash collisions when they happen
Instead of a map of slices of `*memSeries`, ready for any of them to
hold series where hash values collide, split into a map of `*memSeries`
and a map of slices which is usually empty, since hash collisions are
a one-in-a-billion thing.

The `del` method gets more complicated, to maintain the invariant that
a series is only in one of the two maps.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
Bryan Boreham ce4e757704 TSDB: refine variable naming in chunk gc
Slight further refactor.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00