Commit graph

1394 commits

Author SHA1 Message Date
Charles Korn 4a1fb3b18c
Fix issue where concatenatingChunkIterator can obscure errors.
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-10-23 11:36:45 +11:00
Jeanette Tan f898005c69 Merge remote-tracking branch 'upstream/main' into sync-upstream-20231018 2023-10-18 11:43:51 +08:00
Levi Harrison dcaca86958
Update dependencies for 2.48 (#12964) 2023-10-15 10:53:59 -04:00
Charles Korn 6c45249fb5
Make the linter happy 2023-10-12 12:29:44 +11:00
Charles Korn 4bf8ec202c
Fix similar issue for Next() as well.
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-10-12 11:57:39 +11:00
Charles Korn 6976e003cb
Fix issue where consumers of chainSampleIterator.Seek() can ignore errors
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-10-11 15:17:10 +11:00
Arve Knudsen 35ab75918a Merge remote-tracking branch 'prometheus/main' into arve/upgrade-exp
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-10-06 16:11:40 +02:00
Bryan Boreham a5a4eab679
Storage: reduce memory allocations when merging series sets (#12938)
Instead of setting to nil and allocating a new slice every time the
merge is advanced, re-use the previous slice.
This is safe because the `currentSets` member is only used inside member
functions, and explicitly copied in `At()`, the only place it leaves the
struct.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-06 12:28:07 +01:00
rakshith210 cdad64002a
Added Azure OAuth support (#12572)
* Added Azure OAuth support

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Added missing comment

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Addressing comment

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Fixed lint issue

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Fix test

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Addressing comments

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Added documentation and updated unit tests

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

* Addressing comments

Signed-off-by: rakshith210 <rakshith.me@gmail.com>

---------

Signed-off-by: rakshith210 <rakshith.me@gmail.com>
2023-10-04 22:16:36 -04:00
Goutham Veeramachaneni 86729d4d7b
Update exp package (#12650) 2023-09-21 22:53:51 +02:00
Björn Rabenstein ade2ee3fe4
Merge pull request #12716 from bboreham/simplify-seek
storage: simplify Seek on BufferedSeriesIterator
2023-09-21 13:56:29 +02:00
Arve Knudsen 85df1d4013 Merge remote-tracking branch 'prometheus/main' into chore/sync-prometheus
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-21 09:19:23 +02:00
William Dumont ce6ad15422 remote-write: TestClientRetryAfter status code 500
and compare the retryAfter values.

Signed-off-by: William Dumont <william.dumont@grafana.com>
2023-09-20 10:25:43 +00:00
William Dumont febd62a23e remote-write: refactor TestClientRetryAfter
The new version features a set of test cases that simplify the addition
of new HTTP status codes.

Signed-off-by: William Dumont <william.dumont@grafana.com>
2023-09-20 10:24:52 +00:00
Bryan Boreham 9b85354acd remote-write: respect Retry-After header on 5xx errors
If the server sent it to us, we should assume it knows better than we
do and respect it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-09-20 10:14:38 +00:00
Paschalis Tsilias c173cd57c9
Add a header to count retried remote write requests (#12729)
Header name is `Retry-Attempt`, only set when >0.

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
2023-09-20 11:11:03 +01:00
György Krajcsovits c9e513f5a3 Merge remote-tracking branch 'upstream/main' into krajo/fork-sync
# Conflicts:
#	storage/merge_test.go
#	tsdb/chunkenc/histogram_meta.go
2023-09-19 17:53:25 +02:00
George Krajcsovits 3512b2d678
storage: make histogram reset handling consistent in chainSampleIterator (#12779)
storage: make histogram reset handling consistent in chainSampleIterator

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-19 17:06:46 +02:00
Arve Knudsen e48d4e5835 Merge remote-tracking branch 'prometheus/main' into chore/sync-prometheus
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-18 09:29:42 +02:00
zenador 69edd8709b
Add warnings (and annotations) to PromQL query results (#12152)
Return annotations (warnings and infos) from PromQL queries

This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations".

Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future.

The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms).

The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later.

Another TODO is to take annotations into account when evaluating recording rules.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-09-14 18:57:31 +02:00
Arve Knudsen 156222cc50
Add context argument to LabelQuerier.LabelValues (#12665)
Add context argument to LabelQuerier.LabelValues and
LabelQuerier.SortedLabelValues.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 16:02:04 +02:00
Arve Knudsen a964349e97
Add context argument to LabelQuerier.LabelNames (#12666)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 10:39:51 +02:00
beorn7 0521ec12af storage: remove obsolete TODO
This was solved one layer deeper with #11687.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-09-13 15:10:22 +02:00
Arve Knudsen 6daee89e5f
Add context argument to Querier.Select (#12660)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-12 12:37:38 +02:00
Dimitar Dimitrov 77ac7ad40a
Merge remote-tracking branch 'upstream/main' into dimitar/pull-upstream 2023-09-05 16:19:00 +02:00
Gregor Zeitlinger f01718262a
Unit tests for native histograms (#12668)
promql: Extend testing framework to support native histograms

This includes both the internal testing framework as well as the rules unit test feature of promtool.

This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily.

---------

Signed-off-by: Harold Dost <h.dost@criteo.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Harold Dost <h.dost@criteo.com>
Co-authored-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
2023-08-25 23:35:42 +02:00
Justin Lei 8ef7dfdeeb
Add a chunk size limit in bytes (#12054)
Add a chunk size limit in bytes

This creates a hard cap for XOR chunks of 1024 bytes.

The limit for histogram chunk is also 1024 bytes, but it is a soft limit as a histogram has a dynamic size, and even a single one could be larger than 1024 bytes.

This also avoids cutting new histogram chunks if the existing chunk has fewer than 10 histograms yet. In that way, we are accepting "jumbo chunks" in order to have at least 10 histograms in a chunk, allowing compression to kick in.

Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-08-24 15:21:17 +02:00
beorn7 aa82fe198f tsdb: Fix histogram validation
So far, `ValidateHistogram` would not detect if the count did not
include the count in the zero bucket. This commit fixes the problem
and updates all the tests that have been undetected offenders so far.

Note that this problem would only ever create false negatives, so we
never falsely rejected to store a histogram because of it.

On the other hand, `ValidateFloatHistogram` has been to strict with
the count being at least as large as the sum of the counts in all the
buckets. Float precision issues could create false positives here, see
products of PromQL evaluations, it's actually quite hard to put an
upper limit no the floating point imprecision. Users could produce the
weirdest expressions, maxing out float precision problems. Therefore,
this commit simply removes that particular check from
`ValidateFloatHistogram`.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 23:04:01 +02:00
Michael Hoffmann 4d8e380269
promql: allow tests to be imported (#12050)
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
2023-08-18 20:48:59 +02:00
Bryan Boreham a018a7ef53 storage: simplify Seek on BufferedSeriesIterator
Small tweak to call a simpler method

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-17 07:42:18 +01:00
Bryan Boreham d2ae8dc3cb remote-write: add http.resend_count tracing attribute
As recommended by the OpenTelemetry semantic conventions.

https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/http/#http-client
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-11 16:20:12 +00:00
Oleg Zaytsev a54cb4744d Merge branch 'main' into prometheus-2023-07-31-76dd9b547 2023-08-04 09:46:03 +02:00
Goutham Veeramachaneni ad4f514e66
Add OTLP Ingestion endpoint (#12571)
* Add OTLP Ingestion endpoint

We copy files from the otel-collector-contrib. See the README in
`storage/remote/otlptranslator/README.md`.

This supersedes: https://github.com/prometheus/prometheus/pull/11965

Signed-off-by: gouthamve <gouthamve@gmail.com>

* Return a 200 OK

It is what the OTEL Golang SDK expect :(

https://github.com/open-telemetry/opentelemetry-go/issues/4363

Signed-off-by: Goutham <gouthamve@gmail.com>

---------

Signed-off-by: gouthamve <gouthamve@gmail.com>
Signed-off-by: Goutham <gouthamve@gmail.com>
2023-07-28 12:35:28 +02:00
George Krajcsovits 6cd2d1621f
Hide histogram chunk append and reset header internals (#12352)
tsdb: Hide histogram chunk append and reset header internals

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-07-26 15:08:16 +02:00
LHHDZ 7d8f9b0978
remote-write receiver: reuse 'ref' to optimize multiple samples for same series (#12580)
reuse 'ref' to optimize multi samples processing efficiency

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-07-22 14:24:46 +01:00
György Krajcsovits d4e355243a tsdbutil/ChunkFromSamplesGeneric should not panic
Add error handling instead.
Prepares for #12352

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-07-20 17:01:34 +02:00
Jeanette Tan 8035c04624 Merge remote-tracking branch 'upstream/main'
Minor conflicts:
rules/manager.go
tsdb/compact.go
tsdb/db.go
go.mod
2023-07-19 21:40:27 +08:00
Julien Pivotto 0f85e4f41d
Merge pull request #12539 from bboreham/slices-sorts
Replace sort.Slice with faster slices.SortFunc
2023-07-11 13:09:02 +02:00
Bryan Boreham ce153e3fff Replace sort.Sort with faster slices.SortFunc
The generic version is more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-07-10 09:43:45 +00:00
Charles Korn f3697f5242
Merge pull request #513 from grafana/charleskorn/expose-chunk-count
Add `ChunkCount()` method to `ChunkSeries`
2023-07-07 09:42:45 +10:00
Marco Pracucci 7ad111b27e
Merge remote-tracking branch 'remotes/prometheus/main' into update-upstream 2023-07-06 15:13:54 +02:00
Charles Korn 85fe1b0d83
Fix linting issue. 2023-07-06 15:53:20 +10:00
Charles Korn a0634e7408
Rename EstimatedChunkCount to ChunkCount 2023-07-06 15:42:44 +10:00
Charles Korn a9445622ad
Permit returning an error from EstimatedChunkCount() 2023-07-06 15:41:05 +10:00
Charles Korn 31235c351f
Simplify trickier estimation cases by not estimating at all. 2023-07-06 15:25:11 +10:00
Charles Korn f71a97b460
Address PR feedback: add comment explaining that we don't expect one case to happen often 2023-07-06 14:27:14 +10:00
Charles Korn 1299d98c46
Fix incorrect estimated chunks for NewCompactingChunkSeriesMerger if series are not already sorted. 2023-07-06 13:57:08 +10:00
Charles Korn 8b84856ba5
Add EstimatedChunkCount() method to ChunkSeries
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-07-05 15:34:07 +10:00
rakshith210 27717faa1d Add Azure AD package for remote write (#11944)
* Add Azure AD package for remote write
* Made AzurePublic default and updated configuration.md
* Updated config structure and removed getToken at initialization
* Changed passing context from request

Signed-off-by: Rakshith Padmanabha <rapadman@microsoft.com>
Signed-off-by: rakshith210 <rakshith.me@gmail.com>
2023-07-04 13:39:01 +00:00
Julien Pivotto 986fde06b2
Merge pull request #11688 from damnever/fix/datamodelvalidation-remotewriteapi
Validate the metric names and labels in the remote write handler
2023-07-04 13:52:02 +02:00