Commit graph

311 commits

Author SHA1 Message Date
Ziqi Zhao a93859a52f
Prometheus support parse exemplars from native histogram (#13488)
scrape: support parsing exemplars from native histogram

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-02-14 17:24:40 +01:00
Bryan Boreham ff6c83269c
Merge pull request #13452 from bboreham/go-cmp
Tests: Use DeepEqual replacement using go-cmp, which is more flexible
2024-02-12 15:40:08 +01:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
beorn7 553d92affd model/labels: Fix new lint warning in test
Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-07 18:12:26 +01:00
Bryan Boreham 14b4fbc2ff labels: make InternStrings a no-op for stringlabels version
The current implementation of `InternStrings` will only save memory
when the whole set of labels is identical to one already seen, and this
cannot happen in the one place it is called from in Prometheus,
remote-write, which already detects identical series.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-29 18:54:54 +00:00
Marco Pracucci 501bc6419e
Add ShardedPostings() support to TSDB (#10421)
This PR is a reference implementation of the proposal described in #10420.

In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing).

Follow up work
As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes.

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 11:57:27 +00:00
Bryan Boreham 74b73d1e2c
Labels: Add DropMetricName function, used in PromQL (#13446)
This function is called very frequently when executing PromQL functions,
and we can do it much more efficiently inside Labels.

In the common case that `__name__` comes first in the labels, we simply
re-point to start at the next label, which is nearly free.

`DropMetricName` is now so cheap I removed the cache - benchmarks show
everything still goes faster.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:48:49 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
zenador a3ddfbd1ee
Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392)
Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-01-17 17:06:35 +01:00
tyltr 1fa131ee03
fix slice copy in 1.20 (#13389)
The slices package is added to the standard library in Go 1.21;
we need to import from the exp area to maintain compatibility with Go 1.20.

Signed-off-by: tyltr <tylitianrui@126.com>
2024-01-16 11:06:58 +00:00
Björn Rabenstein 3db4596965
Merge pull request #13299 from vesari/add-unit-parser
protobuf: add unit parser
2024-01-09 14:47:50 +01:00
Arianna Vespri 8f07f9dd90 Avoid creating string for suffix, consider counters without _total suffix
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-28 15:41:38 +01:00
Bryan Boreham 4d44da2deb
Merge pull request #13176 from bboreham/move-metrictype
Move metric type definitions to common/model
2023-12-20 09:30:57 +00:00
Arianna Vespri 9a664b515a Correct order in error message
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:41:37 +01:00
Arianna Vespri 51e78d9a32 Delete debugging lines, amend error message for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:31:58 +01:00
Bryan Boreham c83e1fc574 textparse: remove MetricType alias
No backwards-compatibility; make a clean break.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Bryan Boreham 8065bef172 Move metric type definitions to common/model
They are used in multiple repos, so common is a better place for them.
Several packages now don't depend on `model/textparse`, e.g.
`storage/remote`.

Also remove `metadata` struct from `api.go`, since it was identical to
a struct in the `metadata` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Bryan Boreham 0289dd6157 relabel: blank replacement deletes label post-regexp
If `cfg.TargetLabel` is a template like `$1`, it won't match any labels,
so no point calling `lb.Del` with it.

Similarly if `target` is not a valid label name, it won't match any
labels, so don't call with that either.

The intention seems to have been that a blank _value_ would delete the
target, so change that code to use `target` instead of `cfg.TargetLabel`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 16:39:01 +00:00
Bryan Boreham 000182e4b8 relabel: check validity of all test cases
Thought this would be a nice check on the `Validate()` function, but
some of the test cases needed tweaking to pass.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 15:03:24 +00:00
Bryan Boreham 2d4c367d87 relabel: stricter check that target labels are valid
For `Lowercase`, `KeepEqual`, etc., we do not expand a regexp, so
the target label name must not contain anything like `${1}`.

Also for the common case that the `Replace` target does not require any
template expansion, check that the entire string passes label name
validity rules.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 14:58:59 +00:00
Bryan Boreham 9ab7e3b3de relabel: refactor: extract config.Validate method
And add a test for it, which fails because validation is not strong enough.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 14:54:55 +00:00
Arianna Vespri 3d6a92a25e Merge remote-tracking branch 'prometheus/main' into add-unit-parser 2023-12-17 14:11:55 +01:00
Arianna Vespri b65021d7a5 Get conditional right
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-17 14:09:14 +01:00
Arianna Vespri 9fb1e9715c Go on adding protobuf parsing for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-14 17:00:52 +01:00
Arianna Vespri 5d5acf5744 Add unit protobuf parser
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-12 12:40:08 +01:00
Björn Rabenstein 928d07e3bd
Merge branch 'main' into arve/typos
Signed-off-by: Björn Rabenstein <beorn@grafana.com>
2023-12-12 12:02:03 +01:00
Arthur Silva Sens 5082655392
Append Created Timestamps (#12733)
* Append created timestamps.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Log when created timestamps are ignored

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Proposed changes to Append CT PR.

Changes:

* Changed textparse Parser interface for consistency and robustness.
* Changed CT interface to be more explicit and handle validation.
* Simplified test, change scrapeManager to allow testability.
* Added TODOs.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Updates.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Refactor head_appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Fix linter issues

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Use model.Sample in head appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
2023-12-11 08:43:42 +00:00
Filip Petkovski 9008271df5
Use cap to determine slice capacity
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:23:01 +01:00
Filip Petkovski 10a82f87fd
Enable reusing memory when converting between histogram types
The 'ToFloat' method on integer histograms currently allocates new memory
each time it is called.

This commit adds an optional *FloatHistogram parameter that can be used
to reuse span and bucket slices. It is up to the caller to make sure the
input float histogram is not used anymore after the call.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:22:59 +01:00
Björn Rabenstein 85078b968f
Merge pull request #13208 from linasm/panic-free-float-histogram-add-sub
FloatHistogram.Add/Sub: handle any schema change
2023-12-07 20:12:21 +01:00
Björn Rabenstein de1ec731c7
Merge pull request #13212 from linasm/optimize-floatBucketIterator-for-common-case
Histograms: optimize floatBucketIterator for common case
2023-12-07 20:11:58 +01:00
Linas Medziunas bd895baefc FloatHistogram.Add/Sub: handle any schema change
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-12-07 20:50:54 +02:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Arve Knudsen 237bfea46b chunks.Reader: Fix typo in ChunkOrIterable doc string.
Also fix comment typo in `FloatHistogram.Sub`.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-12-07 08:28:45 +01:00
Oleksandr Redko 2a75604f8e
Enable default revive rules (#13068)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-11-29 17:23:34 +00:00
Linas Medziunas 048886ae8a Histograms: optimize floatBucketIterator for common case
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-29 08:54:05 +02:00
Linas Medziunas 49613823f8 Reuse slices in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-28 15:45:31 +02:00
Björn Rabenstein a6d4b8d97b
Merge pull request #13190 from linasm/fix-float-histogram-add-sub-mutating-argument
Fix FloatHistogram.Add/Sub mutating its argument
2023-11-28 14:04:50 +01:00
Linas Medziunas f99ecc376e Fix FloatHistogram.Add/Sub mutating its argument
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-26 09:26:34 +02:00
Linas Medziunas ea1862aab4 Explicit schema check in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-25 11:38:15 +02:00
Bryan Boreham f0e1b592ab Scraping: use slices.sort for exemplars
The sort implementation using Go generics is used everywhere else
in Prometheus.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-24 14:42:26 +00:00
Bryan Boreham a3e02f35d6 labels: extract common code between slice and stringlabels
This reduces bulk and should avoid issues if a fix is made in one file
and not the other.

A few methods now call `Range()` instead of `range`, but nothing
performance-sensitive.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-17 18:20:03 +00:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Bryan Boreham 1bfb3ed062
Labels: reduce allocations when creating from TSDB WAL (#13044)
* Labels: reduce allocations when creating from TSDB

When reading the WAL, by passing references into the buffer we can avoid
copying strings under `-tags stringlabels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-14 11:36:35 +00:00
Ziqi Zhao e250f09b5d
change origin schema in ReduceResolution method of histogram and float histogram (#13116)
* change origin schema in ReduceResolution method of histogram and float histogram

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-10 14:33:34 +01:00
Ziqi Zhao ab2a7bb74f
add generic shrink function (#13001)
Add `ReduceResolution` method to `Histogram` and `FloatHistogram`

This takes the original `mergeToSchema` function and turns it into a more generic `reduceResolution` function, which is the building block for the new methods.

The methods will help with addressing #12864.

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-08 14:43:05 +01:00
Matthieu MOREL 05fba53e57 web : use Go standard package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 06:37:59 +00:00
Julien Pivotto cf01ec2119
Merge pull request #13091 from mmorel-35/errorlint/util
util: use Go standard errors package
2023-11-07 21:07:25 -06:00
Linas Medziunas ebed7d0612 Change Validate to be a method on histogram structs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:47:59 +02:00
Linas Medziunas 1f8aea11d6 Move histogram validation code to model/histogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00