Commit graph

405 commits

Author SHA1 Message Date
Marco Pracucci bfec57bd2e
Further optimise FastRegexMatcher
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-25 10:40:57 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
zenador a3ddfbd1ee
Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392)
Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-01-17 17:06:35 +01:00
tyltr 1fa131ee03
fix slice copy in 1.20 (#13389)
The slices package is added to the standard library in Go 1.21;
we need to import from the exp area to maintain compatibility with Go 1.20.

Signed-off-by: tyltr <tylitianrui@126.com>
2024-01-16 11:06:58 +00:00
Björn Rabenstein 3db4596965
Merge pull request #13299 from vesari/add-unit-parser
protobuf: add unit parser
2024-01-09 14:47:50 +01:00
Arianna Vespri 8f07f9dd90 Avoid creating string for suffix, consider counters without _total suffix
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-28 15:41:38 +01:00
Bryan Boreham 4d44da2deb
Merge pull request #13176 from bboreham/move-metrictype
Move metric type definitions to common/model
2023-12-20 09:30:57 +00:00
Arianna Vespri 9a664b515a Correct order in error message
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:41:37 +01:00
Arianna Vespri 51e78d9a32 Delete debugging lines, amend error message for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:31:58 +01:00
Bryan Boreham c83e1fc574 textparse: remove MetricType alias
No backwards-compatibility; make a clean break.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Bryan Boreham 8065bef172 Move metric type definitions to common/model
They are used in multiple repos, so common is a better place for them.
Several packages now don't depend on `model/textparse`, e.g.
`storage/remote`.

Also remove `metadata` struct from `api.go`, since it was identical to
a struct in the `metadata` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Bryan Boreham 0289dd6157 relabel: blank replacement deletes label post-regexp
If `cfg.TargetLabel` is a template like `$1`, it won't match any labels,
so no point calling `lb.Del` with it.

Similarly if `target` is not a valid label name, it won't match any
labels, so don't call with that either.

The intention seems to have been that a blank _value_ would delete the
target, so change that code to use `target` instead of `cfg.TargetLabel`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 16:39:01 +00:00
Bryan Boreham 000182e4b8 relabel: check validity of all test cases
Thought this would be a nice check on the `Validate()` function, but
some of the test cases needed tweaking to pass.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 15:03:24 +00:00
Bryan Boreham 2d4c367d87 relabel: stricter check that target labels are valid
For `Lowercase`, `KeepEqual`, etc., we do not expand a regexp, so
the target label name must not contain anything like `${1}`.

Also for the common case that the `Replace` target does not require any
template expansion, check that the entire string passes label name
validity rules.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 14:58:59 +00:00
Bryan Boreham 9ab7e3b3de relabel: refactor: extract config.Validate method
And add a test for it, which fails because validation is not strong enough.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-18 14:54:55 +00:00
Arianna Vespri 3d6a92a25e Merge remote-tracking branch 'prometheus/main' into add-unit-parser 2023-12-17 14:11:55 +01:00
Arianna Vespri b65021d7a5 Get conditional right
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-17 14:09:14 +01:00
Arianna Vespri 9fb1e9715c Go on adding protobuf parsing for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-14 17:00:52 +01:00
Arianna Vespri 5d5acf5744 Add unit protobuf parser
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-12 12:40:08 +01:00
Björn Rabenstein 928d07e3bd
Merge branch 'main' into arve/typos
Signed-off-by: Björn Rabenstein <beorn@grafana.com>
2023-12-12 12:02:03 +01:00
Arthur Silva Sens 5082655392
Append Created Timestamps (#12733)
* Append created timestamps.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Log when created timestamps are ignored

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Proposed changes to Append CT PR.

Changes:

* Changed textparse Parser interface for consistency and robustness.
* Changed CT interface to be more explicit and handle validation.
* Simplified test, change scrapeManager to allow testability.
* Added TODOs.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Updates.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Refactor head_appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Fix linter issues

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Use model.Sample in head appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
2023-12-11 08:43:42 +00:00
Filip Petkovski 9008271df5
Use cap to determine slice capacity
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:23:01 +01:00
Filip Petkovski 10a82f87fd
Enable reusing memory when converting between histogram types
The 'ToFloat' method on integer histograms currently allocates new memory
each time it is called.

This commit adds an optional *FloatHistogram parameter that can be used
to reuse span and bucket slices. It is up to the caller to make sure the
input float histogram is not used anymore after the call.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:22:59 +01:00
Björn Rabenstein 85078b968f
Merge pull request #13208 from linasm/panic-free-float-histogram-add-sub
FloatHistogram.Add/Sub: handle any schema change
2023-12-07 20:12:21 +01:00
Björn Rabenstein de1ec731c7
Merge pull request #13212 from linasm/optimize-floatBucketIterator-for-common-case
Histograms: optimize floatBucketIterator for common case
2023-12-07 20:11:58 +01:00
Linas Medziunas bd895baefc FloatHistogram.Add/Sub: handle any schema change
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-12-07 20:50:54 +02:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Arve Knudsen 237bfea46b chunks.Reader: Fix typo in ChunkOrIterable doc string.
Also fix comment typo in `FloatHistogram.Sub`.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-12-07 08:28:45 +01:00
Oleksandr Redko 2a75604f8e
Enable default revive rules (#13068)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-11-29 17:23:34 +00:00
Linas Medziunas 048886ae8a Histograms: optimize floatBucketIterator for common case
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-29 08:54:05 +02:00
Linas Medziunas 49613823f8 Reuse slices in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-28 15:45:31 +02:00
Björn Rabenstein a6d4b8d97b
Merge pull request #13190 from linasm/fix-float-histogram-add-sub-mutating-argument
Fix FloatHistogram.Add/Sub mutating its argument
2023-11-28 14:04:50 +01:00
Linas Medziunas f99ecc376e Fix FloatHistogram.Add/Sub mutating its argument
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-26 09:26:34 +02:00
Linas Medziunas ea1862aab4 Explicit schema check in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-25 11:38:15 +02:00
Bryan Boreham f0e1b592ab Scraping: use slices.sort for exemplars
The sort implementation using Go generics is used everywhere else
in Prometheus.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-24 14:42:26 +00:00
Bryan Boreham a3e02f35d6 labels: extract common code between slice and stringlabels
This reduces bulk and should avoid issues if a fix is made in one file
and not the other.

A few methods now call `Range()` instead of `range`, but nothing
performance-sensitive.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-17 18:20:03 +00:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Bryan Boreham 1bfb3ed062
Labels: reduce allocations when creating from TSDB WAL (#13044)
* Labels: reduce allocations when creating from TSDB

When reading the WAL, by passing references into the buffer we can avoid
copying strings under `-tags stringlabels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-14 11:36:35 +00:00
Ziqi Zhao e250f09b5d
change origin schema in ReduceResolution method of histogram and float histogram (#13116)
* change origin schema in ReduceResolution method of histogram and float histogram

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-10 14:33:34 +01:00
Ziqi Zhao ab2a7bb74f
add generic shrink function (#13001)
Add `ReduceResolution` method to `Histogram` and `FloatHistogram`

This takes the original `mergeToSchema` function and turns it into a more generic `reduceResolution` function, which is the building block for the new methods.

The methods will help with addressing #12864.

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-08 14:43:05 +01:00
Matthieu MOREL 05fba53e57 web : use Go standard package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 06:37:59 +00:00
Julien Pivotto cf01ec2119
Merge pull request #13091 from mmorel-35/errorlint/util
util: use Go standard errors package
2023-11-07 21:07:25 -06:00
Linas Medziunas ebed7d0612 Change Validate to be a method on histogram structs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:47:59 +02:00
Linas Medziunas 1f8aea11d6 Move histogram validation code to model/histogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
Matthieu MOREL fe057fc60d use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-03 07:26:31 +00:00
Björn Rabenstein fae4561369
Merge pull request #12954 from linasm/linasm/optimize-floatBucketIterator2
Histogram performance: optimize floatBucketIterator
2023-11-02 23:59:32 +01:00
Oleksandr Redko fa90ca46e5 ci(lint): enable godot; append dot at the end of comments
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 19:53:38 +02:00
Linas Medžiūnas e3e22abadb
Merge branch 'main' into linasm/optimize-floatBucketIterator2
Signed-off-by: Linas Medžiūnas <linasm@users.noreply.github.com>
2023-10-24 16:27:10 +03:00
beorn7 b428416f06 textparse: Update comment about timestamp_ms protobuf parsing
By now, we know better what the plan is.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-10-19 17:54:42 +02:00
Arthur Silva Sens ef8e6ae780
Parse created timestamps from Prometheus Protobuf (#12973)
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
2023-10-18 19:04:02 +01:00
Marc Tuduri 1ce066e51c
More periods
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:53:50 +02:00
Marc Tuduri af7c31ee10
PR feedback
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:53:50 +02:00
Marc Tuduri 8fededf6ad
promql(histograms): Change sample total calculation for histograms
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:51:11 +02:00
Björn Rabenstein f33bffa788
Merge pull request #12891 from linasm/fix-gaps-in-histogram-equals
Fix NaN checks in [Float]Histogram.Equals method
2023-10-18 00:35:17 +02:00
Linas Medziunas 62bbb81e29 Mention bucket values in the comment
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-10-14 21:30:40 +03:00
Linas Medziunas ec823d9daf Update comments, bitwise comparison of float buckets
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-10-09 16:09:46 +03:00
Linas Medziunas c5c5c569fa Histogram performance: optimize floatBucketIterator
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-10-09 09:40:59 +03:00
Matthieu MOREL 0a513f827d ci(lint): enable errorlint linter on model
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-09-29 21:11:33 +00:00
Julien Pivotto c52db2b196 Remove duplicate tests
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-09-29 10:41:13 +02:00
johncming 5d68ebb207 pkg/rulefmt: fix bug of validate.
Signed-off-by: johncming <johncming@yahoo.com>
2023-09-29 10:35:07 +02:00
Linas Medziunas 3c047a3518 Expand docs comments
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-28 09:08:09 +03:00
Linas Medziunas dfb6292600 Compare FloatHistogram.[Zero]Count float values as binary
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-28 09:06:54 +03:00
Linas Medziunas cbd01fc296 Fix NaN sum check in [Float]Histogram.Equals method
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-25 16:03:55 +03:00
Goutham Veeramachaneni 86729d4d7b
Update exp package (#12650) 2023-09-21 22:53:51 +02:00
Gregor Zeitlinger f01718262a
Unit tests for native histograms (#12668)
promql: Extend testing framework to support native histograms

This includes both the internal testing framework as well as the rules unit test feature of promtool.

This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily.

---------

Signed-off-by: Harold Dost <h.dost@criteo.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Harold Dost <h.dost@criteo.com>
Co-authored-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
2023-08-25 23:35:42 +02:00
zenador 54aaa2bd7e
Add histogram_stdvar and histogram_stddev functions (#12614)
* Add new function: histogram_stdvar and histogram_stddev

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-08-24 21:02:14 +02:00
Björn Rabenstein 798c5737a0
Merge pull request #12711 from fatsheep9146/floathistogram-addsub-enhance
enhance float histogram add and sub method
2023-08-24 13:14:10 +02:00
Ziqi Zhao de172049ab fix lint error
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-24 07:27:33 +08:00
Ziqi Zhao d3633d4e76
Update model/histogram/float_histogram.go
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-24 07:17:23 +08:00
Ziqi Zhao 893f97556f use switch instead of if-else to fix lint error
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-23 13:13:25 +08:00
Ziqi Zhao 788061e509 remove unused addBucket function
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-23 12:55:59 +08:00
Ziqi Zhao eab3c93e80 make code ready for review
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-23 12:52:24 +08:00
beorn7 65ccf4460a textparse: Fix endless loop #12731
PR #12557 introduced the possibility of parsing multiple exemplars per
native histograms. It did so by requiring the `Exemplar` method of the
parser to be called repeatedly until it returns false. However, the
protobuf parser code wasn't correctly updated for the old case of a
single exemplar for a classic bucket (if actually parsed as a classic
bucket) and a single exemplar on a counter. In those cases, the method
would return `true` forever, yielding the same exemplar again and
again, leading to an endless loop.

With this fix, the state is now tracked and the single exemplar is
only returned once.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 21:03:54 +02:00
beorn7 3d9a830f2f textparse: Expose #12731 in protobufparse_test.go
Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 20:53:31 +02:00
György Krajcsovits e846736134 Fix typo
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-08-21 14:13:49 +02:00
Ziqi Zhao 4787c879bc add more elaborate benchmark test
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-21 13:28:06 +08:00
Ziqi Zhao bf880a6e77 enhance floathistogram add and sub method
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-08-16 22:26:31 +08:00
Bryan Boreham d6e1b1acdb
Merge pull request #12681 from prometheus/labels-unused-code
labels: remove some unused code
2023-08-14 15:48:17 +01:00
Bryan Boreham 5007fa305d
Merge pull request #12680 from bboreham/faster-has
labels: improve Has() method for stringlabels build
2023-08-14 15:47:47 +01:00
Bryan Boreham ce260b1fe1 labels: remove some unused code
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-13 15:14:53 +01:00
Bryan Boreham b5c6807fea Labels.Has quick check on first character
Exit early if we've gone past - labels are sorted in order.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-13 15:11:22 +01:00
Bryan Boreham 33aab1b2cc labels: extend benchmark for Has()
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-13 14:55:50 +01:00
Ziqi Zhao ed1b307bca
enhance FloatHistogram CopyToSchema method (#12596)
histogram: Improve performance of FloatHistogram.CopyToSchema

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-07-27 13:27:13 +02:00
beorn7 c58e20ad0e histogram: Identify native histograms even without observations
Native histograms without observations and with a zero threshold of
zero look the same as classic histograms in the protobuf exposition
format. According to
https://github.com/prometheus/client_golang/issues/1127 , the idea is
to add a no-op span to those histograms to mark them as native
histograms. This commit enables Prometheus to detect that no-op span
and adds a doc comment to the proto spec describing the behavior.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-26 17:42:30 +02:00
Björn Rabenstein 0e12f11d61
Merge pull request #12583 from prometheus/release-2.46
Merge release-2.46 into main
2023-07-20 18:29:44 +02:00
Julien Pivotto 7905594b52
Merge pull request #12557 from prometheus/beorn7/histogram
scrape: Enable ingestion of multiple exemplars per sample
2023-07-20 15:19:28 +02:00
Björn Rabenstein b2567aeacd
Merge pull request #12575 from prometheus/beorn7/histogram2
histogram: Fix bounds of buckets returned by floatBucketIterator
2023-07-20 13:20:51 +02:00
beorn7 9aadd54786 histogram: Fix bounds of buckets returned by floatBucketIterator
The bounds weren't really used so far, so no actual bug in the code so
far. But it's obviously confusing if the bounds returned by a
floatBucketIterator with a target schema different from the original
schema are wrong.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 18:19:18 +02:00
beorn7 071f4bbea4 histograms: Fix parsing float histograms without zero bucket
If a float histogram has a zero bucket with a threshold of zero _and_
an empty zero bucket, it wasn't identified as a native histogram
because the `isNativeHistogram` helper function only looked at integer
buckets.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 15:29:11 +02:00
beorn7 2ea8df4734 histogram: Expose #12305
Native histograms without a zero threshold aren't federated properly.

This adds a test to prove the specific failure mode, which is that
histograms with a zero threshold of zero are federated as classic
histograms.

The underlying reason is that the protobuf parser identifies a native
histogram by detecting a zero bucket or by detecting integer buckets.
Therefore, a float histogram with a zero threshold of zero and an
unpopulated zero bucket falls through the cracks (no integer buckets,
no zero bucket).

This commit also addse a test case for the latter.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 15:29:11 +02:00
Julien Pivotto b7028c1722
Merge pull request #11426 from douglascamata/document-and-test-relabel-memory-reuse
Better document possible relabel input modification
2023-07-18 13:16:54 +02:00
cui fliter 096ceca44f
remove repetitive words (#12556)
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-07-13 15:53:40 +02:00
beorn7 0e3f35324b scrape: Enable ingestion of multiple exemplars per sample
This has become a requirement for native histograms, as a single
histogram sample commonly has many buckets, so that providing many
exemplars makes sense.

Since OM text doesn't support native histograms yet, the test had to
be expanded to also support protobuf test cases.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-13 14:16:10 +02:00
beorn7 da047c6857 histograms: Fix bug #12552
The problem was the following:

When trying to parse native histograms and classic histograms in
parallel, the parser would first parse the histogram proto messages as
a native histogram and then parse the same message again, but now as a
classic histogram. Afterwards, it would forget that it was dealing
with a metric family that contains native histograms and would parse
the rest of the metric family as classic histograms only. The fix is
to check again after being done with a classic histogram.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-12 18:47:00 +02:00
beorn7 1c3bd04bea histograms: Modify test to expose bug #12552
Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-12 18:17:35 +02:00
Douglas Camata 47819fef01
Merge branch 'main' of github.com:prometheus/prometheus into document-and-test-relabel-memory-reuse
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
2023-07-12 15:01:29 +02:00
Ziqi Zhao 42d9169ba1 enhance histogram_quantile to get min/max value
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-07-12 04:29:54 +08:00
Carrie Edwards a462f7fa21 Add function for iterating through all buckets in reverse to find max bucket
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2023-07-11 21:51:20 +08:00
Bryan Boreham e1115ae58d
labels: improve Get method for stringlabels build (#12485)
Inline one call to `decodeString`, and skip decoding the value string
until we find a match for the name.
Do a quick check on the first character in each string,
and exit early if we've gone past - labels are sorted in order.

Also improve tests and benchmark:
* labels: test Get with varying lengths - it's not typical for Prometheus labels to all be the same length.
* extend benchmark with label not found

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-06-26 18:35:22 +01:00
Bryan Boreham 87d08abe11
labels: faster Compare function when using -tags stringlabels (#12451)
Instead of unpacking every individual string, we skip to the point
where there is a difference, going 8 bytes at a time where possible.

Add benchmark for Compare; extend tests too.

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-06-20 20:58:47 +01:00
Oleg Zaytsev 6a18962cfa
mv labels_string.go labels_stringlabels.go (#12328)
This is a minor cosmetical change, but my IDE (and I guess many of them)
nests `labels_string.go` under `labels.go` because it assumes it's the
file generated by the `stringer` tool, which follows that naming
pattern.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-06-13 09:38:00 +01:00
Michael Hoffmann 344c8ff97c
feat: dont compile regex matcher if we know its a literal (#12434)
labels: dont compile regex matcher if we know its a literal

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

Co-authored-by: Sharad <sharadgaur@gmail.com>
2023-06-07 21:54:30 +01:00
zenador 191bf9055b
Handle more arithmetic operators for native histograms (#12262)
Handle more arithmetic operators and aggregators for native histograms

This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function.

Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them).

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-05-16 21:15:20 +02:00
Bryan Boreham a073e04a9b
Merge pull request #12366 from prometheus/release-2.44
Merge release 2.44 back to main
2023-05-16 18:06:29 +01:00
beorn7 9e500345f3 textparse/scrape: Add option to scrape both classic and native histograms
So far, if a target exposes a histogram with both classic and native
buckets, a native-histogram enabled Prometheus would ignore the
classic buckets. With the new scrape config option
`scrape_classic_histograms` set, both buckets will be ingested,
creating all the series of a classic histogram in parallel to the
native histogram series. For example, a histogram `foo` would create a
native histogram series `foo` and classic series called `foo_sum`,
`foo_count`, and `foo_bucket`.

This feature can be used in a migration strategy from classic to
native histograms, where it is desired to have a transition period
during which both native and classic histograms are present.

Note that two bugs in classic histogram parsing were found and fixed
as a byproduct of testing the new feature:

1. Series created from classic _gauge_ histograms didn't get the
   _sum/_count/_bucket prefix set.
2. Values of classic _float_ histograms weren't parsed properly.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-05-13 01:32:25 +02:00
Bryan Boreham 7a48a266b6
labels: respect Set after Del in Builder (#12322)
* labels: respect Set after Del in Builder

The implementations are not symmetric between `Set()` and `Del()`, so
we must be careful. Add tests for this, both in labels and in relabel
where the issue was reported.

Also make the slice implementation consistent re `slices.Contains`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-05-03 11:59:27 +01:00
cui fliter 276ca6a883 fix some comments
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-04-25 14:19:16 +08:00
Matthieu MOREL bae9a21200
Merge branch 'main' into linter/nilerr
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-19 19:56:39 +02:00
beorn7 5b53aa1108 style: Replace else if cascades with switch
Wiser coders than myself have come to the conclusion that a `switch`
statement is almost always superior to a statement that includes any
`else if`.

The exceptions that I have found in our codebase are just these two:

* The `if else` is followed by an additional statement before the next
  condition (separated by a `;`).
* The whole thing is within a `for` loop and `break` statements are
  used. In this case, using `switch` would require tagging the `for`
  loop, which probably tips the balance.

Why are `switch` statements more readable?

For one, fewer curly braces. But more importantly, the conditions all
have the same alignment, so the whole thing follows the natural flow
of going down a list of conditions. With `else if`, in contrast, all
conditions but the first are "hidden" behind `} else if `, harder to
spot and (for no good reason) presented differently from the first
condition.

I'm sure the aforemention wise coders can list even more reasons.

In any case, I like it so much that I have found myself recommending
it in code reviews. I would like to make it a habit in our code base,
without making it a hard requirement that we would test on the CI. But
for that, there has to be a role model, so this commit eliminates all
`if else` occurrences, unless it is autogenerated code or fits one of
the exceptions above.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:22:31 +02:00
beorn7 c3c7d44d84 lint: Adjust to the lint warnings raised by current versions of golint-ci
We haven't updated golint-ci in our CI yet, but this commit prepares
for that.

There are a lot of new warnings, and it is mostly because the "revive"
linter got updated. I agree with most of the new warnings, mostly
around not naming unused function parameters (although it is justified
in some cases for documentation purposes – while things like mocks are
a good example where not naming the parameter is clearer).

I'm pretty upset about the "empty block" warning to include `for`
loops. It's such a common pattern to do something in the head of the
`for` loop and then have an empty block. There is still an open issue
about this: https://github.com/mgechev/revive/issues/810 I have
disabled "revive" altogether in files where empty blocks are used
excessively, and I have made the effort to add individual
`// nolint:revive` where empty blocks are used just once or twice.
It's borderline noisy, though, but let's go with it for now.

I should mention that none of the "empty block" warnings for `for`
loop bodies were legitimate.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:10:10 +02:00
Bryan Boreham 1801cd4196 labels: small optimization to stringlabels
Add a fast path for the common case that a string is less than 127 bytes
long, to skip a shift and the loop.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-14 15:06:57 +00:00
Matthieu MOREL fb3eb21230 enable gocritic, unconvert and unused linters
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-13 19:20:22 +00:00
Bryan Boreham 10cc60af01 labels: add ScratchBuilder.Overwrite for slice implementation
This is a method used by some downstream projects; it was created to
optimize the implementation in `labels_string.go` but we should have one
for both implementations so the same code works with either.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-13 11:07:54 +00:00
Ganesh Vernekar 5588cab8b2
Merge pull request #12173 from bboreham/builder-no-empty-labels
labels: simplify call to get Labels from Builder
2023-04-04 12:02:55 +05:30
Bryan Boreham e917202766 labels: make sure estimated size is not negative
Deleted labels are remembered, even if they were not in `base` or were
removed from `add`, so `base+add-del` could go negative.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-02 11:17:09 +01:00
Bryan Boreham ee1157c14a labels: shrink stack arrays in Builder.Range
Go spends some time initializing all the elements of these arrays to
zero, so reduce the size from 1024 to 128. This is still much bigger
than we ever expect for a set of labels.

(If someone does have more than 128 labels it will still work, but via
heap allocation.)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-22 17:14:43 +00:00
Bryan Boreham b987afa7ef labels: simplify call to get Labels from Builder
It took a `Labels` where the memory could be re-used, but in practice
this hardly ever benefitted. Especially after converting `relabel.Process`
to `relabel.ProcessBuilder`.

Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil`
so the slice was reallocated multiple times by `append`.

Lastly `Builder.Labels()` now estimates that the final size will depend
on labels added and deleted.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-22 17:05:20 +00:00
Bryan Boreham 3743d87c56 labels: cope with mutating Builder during Range call
Although we had a different slice, the underlying memory was the same so
any changes meant we could skip some values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-16 13:28:15 +00:00
Bryan Boreham 3c4ab7a069 labels: add test for Builder.Range
Including mutating the Builder being Ranged over.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-16 13:25:55 +00:00
Björn Rabenstein 847093479b
Merge pull request #11978 from trevorwhitney/set-counter-hint
Set `CounterResetHint` and use in recording rules
2023-03-14 21:52:41 +01:00
Trevor Whitney e3513d1dd2
Change nested ifs to a switch
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:22:20 -06:00
Trevor Whitney b4e324f637
Handle valid cases of mismatched hints when adding
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:22:14 -06:00
Trevor Whitney dd94ebb87b
promql: set CounterResetHint after rate and sum
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:21:59 -06:00
Julien Pivotto 5583c77b3a
Merge pull request #12095 from damnever/unnecessary-sort
Remove unnecessary sort
2023-03-09 13:12:02 +01:00
Xiaochao Dong (@damnever) 36fc1158b5 Remove unnecessary sort
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-03-08 15:36:02 +08:00
Bryan Boreham 11d019ed5a relabel: keep intermediate results in labels.Builder
Save work converting between Builder and Labels.

Also expose ProcessBuilder, so callers can supply a Builder.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-07 17:21:37 +00:00
Bryan Boreham d740abf0c6 model/labels: add Get and Range to Builder
This lets relabelling work on a `Builder` rather than converting to and
from `Labels` on every rule.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-07 17:20:54 +00:00
Bryan Boreham ff993b279a
Merge pull request #12073 from bboreham/slices-sort2
labels: use slices.Sort for better performance
2023-03-07 09:31:50 +00:00
Bryan Boreham 38c6d3da9f labels: use slices.Sort for better performance
The difference is modest, but we've used `slices.Sort` in lots of other
places so why not here.

name     old time/op    new time/op    delta
Builder    1.04µs ± 3%    0.95µs ± 3%   -8.27%  (p=0.008 n=5+5)

name     old alloc/op   new alloc/op   delta
Builder      312B ± 0%      288B ± 0%   -7.69%  (p=0.008 n=5+5)

name     old allocs/op  new allocs/op  delta
Builder      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.008 n=5+5)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 18:22:49 +00:00
Bryan Boreham a07a0be024 Add benchmark for labels.Builder
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 18:21:58 +00:00
Bryan Boreham 30297f0d9b stringlabels: size buffer for added labels
This makes the buffer the correct size for the common case that labels
have only been added. It will be too large for the case that labels are
changed, but the current buffer resize logic in `appendLabelTo` doubles
the buffer, so a small over-estimate is better.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 16:29:21 +00:00
Jonatan Ivanov 9225d118dc
Add Exemplars support for all time series
This change removes restrictions to allow adding exemplars
to all time series. It also contains some improvements in test values
so that it is easier to track what is tested.
The advantage of doing this is having a little less error-prone tests:
"yy" is not really descriptive but "counter-test" can give people
a better idea about what is tested so it is harder to make mistakes.

Closes gh-11982

Signed-off-by: Jonatan Ivanov <jonatan.ivanov@gmail.com>
2023-03-03 17:09:17 -08:00
Renning Bruns 5ec1b4baaf
make hashmod a lot easier to read and a little faster (#11551)
Previous code was effectively doing BigEndian.Uint64, so call that and save time.
An md5.Sum result is always 16 bytes. The first 8 are not used in the result, just as before.

Signed-off-by: Renning Bruns <ren@renmail.net>
2023-02-28 17:36:58 +00:00
Bryan Boreham 35026fb26d
Merge pull request #11746 from prometheus/remove-microbenchmarks
These benchmarks were testing things related to what Prometheus does, but not testing actual Prometheus code. 
Moved the label-copying benchmark into the labels package.
2023-02-23 12:33:24 +01:00
Levi Harrison 64ff6bece6
Merge pull request #11682 from dgrisonnet/parsing-errors
Improve the Prometheus parser error outputs to be more comprehensive
2023-02-22 17:35:27 -05:00
Bryan Boreham f03b8d0968 Add benchmark copying labels
Taken from previous tsdb/test/BenchmarkLabelsClone.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-02-22 16:36:45 +00:00
Damien Grisonnet 0fee615719 model/textparse: improve openmetrics error outputs
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2023-02-22 17:19:01 +01:00
Bryan Boreham 6136ae67e0 labels: shrink by making internals a single string
This commit adds an alternate implementation for `labels.Labels`, behind
a build tag `stringlabels`.

Instead of storing label names and values as individual strings, they
are all concatenated into one string in this format:

    [len][name0][len][value0][len][name1][len][value1]...

The lengths are varint encoded so usually a single byte.

The previous `[]string` had 24 bytes of overhead for the slice and 16
for each label name and value; this one has 16 bytes overhead plus 1
for each name and value.

In `ScratchBuilder.Overwrite` and `Labels.Hash` we use an unsafe
conversion from string to byte slice. `Overwrite` is explicitly unsafe,
but for `Hash` this is a pure performance hack.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-02-22 15:34:23 +00:00
Damien Grisonnet e3b5416d05 model/textparse: improve error outputs
Parsing errors in the Prometheus HTTP format parser are very hard to
investigate since they only approximately indicate what is going wrong
in the parser and don't provide any information about the incorrect
input. As such it is very hard to tell what is wrong in the format
exposed by the application.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2023-02-03 16:54:57 +01:00
Julien Pivotto 8e500dbd39 Add rulefmt tests
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-01-19 10:44:35 +01:00
Julien Pivotto ce55e5074d Add 'keep_firing_for' field to alerting rules
This commit adds a new 'keep_firing_for' field to Prometheus alerting
rules. The 'resolve_delay' field specifies the minimum amount of time
that an alert should remain firing, even if the expression does not
return any results.

This feature was discussed at a previous dev summit, and it was
determined that a feature like this would be useful in order to allow
the expression time to stabilize and prevent confusing resolved messages
from being propagated through Alertmanager.

This approach is simpler than having two PromQL queries, as was
sometimes discussed, and it should be easy to implement.

This commit does not include tests for the 'resolve_delay' field.  This
is intentional, as the purpose of this commit is to gather comments on
the proposed design of the 'resolve_delay' field before implementing
tests. Once the design of the 'resolve_delay' field has been finalized,
a follow-up commit will be submitted with tests."

See https://github.com/prometheus/prometheus/issues/11570

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-01-13 12:11:39 +01:00
beorn7 3e5ad99c33 textparse: Minor doc comment improvement
Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-11 00:30:55 +01:00
beorn7 3f977e89ab textparse: Add tests for float and gauge histograms
Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-10 18:42:05 +01:00
beorn7 c26891043f textparse: Set CounterResetHint
If a (float or integer) histogram is a gauge histogram, set the
CounterResetHint accordingly. (The default value is fine for the
normal counter histograms.)

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-10 16:25:23 +01:00
Björn Rabenstein f07adbd45f
Merge pull request #11814 from prometheus/beorn7/protobuf
textparse: Add gauge histogram support
2023-01-10 15:18:44 +01:00
Ganesh Vernekar 57bcbf1888
Merge pull request #11783 from codesome/gauge-histogram
tsdb: Add gauge histogram support
2023-01-10 19:06:08 +05:30
Ganesh Vernekar a87e7e9e33
tsdb: Add counter reset hint to histograms and support in WAL
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:41:53 +05:30
Ganesh Vernekar c94a41c4b2
Merge pull request #11785 from Fish-pro/erroris
Use errors.Is to check for a specific error
2023-01-10 14:56:14 +05:30
beorn7 b5d4a94e9d textparse: Add gauge histogram support to protobuf parsing
With this commit, the parser stops to see a gauge histogram (whether
native or conventional) as an unexpected metric type. It ingests it
normally, it even sets the `GaugeHistogram` type in the metadata (as
it has already done for a conventional gauge histogram scraped using
OpenMetrics), but it otherwise treats it as a normal counter-like
histogram.

Once #11783 is merged, though, it should be very easy to utilize the
type information.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-05 15:39:10 +01:00
beorn7 e9d9bb1b08 textparse: Handle unknown metric types in protobuf gracefully
So far, the parser hasn't validated that the type is valid in the
`Next()` call. Later, in the `Series()` call, however, it assumes that
we will only see valid types and therefore panics with `encountered
unexpected metric type, this is a bug`.

This commit fixes said bug by adding validation to the `Next()` call.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-05 15:21:18 +01:00