Commit graph

89 commits

Author SHA1 Message Date
Manik Rana e9391dd61f feat: document deepcopyparser
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 9c4bc807be fix: comments
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 8ea8352fdc chore: cleanup
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana e37930d973 chore: update go fmt version
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 4102f415d3 fix: return nil when mtype doesn't match
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 50615b4b80 chore: remove unneeded comments
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana eba98775c7 fix: retore failing tests
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 7708b78a43 fix: restore previous tests
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 96e850df7e feat: advance parser and parse histograms + summary
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana ea9c8cb5a3 fix: check if err from p.Next()
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana e841e00eed feat: multiple changes
- implement changes from pair programming session
- use newParse.val()
- advance parser p if ct is found

Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 188d4cc927 chore: add comments
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana cbb4e07aa2 chore: remove unneeded test
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 03f2a90394 feat: return ct after finding it
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Manik Rana 7e8899241a feat: initial implement of createedTimestamp() with tests
Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-12 00:48:36 +05:30
Arthur Silva Sens 07355c9199
Bump client_golang to 1.19
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
2024-03-06 09:11:13 -03:00
Ondrej Kokes 2e30f1231b docs: textparse.Parser return type mismatch
The docs suggest the Next method returns a bool, but that's not the case (`Entry` is an int).

```
	// Next advances the parser to the next sample. It returns false if no
	// more samples were read or an error occurred.
	Next() (Entry, error)
```

The docs were first added in d80a3de235 in 2017. Back then the signature was
indeed `func (p *Parser) Next() bool`. But then it got refactored in 76a4a46cb0
and the signature changed with it, yet docs stayed the same - and eventually made their way into the `Parser` interface.

However, the Protobuf parser does have the right wording: 5de2df752f

```
// Next advances the parser to the next "sample" (emulating the behavior of a
// text format parser). It returns (EntryInvalid, io.EOF) if no samples were
// read.
```

Changing all other implementations (and the interface itself) to match this doc.

Signed-off-by: Ondrej Kokes <ondrej.kokes@gmail.com>
2024-02-29 16:45:05 +01:00
Bryan Boreham ba6aa95222 refactor: metrics parsing: extract common checking code
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 16:37:41 +00:00
Bryan Boreham eff3a13e19 model/textparse: parsers take a labels SymbolTable
This allows strings to be interned to save memory.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 93b72ec5dd tsdb: create SymbolTables for labels as required
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Łukasz Mierzwa 5597020a60 Use github.com/klauspost/compress for gzip and zlib
klauspost/compress is a high quality drop-in replacement for common Go
compression libraries. Since Prometheus sends out a lot of HTTP requests
that often return compressed output having improved compression
libraries helps to save cpu & memory resources.
On a test Prometheus server I was able to see cpu reduction from 31 to
30 cores.

Benchmark results:

name                                old time/op    new time/op    delta
TargetScraperGzip/metrics=1-8         69.4µs ± 4%    69.2µs ± 3%     ~     (p=0.122 n=50+50)
TargetScraperGzip/metrics=100-8       84.3µs ± 2%    80.9µs ± 2%   -4.02%  (p=0.000 n=48+46)
TargetScraperGzip/metrics=1000-8       296µs ± 1%     274µs ±14%   -7.35%  (p=0.000 n=47+45)
TargetScraperGzip/metrics=10000-8     2.06ms ± 1%    1.66ms ± 2%  -19.34%  (p=0.000 n=47+45)
TargetScraperGzip/metrics=100000-8    20.9ms ± 2%    17.5ms ± 3%  -16.50%  (p=0.000 n=49+50)

name                                old alloc/op   new alloc/op   delta
TargetScraperGzip/metrics=1-8         6.06kB ± 0%    6.07kB ± 0%   +0.24%  (p=0.000 n=48+48)
TargetScraperGzip/metrics=100-8       7.04kB ± 0%    6.89kB ± 0%   -2.17%  (p=0.000 n=49+50)
TargetScraperGzip/metrics=1000-8      9.02kB ± 0%    8.35kB ± 1%   -7.49%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=10000-8     18.1kB ± 1%    16.1kB ± 2%  -10.87%  (p=0.000 n=47+47)
TargetScraperGzip/metrics=100000-8    1.21MB ± 0%    1.01MB ± 2%  -16.69%  (p=0.000 n=36+50)

name                                old allocs/op  new allocs/op  delta
TargetScraperGzip/metrics=1-8           71.0 ± 0%      72.0 ± 0%   +1.41%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=100-8         81.0 ± 0%      76.0 ± 0%   -6.17%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=1000-8        92.0 ± 0%      83.0 ± 0%   -9.78%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=10000-8       93.0 ± 0%      91.0 ± 0%   -2.15%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=100000-8       111 ± 0%       135 ± 1%  +21.89%  (p=0.000 n=40+50)

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2024-02-22 17:08:15 +00:00
Björn Rabenstein ac10cd4d99
Merge pull request #13271 from ywwg/owilliams/utf8
UTF-8: Add partial support for parsing UTF-8 metric and label names
2024-02-16 11:42:55 +01:00
Owen Williams a28d7865ad UTF-8: Add support for parsing UTF8 metric and label names
This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats.
This grammar will also be valid for non-UTF-8 names.
UTF-8 names will not be considered valid unless model.NameValidationScheme is changed.

This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/.

Part of https://github.com/prometheus/prometheus/issues/13095

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-02-15 14:34:37 -05:00
Ziqi Zhao a93859a52f
Prometheus support parse exemplars from native histogram (#13488)
scrape: support parsing exemplars from native histogram

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-02-14 17:24:40 +01:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Björn Rabenstein 3db4596965
Merge pull request #13299 from vesari/add-unit-parser
protobuf: add unit parser
2024-01-09 14:47:50 +01:00
Arianna Vespri 8f07f9dd90 Avoid creating string for suffix, consider counters without _total suffix
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-28 15:41:38 +01:00
Arianna Vespri 9a664b515a Correct order in error message
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:41:37 +01:00
Arianna Vespri 51e78d9a32 Delete debugging lines, amend error message for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-20 09:31:58 +01:00
Bryan Boreham c83e1fc574 textparse: remove MetricType alias
No backwards-compatibility; make a clean break.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Bryan Boreham 8065bef172 Move metric type definitions to common/model
They are used in multiple repos, so common is a better place for them.
Several packages now don't depend on `model/textparse`, e.g.
`storage/remote`.

Also remove `metadata` struct from `api.go`, since it was identical to
a struct in the `metadata` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Arianna Vespri 3d6a92a25e Merge remote-tracking branch 'prometheus/main' into add-unit-parser 2023-12-17 14:11:55 +01:00
Arianna Vespri b65021d7a5 Get conditional right
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-17 14:09:14 +01:00
Arianna Vespri 9fb1e9715c Go on adding protobuf parsing for unit
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-14 17:00:52 +01:00
Arianna Vespri 5d5acf5744 Add unit protobuf parser
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-12 12:40:08 +01:00
Arthur Silva Sens 5082655392
Append Created Timestamps (#12733)
* Append created timestamps.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Log when created timestamps are ignored

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Proposed changes to Append CT PR.

Changes:

* Changed textparse Parser interface for consistency and robustness.
* Changed CT interface to be more explicit and handle validation.
* Simplified test, change scrapeManager to allow testability.
* Added TODOs.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Updates.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Refactor head_appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Fix linter issues

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Use model.Sample in head appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
2023-12-11 08:43:42 +00:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Matthieu MOREL fe057fc60d use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-03 07:26:31 +00:00
beorn7 b428416f06 textparse: Update comment about timestamp_ms protobuf parsing
By now, we know better what the plan is.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-10-19 17:54:42 +02:00
Arthur Silva Sens ef8e6ae780
Parse created timestamps from Prometheus Protobuf (#12973)
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
2023-10-18 19:04:02 +01:00
Matthieu MOREL 0a513f827d ci(lint): enable errorlint linter on model
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-09-29 21:11:33 +00:00
beorn7 65ccf4460a textparse: Fix endless loop #12731
PR #12557 introduced the possibility of parsing multiple exemplars per
native histograms. It did so by requiring the `Exemplar` method of the
parser to be called repeatedly until it returns false. However, the
protobuf parser code wasn't correctly updated for the old case of a
single exemplar for a classic bucket (if actually parsed as a classic
bucket) and a single exemplar on a counter. In those cases, the method
would return `true` forever, yielding the same exemplar again and
again, leading to an endless loop.

With this fix, the state is now tracked and the single exemplar is
only returned once.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 21:03:54 +02:00
beorn7 3d9a830f2f textparse: Expose #12731 in protobufparse_test.go
Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 20:53:31 +02:00
György Krajcsovits e846736134 Fix typo
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-08-21 14:13:49 +02:00
beorn7 c58e20ad0e histogram: Identify native histograms even without observations
Native histograms without observations and with a zero threshold of
zero look the same as classic histograms in the protobuf exposition
format. According to
https://github.com/prometheus/client_golang/issues/1127 , the idea is
to add a no-op span to those histograms to mark them as native
histograms. This commit enables Prometheus to detect that no-op span
and adds a doc comment to the proto spec describing the behavior.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-26 17:42:30 +02:00
Julien Pivotto 7905594b52
Merge pull request #12557 from prometheus/beorn7/histogram
scrape: Enable ingestion of multiple exemplars per sample
2023-07-20 15:19:28 +02:00
beorn7 071f4bbea4 histograms: Fix parsing float histograms without zero bucket
If a float histogram has a zero bucket with a threshold of zero _and_
an empty zero bucket, it wasn't identified as a native histogram
because the `isNativeHistogram` helper function only looked at integer
buckets.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 15:29:11 +02:00
beorn7 2ea8df4734 histogram: Expose #12305
Native histograms without a zero threshold aren't federated properly.

This adds a test to prove the specific failure mode, which is that
histograms with a zero threshold of zero are federated as classic
histograms.

The underlying reason is that the protobuf parser identifies a native
histogram by detecting a zero bucket or by detecting integer buckets.
Therefore, a float histogram with a zero threshold of zero and an
unpopulated zero bucket falls through the cracks (no integer buckets,
no zero bucket).

This commit also addse a test case for the latter.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 15:29:11 +02:00
beorn7 0e3f35324b scrape: Enable ingestion of multiple exemplars per sample
This has become a requirement for native histograms, as a single
histogram sample commonly has many buckets, so that providing many
exemplars makes sense.

Since OM text doesn't support native histograms yet, the test had to
be expanded to also support protobuf test cases.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-13 14:16:10 +02:00