94 commits

Author	SHA1	Message	Date
György Krajcsovits	71fd2d93a9	Merge branch 'main' into nhcb-scrape-impl ... # Conflicts: # config/config.go # scrape/scrape.go	2024-09-25 13:43:57 +02:00
Jeanette Tan	97ba2fc39d	use caps for NHCB ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:38:30 +02:00
Jeanette Tan	f35c6649e4	don't blindly convert series with the classic histogram name suffixes if they are not actually histograms based on metadata ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:38:15 +02:00
Jeanette Tan	e3899187da	expand tests for protobuf and fix problems ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:38:08 +02:00
Jeanette Tan	cd498964e6	expand tests and support conversion to nhcb in the middle of scrape ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:38:04 +02:00
Jeanette Tan	41c7f7d352	don't reuse the buffer ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:38:01 +02:00
Jeanette Tan	57bde06d2c	add doc comments ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:37:56 +02:00
Jeanette Tan	cbd5488cd3	skip nhcb conversion if there is native histogram of same name ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:37:51 +02:00
Jeanette Tan	0e5072b873	keep only 1 nhcb in memory at at time ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:37:46 +02:00
Jeanette Tan	172d4f2405	insert nhcb parser as intermediate layer ... Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2024-09-25 13:37:37 +02:00
Bryan Boreham	31c5760551	Neater string vs byte-slice conversions (#14425) ... unsafe.Slice and unsafe.StringData were added in Go 1.20 Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-21 12:19:21 +02:00
Manik Rana	315165e49d	refac: make typeRequiresCT private ... Signed-off-by: Manik Rana <manikrana54@gmail.com>	2024-09-20 21:56:19 +05:30
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868) ... * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2024-09-10 22:32:03 +02:00
Björn Rabenstein	4ffb74dae4	Merge pull request #14810 from bboreham/fix-proto-parse-exemplar ... [BUGFIX] Protobuf scraping: reset exemplar position	2024-09-03 15:05:59 +02:00
Bryan Boreham	d599c4b28c	[BUGFIX] Protobuf scraping: reset exemplar position ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-02 16:43:50 +01:00
Bryan Boreham	3aaf2c3c9d	[TESTS] Add second native histogram with exemplars in TestProtobufParse ... This test fails. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-02 16:43:50 +01:00
Bryan Boreham	e8ecc11816	[TESTS] Fix up item numbers in TestProtobufParse ... If an error is thrown by the test, these numbers help you see which item is wrong. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-09-02 16:43:49 +01:00
riskrole	406bf775aa	chore: fix some comments ... Signed-off-by: riskrole <yuhang@before.tech>	2024-08-28 11:26:57 +08:00
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments ... Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2024-08-22 17:36:11 +02:00
Manik Rana	02c465bf58	textparse: Implement CreatedTimestamp() in openmetricsparse.go (#14356) ... * feat: initial implement of createedTimestamp() with tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: return ct after finding it Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove unneeded test Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: multiple changes - implement changes from pair programming session - use newParse.val() - advance parser p if ct is found Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: check if err from p.Next() Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: advance parser and parse histograms + summary Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: restore previous tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: retore failing tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove unneeded comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: return nil when mtype doesn't match Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: update go fmt version Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: cleanup Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: document deepcopyparser Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: cover edgecase of `gauge_created` in CreatedTimestamp() Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: readability updates Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: dedeuplicate labeldiff checks Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add tests for new label functions Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: document CreatedTimestamp func Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: optimize `CreatedTimestamp()` - Use refactored CreatedTimestamp function with bug fixes - Remove unused code in labels.go - Improve code documentation Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add tests and lint fixes Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: remove mName Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add tests for CT parse failures and deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: edit expectCT struct Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: add new label in deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: use p.builder in deepCopy Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: add NewMetricsParserWithOpts Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * chore: comments Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * chore: rename var Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: add condition for OM fuzzing Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: build tags Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: default skipCT to false Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: rename skipCT to skipCTSeries Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: formatting Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments and readability updates Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * refac: remove NewOpenMetricsParserWithOpts Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: extract skipCTSeries logic from parseMetricSuffix Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: inline create a NewOpenMetricsParser Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: comments Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * refac: improve error handling Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: return error instead of nil Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: remove skipCT check from tBraceOpen Signed-off-by: Manik Rana <manikrana54@gmail.com> * Pair programming with Manik, Arthur and Daniel. Signed-off-by: bwplotka <bwplotka@gmail.com> * chore: comments and use helper funcs Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> --------- Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2024-08-08 12:35:35 +01:00
Arthur Silva Sens	07355c9199	Bump client_golang to 1.19 ... Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>	2024-03-06 09:11:13 -03:00
Ondrej Kokes	2e30f1231b	docs: textparse.Parser return type mismatch ... The docs suggest the Next method returns a bool, but that's not the case (`Entry` is an int). ``` // Next advances the parser to the next sample. It returns false if no // more samples were read or an error occurred. Next() (Entry, error) ``` The docs were first added in d80a3de235 in 2017. Back then the signature was indeed `func (p *Parser) Next() bool`. But then it got refactored in 76a4a46cb0 and the signature changed with it, yet docs stayed the same - and eventually made their way into the `Parser` interface. However, the Protobuf parser does have the right wording: 5de2df752f ``` // Next advances the parser to the next "sample" (emulating the behavior of a // text format parser). It returns (EntryInvalid, io.EOF) if no samples were // read. ``` Changing all other implementations (and the interface itself) to match this doc. Signed-off-by: Ondrej Kokes <ondrej.kokes@gmail.com>	2024-02-29 16:45:05 +01:00
Bryan Boreham	ba6aa95222	refactor: metrics parsing: extract common checking code ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-26 16:37:41 +00:00
Bryan Boreham	eff3a13e19	model/textparse: parsers take a labels SymbolTable ... This allows strings to be interned to save memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-26 11:45:25 +00:00
Bryan Boreham	93b72ec5dd	tsdb: create SymbolTables for labels as required ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-26 11:45:25 +00:00
Łukasz Mierzwa	5597020a60	Use github.com/klauspost/compress for gzip and zlib ... klauspost/compress is a high quality drop-in replacement for common Go compression libraries. Since Prometheus sends out a lot of HTTP requests that often return compressed output having improved compression libraries helps to save cpu & memory resources. On a test Prometheus server I was able to see cpu reduction from 31 to 30 cores. Benchmark results: name old time/op new time/op delta TargetScraperGzip/metrics=1-8 69.4µs ± 4% 69.2µs ± 3% ~ (p=0.122 n=50+50) TargetScraperGzip/metrics=100-8 84.3µs ± 2% 80.9µs ± 2% -4.02% (p=0.000 n=48+46) TargetScraperGzip/metrics=1000-8 296µs ± 1% 274µs ±14% -7.35% (p=0.000 n=47+45) TargetScraperGzip/metrics=10000-8 2.06ms ± 1% 1.66ms ± 2% -19.34% (p=0.000 n=47+45) TargetScraperGzip/metrics=100000-8 20.9ms ± 2% 17.5ms ± 3% -16.50% (p=0.000 n=49+50) name old alloc/op new alloc/op delta TargetScraperGzip/metrics=1-8 6.06kB ± 0% 6.07kB ± 0% +0.24% (p=0.000 n=48+48) TargetScraperGzip/metrics=100-8 7.04kB ± 0% 6.89kB ± 0% -2.17% (p=0.000 n=49+50) TargetScraperGzip/metrics=1000-8 9.02kB ± 0% 8.35kB ± 1% -7.49% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 18.1kB ± 1% 16.1kB ± 2% -10.87% (p=0.000 n=47+47) TargetScraperGzip/metrics=100000-8 1.21MB ± 0% 1.01MB ± 2% -16.69% (p=0.000 n=36+50) name old allocs/op new allocs/op delta TargetScraperGzip/metrics=1-8 71.0 ± 0% 72.0 ± 0% +1.41% (p=0.000 n=50+50) TargetScraperGzip/metrics=100-8 81.0 ± 0% 76.0 ± 0% -6.17% (p=0.000 n=50+50) TargetScraperGzip/metrics=1000-8 92.0 ± 0% 83.0 ± 0% -9.78% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 93.0 ± 0% 91.0 ± 0% -2.15% (p=0.000 n=50+50) TargetScraperGzip/metrics=100000-8 111 ± 0% 135 ± 1% +21.89% (p=0.000 n=40+50) Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2024-02-22 17:08:15 +00:00
Björn Rabenstein	ac10cd4d99	Merge pull request #13271 from ywwg/owilliams/utf8 ... UTF-8: Add partial support for parsing UTF-8 metric and label names	2024-02-16 11:42:55 +01:00
Owen Williams	a28d7865ad	UTF-8: Add support for parsing UTF8 metric and label names ... This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats. This grammar will also be valid for non-UTF-8 names. UTF-8 names will not be considered valid unless model.NameValidationScheme is changed. This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/. Part of https://github.com/prometheus/prometheus/issues/13095 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-02-15 14:34:37 -05:00
Ziqi Zhao	a93859a52f	Prometheus support parse exemplars from native histogram (#13488) ... scrape: support parsing exemplars from native histogram --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2024-02-14 17:24:40 +01:00
Bryan Boreham	17f48f2b3b	Tests: use replacement DeepEquals in more places ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-08 19:32:33 +00:00
Björn Rabenstein	3db4596965	Merge pull request #13299 from vesari/add-unit-parser ... protobuf: add unit parser	2024-01-09 14:47:50 +01:00
Arianna Vespri	8f07f9dd90	Avoid creating string for suffix, consider counters without _total suffix ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-28 15:41:38 +01:00
Arianna Vespri	9a664b515a	Correct order in error message ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-20 09:41:37 +01:00
Arianna Vespri	51e78d9a32	Delete debugging lines, amend error message for unit ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-20 09:31:58 +01:00
Bryan Boreham	c83e1fc574	textparse: remove MetricType alias ... No backwards-compatibility; make a clean break. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2023-12-19 18:56:54 +00:00
Bryan Boreham	8065bef172	Move metric type definitions to common/model ... They are used in multiple repos, so common is a better place for them. Several packages now don't depend on `model/textparse`, e.g. `storage/remote`. Also remove `metadata` struct from `api.go`, since it was identical to a struct in the `metadata` package. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2023-12-19 18:56:54 +00:00
Arianna Vespri	3d6a92a25e	Merge remote-tracking branch 'prometheus/main' into add-unit-parser	2023-12-17 14:11:55 +01:00
Arianna Vespri	b65021d7a5	Get conditional right ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-17 14:09:14 +01:00
Arianna Vespri	9fb1e9715c	Go on adding protobuf parsing for unit ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-14 17:00:52 +01:00
Arianna Vespri	5d5acf5744	Add unit protobuf parser ... Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>	2023-12-12 12:40:08 +01:00
Arthur Silva Sens	5082655392	Append Created Timestamps (#12733) ... * Append created timestamps. Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Log when created timestamps are ignored Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Proposed changes to Append CT PR. Changes: * Changed textparse Parser interface for consistency and robustness. * Changed CT interface to be more explicit and handle validation. * Simplified test, change scrapeManager to allow testability. * Added TODOs. Signed-off-by: bwplotka <bwplotka@gmail.com> * Updates. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * Refactor head_appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Fix linter issues Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Use model.Sample in head appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> --------- Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: bwplotka <bwplotka@gmail.com>	2023-12-11 08:43:42 +00:00
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254) ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2023-12-07 11:35:01 +00:00
zenador	32ee1b15de	Fix error on ingesting out-of-order exemplars (#13021) ... Fix and improve ingesting exemplars for native histograms. See code comment for a detailed explanation of the algorithm. Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed. Also note that we now do not ingest exemplars without timestamps for native histograms anymore. Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: zenador <zenador@users.noreply.github.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2023-11-16 15:07:37 +01:00
Matthieu MOREL	fe057fc60d	use Go standard errors package ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2023-11-03 07:26:31 +00:00
beorn7	b428416f06	textparse: Update comment about timestamp_ms protobuf parsing ... By now, we know better what the plan is. Signed-off-by: beorn7 <beorn@grafana.com>	2023-10-19 17:54:42 +02:00
Arthur Silva Sens	ef8e6ae780	Parse created timestamps from Prometheus Protobuf (#12973) ... Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>	2023-10-18 19:04:02 +01:00
Matthieu MOREL	0a513f827d	ci(lint): enable errorlint linter on model ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2023-09-29 21:11:33 +00:00
beorn7	65ccf4460a	textparse: Fix endless loop #12731 ... PR #12557 introduced the possibility of parsing multiple exemplars per native histograms. It did so by requiring the `Exemplar` method of the parser to be called repeatedly until it returns false. However, the protobuf parser code wasn't correctly updated for the old case of a single exemplar for a classic bucket (if actually parsed as a classic bucket) and a single exemplar on a counter. In those cases, the method would return `true` forever, yielding the same exemplar again and again, leading to an endless loop. With this fix, the state is now tracked and the single exemplar is only returned once. Signed-off-by: beorn7 <beorn@grafana.com>	2023-08-22 21:03:54 +02:00
beorn7	3d9a830f2f	textparse: Expose #12731 in protobufparse_test.go ... Signed-off-by: beorn7 <beorn@grafana.com>	2023-08-22 20:53:31 +02:00
György Krajcsovits	e846736134	Fix typo ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2023-08-21 14:13:49 +02:00