mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2025-01-03 18:07:27 -08:00
Code Issues Actions 10 Packages Projects Releases Wiki Activity
13148 commits 274 branches 448 tags 658 MiB
Commit graph

345 commits

Author SHA1 Message Date
Jeanette Tan 172d4f2405 insert nhcb parser as intermediate layer 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:37:37 +02:00
Jeanette Tan f596f17024 allow option to convert classic histograms to nhcb entirely (don't append classic histogram series) 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:37:30 +02:00
Jeanette Tan 02d5abf60e don't use cache for nhcb maps 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:37:26 +02:00
Jeanette Tan 0a321fe4d8 improve new scrape test 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:37:20 +02:00
Jeanette Tan 4503145c8b convert classic histograms to int nhcb where possible instead 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:37:08 +02:00
Jeanette Tan 62e7f0438d implement basic conversion of classic to nhcb in scrape 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-09-25 13:36:56 +02:00
György Krajcsovits 79020b1e85 Comment float histogram as well 
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
 2024-06-18 15:22:03 +02:00
György Krajcsovits c309f50ee7 Add comment to state intent of check 
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
 2024-06-18 15:21:17 +02:00
György Krajcsovits 0793a26d96 native histograms: only reduce resolution for exponential histograms 
Currently we can only reduce the resolution of exponential native
histograms, so checking the schema for that is slightly more precise
than checking against max schema.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
 2024-06-11 11:17:14 +02:00
Jeanette Tan 14f8dded39 Merge branch 'main' into nhcb 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-06-07 19:17:14 +08:00
Jeanette Tan 9adc1699c3 fix according to code review 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-06-07 18:50:59 +08:00
Oleksandr Redko f10c3454e9 Enable perfsprint linter and fix up code 
Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>
 2024-05-15 17:51:05 +03:00
Jeanette Tan 796b1bbfde Merge branch 'main' into nhcb 
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-05-08 19:11:39 +08:00
Arthur Silva Sens 7aacef9b42
bugfix: Decouple native histogram ingestions and protobuf parsing 
Up until this point, if a scrape was done with the protobuf format Prometheus would always try to ingest native histograms even with the feature flag disabled. This causes problems with other feature-flags that depend on the protobuf format, like 'created-timestamp-zero-ingestion'. This commit decouples native histogram parsing from ingestion, making sure ingestion only happens when the 'native-histogram' feature-flag is enabled.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
 2024-04-24 17:02:52 -03:00
György Krajcsovits bcafa5f1f9 Merge remote-tracking branch 'upstream/main' into update-nhcb 2024-04-24 11:06:59 +02:00
Matthieu MOREL 6f595c6762
golangci-lint: enable whitespace linter (#13905) 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-04-11 09:27:54 +01:00
Matthieu MOREL d496687c8e golangci-lint: enable usestdlibvars linter 
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
 2024-04-08 19:26:23 +00:00
David Ashpole c755fa9935
support unregistering scrape manager metrics 
Signed-off-by: David Ashpole <dashpole@google.com>
 2024-04-05 16:00:52 +00:00
György Krajcsovits 2a4aa085d2 Merge branch 'main' into nhcb 2024-03-27 18:42:10 +01:00
Ziqi Zhao 64dfd8a158
fix the bug of setting native histogram min bucket factor (#13846) 
* fix the bug of setting native histogram min bucket factor

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

* Add unit test for checking that min_bucket_factor is correctly applied

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
 2024-03-27 16:32:37 +01:00
György Krajcsovits a3d1a46eda Merge branch 'main' into nhcb 2024-03-22 14:51:48 +01:00
zenador 4acbb7dea6
Add custom buckets to native histogram chunks encoding (#13706) 
* add custom bounds to chunks encoding
* change custom buckets schema number
* rename custom bounds to custom values

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-03-22 14:36:39 +01:00
Bryan Boreham 5ed21c0d76
Merge pull request #12933 from prymitive/duplicated_samples 
When Prometheus scrapes a target and it sees the same time series repeated multiple times it currently silently ignores that. This change adds a test for that and fixes the scrape loop so that:

* Only first sample for each unique time series is appended
* Duplicated samples increment the prometheus_target_scrapes_sample_duplicate_timestamp_total metric
This allows one to identify such scrape jobs and targets.

Also fix some tests and benchmark.
 2024-03-16 09:18:46 +00:00
Bryan Boreham 6c41ec984f [BUGFIX] Scraping: Tolerance should be max 1% of interval 
Previous code set it at minimum 1%, which was not intended.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-03-08 10:18:18 +00:00
Julien fcdb6c2b14
Merge pull request #13624 from roidelapluie/tolerance 
Always align scrapes even if tolerance is bigger than 1% of scrape interval
 2024-03-04 16:37:00 +01:00
machine424 f477e0539a
Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 
Prevent adding back golang.org/x/exp/slices.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
 2024-02-28 14:54:53 +01:00
György Krajcsovits 5d0a0a7542 Add custom buckets to native histogram model (#13592) 
* add custom buckets to native histogram model
* simple copy for custom bounds
* return errors for unsupported add/sub operations
* add test cases for string and update appendhistogram in scrape to account for new schema
* check fields which are supposed to be unused but may affect results in equals
* allow appending custom buckets histograms regardless of max schema

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
 2024-02-28 14:06:43 +01:00
Łukasz Mierzwa c013a3c1b5 Check of duplicated samples directly in scrapeCache.get() 
Avoid extra map lookup by hooking check into cache get.

```
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/scrape
cpu: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
                     │  main.txt   │              new.txt               │
                     │   sec/op    │   sec/op     vs base               │
ScrapeLoopAppend-8     66.72µ ± 0%   66.89µ ± 0%       ~ (p=0.879 n=50)
ScrapeLoopAppendOM-8   66.61µ ± 0%   66.89µ ± 1%       ~ (p=0.115 n=50)
geomean                66.66µ        66.89µ       +0.34%

                     │   main.txt    │               new.txt                │
                     │     B/op      │     B/op      vs base                │
ScrapeLoopAppend-8     20.17Ki ±  1%   20.12Ki ± 1%        ~ (p=0.343 n=50)
ScrapeLoopAppendOM-8   20.38Ki ± 10%   17.99Ki ± 2%  -11.69% (p=0.017 n=50)
geomean                20.27Ki         19.03Ki        -6.14%

                     │  main.txt  │               new.txt               │
                     │ allocs/op  │ allocs/op   vs base                 │
ScrapeLoopAppend-8     11.00 ± 0%   11.00 ± 0%       ~ (p=1.000 n=50) ¹
ScrapeLoopAppendOM-8   12.00 ± 0%   12.00 ± 0%       ~ (p=1.000 n=50) ¹
geomean                11.49        11.49       +0.00%
¹ all samples are equal
```

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-27 12:09:37 +00:00
Łukasz Mierzwa 21f8b35f5b Move staleness tracking out of checkAddError() calls 
This call bloats checkAddError signature and logic, we can and should call it from the main scrape logic.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-27 11:36:16 +00:00
Łukasz Mierzwa 55dcaab41b Fix TestScrapeLoopDiscardDuplicateLabels test 
This test calls Rollback() which is normally called from within append code.
Doing so means that staleness tracking data is outdated and need to by cycled manually.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-27 11:36:16 +00:00
Łukasz Mierzwa 50c81bed86 Check for duplicated series on a scrape 
When Prometheus scrapes a target and it sees the same time series repeated multiple times it currently silently ignores that.
This change adds a test for that and fixes the scrape loop so that:

- Only first sample for each unique time series is appended
- Duplicated samples increment the prometheus_target_scrapes_sample_duplicate_timestamp_total metric

This allows one to identify such scrape jobs and targets.

Benchmark results:

```
name                            old time/op    new time/op    delta
ScrapeLoopAppend-8                64.8µs ± 2%    71.1µs ±20%   +9.75%  (p=0.000 n=10+10)
ScrapeLoopAppendOM-8              64.2µs ± 1%    68.5µs ± 7%   +6.71%  (p=0.000 n=9+10)
TargetsFromGroup/1_targets-8      14.2µs ± 1%    14.5µs ± 1%   +1.99%  (p=0.000 n=10+10)
TargetsFromGroup/10_targets-8      149µs ± 1%     152µs ± 1%   +2.05%  (p=0.000 n=9+10)
TargetsFromGroup/100_targets-8    1.49ms ± 4%    1.48ms ± 1%     ~     (p=0.796 n=10+10)

name                            old alloc/op   new alloc/op   delta
ScrapeLoopAppend-8                19.9kB ± 1%    17.8kB ± 3%  -10.23%  (p=0.000 n=8+10)
ScrapeLoopAppendOM-8              19.9kB ± 1%    18.3kB ±10%   -8.14%  (p=0.001 n=9+10)
TargetsFromGroup/1_targets-8      2.43kB ± 0%    2.43kB ± 0%   -0.15%  (p=0.045 n=10+10)
TargetsFromGroup/10_targets-8     24.3kB ± 0%    24.3kB ± 0%     ~     (p=0.083 n=10+9)
TargetsFromGroup/100_targets-8     243kB ± 0%     243kB ± 0%     ~     (p=0.720 n=9+10)

name                            old allocs/op  new allocs/op  delta
ScrapeLoopAppend-8                  9.00 ± 0%      9.00 ± 0%     ~     (all equal)
ScrapeLoopAppendOM-8                10.0 ± 0%      10.0 ± 0%     ~     (all equal)
TargetsFromGroup/1_targets-8        40.0 ± 0%      40.0 ± 0%     ~     (all equal)
TargetsFromGroup/10_targets-8        400 ± 0%       400 ± 0%     ~     (all equal)
TargetsFromGroup/100_targets-8     4.00k ± 0%     4.00k ± 0%     ~     (all equal)
```

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-27 11:36:16 +00:00
Łukasz Mierzwa 1a8ea78207 Fix BenchmarkScrapeLoopAppendOM 
OpenMetrics requires EOF comment at the end of metrics body, but the makeTestMetrics() function doesn't append it.
This means this benchmark tests a response with errors but I don't think that was the intention.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-27 11:36:16 +00:00
Bryan Boreham 5f50d974c9 scraping: reset symbol table periodically 
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-26 11:45:25 +00:00
Bryan Boreham 4e748b9cd8 scraping: re-use labels Builder in scrape report metrics 
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-26 11:45:25 +00:00
Bryan Boreham abb3a62f04 scraping: re-use symbol table for scrape loops 
One symbol table for all loops in the same scrape pool, i.e. from the
same job.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-26 11:45:25 +00:00
Bryan Boreham 0403d098e1 scraping: re-use symbolTable for target discovery 
Call labels.NewBuilderWithSymbolTable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-26 11:45:25 +00:00
Bryan Boreham 9ba13de220 scraping: pass a Builder to get Target labels 
This saves memory allocations from making a new Builder every time.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-26 11:45:25 +00:00
Łukasz Mierzwa 5597020a60 Use github.com/klauspost/compress for gzip and zlib 
klauspost/compress is a high quality drop-in replacement for common Go
compression libraries. Since Prometheus sends out a lot of HTTP requests
that often return compressed output having improved compression
libraries helps to save cpu & memory resources.
On a test Prometheus server I was able to see cpu reduction from 31 to
30 cores.

Benchmark results:

name                                old time/op    new time/op    delta
TargetScraperGzip/metrics=1-8         69.4µs ± 4%    69.2µs ± 3%     ~     (p=0.122 n=50+50)
TargetScraperGzip/metrics=100-8       84.3µs ± 2%    80.9µs ± 2%   -4.02%  (p=0.000 n=48+46)
TargetScraperGzip/metrics=1000-8       296µs ± 1%     274µs ±14%   -7.35%  (p=0.000 n=47+45)
TargetScraperGzip/metrics=10000-8     2.06ms ± 1%    1.66ms ± 2%  -19.34%  (p=0.000 n=47+45)
TargetScraperGzip/metrics=100000-8    20.9ms ± 2%    17.5ms ± 3%  -16.50%  (p=0.000 n=49+50)

name                                old alloc/op   new alloc/op   delta
TargetScraperGzip/metrics=1-8         6.06kB ± 0%    6.07kB ± 0%   +0.24%  (p=0.000 n=48+48)
TargetScraperGzip/metrics=100-8       7.04kB ± 0%    6.89kB ± 0%   -2.17%  (p=0.000 n=49+50)
TargetScraperGzip/metrics=1000-8      9.02kB ± 0%    8.35kB ± 1%   -7.49%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=10000-8     18.1kB ± 1%    16.1kB ± 2%  -10.87%  (p=0.000 n=47+47)
TargetScraperGzip/metrics=100000-8    1.21MB ± 0%    1.01MB ± 2%  -16.69%  (p=0.000 n=36+50)

name                                old allocs/op  new allocs/op  delta
TargetScraperGzip/metrics=1-8           71.0 ± 0%      72.0 ± 0%   +1.41%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=100-8         81.0 ± 0%      76.0 ± 0%   -6.17%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=1000-8        92.0 ± 0%      83.0 ± 0%   -9.78%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=10000-8       93.0 ± 0%      91.0 ± 0%   -2.15%  (p=0.000 n=50+50)
TargetScraperGzip/metrics=100000-8       111 ± 0%       135 ± 1%  +21.89%  (p=0.000 n=40+50)

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-22 17:08:15 +00:00
Łukasz Mierzwa 92e381b8a3 Add a scrape benchmark with gzipped responses 
Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
 2024-02-22 17:07:22 +00:00
Julien Pivotto e22b564049 Always align scrapes even if tolerance is bigger than 1% of scrape interval 
When the scrape tolerance is bigger than 1% of the scrape interval, take
1% of the scrape interval as the tolerance instead of not aligning the
scrape at all.

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
 2024-02-21 15:09:21 +01:00
Owen Williams a28d7865ad UTF-8: Add support for parsing UTF8 metric and label names 
This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats.
This grammar will also be valid for non-UTF-8 names.
UTF-8 names will not be considered valid unless model.NameValidationScheme is changed.

This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/.

Part of https://github.com/prometheus/prometheus/issues/13095

Signed-off-by: Owen Williams <owen.williams@grafana.com>
 2024-02-15 14:34:37 -05:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places 
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-08 19:32:33 +00:00
Bryan Boreham d0dee51aac scrape tests: check NaN values directly 
Normally, a NaN value is never equal to any other value. Compare sample
values via `Float64bits` so that NaN values which are exactly the same
will compare equal.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-08 19:30:20 +00:00
Bryan Boreham 39af788dbd Tests: use replacement DeepEquals using go-cmp 
Use DeepEqual replacement using go-cmp, which is more flexible.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2024-02-08 19:30:20 +00:00
Paweł Szulik b0c538787d Refactor scrape tests to use testify. 
Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>
 2024-02-01 13:51:31 +00:00
Bryan Boreham 4ad9b6df2e
Merge pull request #13336 from machine424/flakky 
scrape_test.go: Increase scrape interval in TestScrapeLoopCache
to reduce potential flakiness.
 2024-01-18 14:12:55 +00:00
Ziqi Zhao df2a0ecf3b
Native Histograms: support native_histogram_min_bucket_factor in scrape_config (#13222) 
Native Histograms: support native_histogram_min_bucket_factor in scrape_config

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
 2024-01-17 16:58:54 +01:00
machine424 2f60177203
scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness 
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
 2023-12-27 19:25:12 +01:00
Julien Pivotto 0763ec841b
Merge pull request #13313 from kalpadiptyaroy/fix-quality-value-accept-header 
bug: Fix quality value in accept header
 2023-12-21 11:40:30 +01:00
Kumar Kalpadiptya Roy b012366c33 Issue #13268: fix quality value in accept header 
Signed-off-by: Kumar Kalpadiptya Roy <kalpadiptya.roy@outlook.com>
 2023-12-21 10:33:05 +05:30