845 commits

Author	SHA1	Message	Date
beorn7	a9243d4d2c	docs: Improve, clarify, and fix documentation on scrape limits ... In detail: - Clarify that label name and value length limits are in byte, not in UTF-8 data points. - More consistent formatting to keep 80 characters line limet. - Clarify various misleading specifications around "per sample", "per scrape", "per scrape config", "per job"... - Fix grammar. Signed-off-by: beorn7 <beorn@grafana.com>	2024-09-24 14:55:54 +02:00
Björn Rabenstein	df9916ef66	Merge pull request #14677 from prometheus/beorn7/histogram ... promql(native histograms): Introduce exponential interpolation	2024-09-19 18:08:59 +02:00
Jan Fajerski	96e5a94d29	promql: rename holt_winters to double_exponential_smoothing ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-19 15:29:01 +02:00
beorn7	6fcd225aee	promql(native histograms): Introduce exponential interpolation ... The linear interpolation (assuming that observations are uniformly distributed within a bucket) is a solid and simple assumption in lack of any other information. However, the exponential bucketing used by standard schemas of native histograms has been chosen to cover the whole range of observations in a way that bucket populations are spread out over buckets in a reasonably way for typical distributions encountered in real-world scenarios. This is the origin of the idea implemented here: If we divide a given bucket into two (or more) smaller exponential buckets, we "most naturally" expect that the samples in the original buckets will split among those smaller buckets in a more or less uniform fashion. With this assumption, we end up with an "exponential interpolation", which therefore appears to be a better match for histograms with exponential bucketing. This commit leaves the linear interpolation in place for NHCB, but changes the interpolation for exponential native histograms to exponential. This affects `histogram_quantile` and `histogram_fraction` (because the latter is more or less the inverse of the former). The zero bucket has to be treated specially because the assumption above would lead to an "interpolation to zero" (the bucket density approaches infinity around zero, and with the postulated uniform usage of buckets, we would end up with an estimate of zero for all quantiles ending up in the zero bucket). We simply fall back to linear interpolation within the zero bucket. At the same time, this commit makes the call to stick with the assumption that the zero bucket only contains positive observations for native histograms without negative buckets (and vice versa). (This is an assumption relevant for interpolation. It is a mostly academic point, as the zero bucket is supposed to be very small anyway. However, in cases where it _is_ relevantly broad, the assumption helps a lot in practice.) This commit also updates and completes the documentation to match both details about interpolation. As a more high level note: The approach here attempts to strike a balance between a more simplistic approach without any assumption, and a more involved approach with more sophisticated assumptions. I will shortly describe both for reference: The "zero assumption" approach would be to not interpolate at all, but _always_ return the harmonic mean of the bucket boundaries of the bucket the quantile ends up in. This has the advantage of minimizing the maximum possible relative error of the quantile estimation. (Depending on the exact definition of the relative error of an estimation, there is also an argument to return the arithmetic mean of the bucket boundaries.) While limiting the maximum possible relative error is a good property, this approach would throw away the information if a quantile is closer to the upper or lower end of the population within a bucket. This can be valuable trending information in a dashboard. With any kind of interpolation, the maximum possible error of a quantile estimation increases to the full width of a bucket (i.e. it more than doubles for the harmonic mean approach, and precisely doubles for the arithmetic mean approach). However, in return the _expectation value_ of the error decreases. The increase of the theoretical maximum only has practical relevance for pathologic distributions. For example, if there are thousand observations within a bucket, they could _all_ be at the upper bound of the bucket. If the quantile calculation picks the 1st observation in the bucket as the relevant one, an interpolation will yield a value close to the lower bucket boundary, while the true quantile value is close to the upper boundary. The "fancy interpolation" approach would be one that analyses the _actual_ distribution of samples in the histogram. A lot of statistics could be applied based on the information we have available in the histogram. This would include the population of neighboring (or even all) buckets in the histogram. In general, the resolution of a native histogram should be quite high, and therefore, those "fancy" approaches would increase the computational cost quite a bit with very little practical benefits (i.e. just tiny corrections of the estimated quantile value). The results are also much harder to reason with. Signed-off-by: beorn7 <beorn@grafana.com>	2024-09-19 14:19:10 +02:00
Jan Fajerski	15cea39136	promql: put holt_winters behind experimental feature flag ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-18 15:39:58 +02:00
Nihal	d0d57ff642	[DOC] Add username_file in basic auth http client config (#12749) ... Signed-off-by: Wasim Nihal <sswasim64@gmail.com>	2024-09-17 12:20:31 +01:00
Augustin Husson	6febfbb3be	put back feature flag 'delayed-compaction' and 'old-ui' (#14909) ... Signed-off-by: Augustin Husson <husson.augustin@gmail.com>	2024-09-16 15:46:28 +02:00
Jan Fajerski	722f33225c	docs/feature_flags.md: drop agent feature flag ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-14 14:03:59 +02:00
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868) ... * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2024-09-10 22:32:03 +02:00
Jan Fajerski	fa318711f4	Merge branch 'main' into 3.0-main-sync-24-09-09 ... Conflicts: cmd/prometheus/main.go docs/command-line/prometheus.md docs/feature_flags.md web/ui/build_ui.sh web/web.go Resolved by dropping the UTF-8 feature flag and adding the `auto-reload-config` feature flag. For the new web ui pick all changes from `main`.	2024-09-09 15:44:22 +02:00
Julius Volz	f1e0939f2f	Document /api/v1/parse_expr endpoint ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2024-09-09 10:27:24 +02:00
Julius Volz	ad21ef3d76	Merge branch 'main' into mantine-ui ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2024-09-06 21:26:00 +02:00
Björn Rabenstein	694d98032b	Merge pull request #14705 from prometheus/owilliams/default-on ... utf8: enable utf-8 support by default	2024-09-06 17:24:37 +02:00
Bartlomiej Plotka	edc83ed164	Update storage.md with the latest around Remote Write 2.0 (#14848) ... * Update storage.md with the latest around Remote Write 2.0 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update storage.md Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	2024-09-06 13:50:10 +01:00
Owen Williams	88bb05c3e8	utf8: enable utf-8 support by default ... This change causes Prometheus to allow all UTF-8 characters in metric and label names. This means that names that were previously invalid and would have been previously rejected will be allowed through. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-09-06 08:48:11 -04:00
Julien	404e577034	Merge pull request #14734 from roidelapluie/scrape_failure_logger ... Add support for logging scrape failures to a specified file	2024-09-06 11:16:50 +02:00
machine424	cc40b65ab4	fix(promtool): use the final database path for --sandbox-dir-root instead of the default value as it may be overridden ... add a regression test for that. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-05 15:25:32 +02:00
Julien	be3597c272	Update autoreload documentation: recommend writing files atomically (#14836) ... Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-09-05 12:50:20 +01:00
machine424	d18fa62ae9	chore(discovery): enable new-service-discovery-manager by default and drop legacymanager package ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-05 12:46:03 +02:00
Julien	ce0f09b125	Scrape: Add scrape_failure_log_file to log Scrape Failures ... Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-09-05 11:01:40 +02:00
Julien	50f5327f83	Merge pull request #14769 from roidelapluie/autoreload ... Support reload config automatically	2024-09-05 10:47:07 +02:00
Julius Volz	c73c3e24d7	Default to serving new (Prometheus 3.0) UI ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2024-09-04 21:33:37 +02:00
Julius Volz	490eef6509	Merge branch 'main' into mantine-ui ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2024-09-04 20:58:19 +02:00
Jan Fajerski	2f6fafd122	Merge branch 'release-3.0' into 3.0-main-sync-24-08-30 ... Conflicts: docs/command-line/prometheus.md regenerate via make cli-documentation	2024-09-04 20:08:31 +02:00
Arthur Silva Sens	ca1f30bb6e	Promote Agent mode to it's own cmdline flag ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-09-04 20:06:35 +02:00
Jan Fajerski	befcfadf78	Fix merge conflicts ... Fix call to newTestEngine(t) in promql/engine_test.go:3214. `agent` feature-flag it's own cmdline flag now. Remove `scrape.name-escaping-scheme` argument. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-04 20:06:17 +02:00
Jan Fajerski	fe4289b502	Merge branch 'main' into HEAD	2024-09-04 18:50:00 +02:00
Julien	deabb30b20	Document custom HTTP headers and add tests ... Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-09-03 13:31:08 +02:00
Julien	93f27409cd	Merge pull request #14660 from roidelapluie/customheaders ... Add documentation for custom headers.	2024-09-03 10:19:01 +02:00
Jan Fajerski	00315ce15e	Merge branch 'main' into 3.0-main-sync-24-08-30 ... using -Xours Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-02 11:27:18 +02:00
Julien	1cd2d0498b	Support reload config automatically ... Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-08-30 17:12:44 +02:00
Arthur Silva Sens	60b9e9fc16	Promote Agent mode to it's own cmdline flag ... Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	2024-08-29 14:45:09 -03:00
Kevin Rawal	b0677263c3	Update basics.md ... fix broken URL for feature flags native histograms in querying prometheus documentation Signed-off-by: Kevin Rawal <84058124+kevinrawal@users.noreply.github.com>	2024-08-29 23:06:31 +05:30
Jorge Creixell	e9e3d64b7c	PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477) ... PromQL engine: Delay deletion of __name__ label to the end of the query evaluation - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error. - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation. - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If `__name__` is used as target label, it won't be dropped at the end of the query evaluation. - Fixes https://github.com/prometheus/prometheus/issues/11397 - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`). - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags. - Note: a feature flag `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792) Example (this always fails, as `__name__` is being dropped by `count_over_time`): ``` count_over_time({__name__!=""}[1m]) => Error executing query: vector cannot contain metrics with the same labelset ``` Before: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => Error executing query: vector cannot contain metrics with the same labelset ``` After: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 ... ``` Signed-off-by: Jorge Creixell <jcreixell@gmail.com> --------- Signed-off-by: Jorge Creixell <jcreixell@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in>	2024-08-29 15:50:39 +02:00
Julien Pivotto	ed2e18d7de	Add documentation for custom headers. ... Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-08-29 10:04:45 +02:00
Suraj Patil	7757794bb3	[ENHANCEMENT] Promtool: Adding labels to time series while creating tsdb blocks (#14403) ... * feat: #14402 - Adding labels to time series while creating tsdb blocks Signed-off-by: Suraj Patil <patilsuraj767@gmail.com>	2024-08-28 12:12:24 +10:00
Bryan Boreham	4202be5e79	Merge branch 'release-2.54' into merge-2.54.1-into-main	2024-08-27 12:04:48 +01:00
Julien	349068ad3e	Merge pull request #14665 from roidelapluie/multiple-listening-addresses ... Add support for multiple listening addresses	2024-08-26 10:31:23 +02:00
Björn Rabenstein	0a00d21cff	Merge pull request #14673 from machine424/docdoccc ... docs(configuration.md): clarify the explanations about some temp labe…	2024-08-22 12:31:46 +02:00
Björn Rabenstein	7fad1ec8ee	Merge pull request #14655 from suntala/suntala/sort-by-label-enhancement ... promql: Fall back to full label sets when sorting by label	2024-08-21 12:28:55 +02:00
Jan Fajerski	5138922b0d	Merge branch 'main' into 3.0-main-sync-24-08-21	2024-08-21 09:09:36 +02:00
Björn Rabenstein	60366a8a05	Merge pull request #14526 from pawarpranav83/release-3.0 ... remove deprecated and replaced remote-write-receiver flag from enable-feature	2024-08-20 19:30:46 +02:00
machine424	88cac6fb49	docs(configuration.md): clarify the explanations about some temp labels and format for better visibility. ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-08-20 15:44:39 +02:00
Julien Pivotto	9b5e7623f4	Add support for multiple listening addresses ... Fixes #9105 Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-08-20 13:09:16 +02:00
ouyang1204@gmail.com	89dee48cc8	fix the issue of failing to match the first network when the container is reconnected to a new network ... Signed-off-by: ouyang1204@gmail.com <ouyang1204@gmail.com>	2024-08-19 21:26:25 +08:00
Ganesh Vernekar	e86e4ed87f	Merge pull request #14490 from ywwg/owilliams/utf8-02-upstream ... feat(utf8): utf-8 content negotiation and flags	2024-08-16 17:23:42 -04:00
Owen Williams	9e7308de38	feat(utf8): utf8 content negotation and flags ... Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-08-16 16:41:27 -04:00
Julien Pivotto	7711cd5ab5	Remove deprecated storage.tsdb.retention flag ... Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien <roidelapluie@o11y.eu>	2024-08-16 13:53:09 +02:00
suntala	ce4eac859a	Link to specific feature flag entry ... Signed-off-by: suntala <arati.rana@grafana.com>	2024-08-14 13:22:12 +02:00
Björn Rabenstein	3f16a2e7de	Merge pull request #14543 from jan--f/3.0-main-sync-24-08-01 ... 3.0 main sync 24 08 01	2024-08-13 15:54:13 +02:00