337 commits

Author	SHA1	Message	Date
George Krajcsovits	1b4e7f74e6	feat(tools): add debug printouts to rules unit testing (#15196) ... * promtool: Add debug flag for rule tests This makes it print out the tsdb state (both input_series and rules that are run) at the end of a test, making reasoning about tests much easier. Signed-off-by: David Leadbeater <dgl@dgl.cx> * Reuse generated test name from junit testing Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: David Leadbeater <dgl@dgl.cx> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: David Leadbeater <dgl@dgl.cx>	2024-10-22 15:24:36 +02:00
Bryan Boreham	70e2d23027	Merge pull request #11474 from clwluvw/group-label ... [FEATURE] rules: add labels at group level	2024-10-21 14:47:12 +01:00
Alan Protasio	c78d5b94af	Disallowing configure AM with the v1 api (#13883) ... * Stop supporting Alertmanager v1 * Disallowing configure AM with the v1 api Signed-off-by: alanprot <alanprot@gmail.com> * Update config/config_test.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update config/config.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Addressing coments Signed-off-by: alanprot <alanprot@gmail.com> * Update notifier/notifier.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update config/config_test.go Co-authored-by: Jan Fajerski <jan--f@users.noreply.github.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Co-authored-by: Jan Fajerski <jan--f@users.noreply.github.com>	2024-10-18 15:23:14 +02:00
Yi	2cabd1b707	config: remove expand-external-labels flag in release 3.0 (#14657) ... remove expand-external-labels feature flag and enabled env arg expansion for external labels by default. Signed-off-by: jyz0309 <45495947@qq.com>	2024-10-17 10:25:05 +02:00
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log ... For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-10-07 15:58:50 -04:00
Matthieu MOREL	ab64966e9d	fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094) ... * fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-10-06 16:35:29 +00:00
Alex Johnson	be0f10054e	Remove no-default-scrape-port featureFlag ... Signed-off-by: Alex Johnson <alex.kattathra.johnson@gmail.com>	2024-09-25 10:13:19 -05:00
Björn Rabenstein	5b9148e552	Merge pull request #14820 from charleskorn/promqltest-native-histogram-format ... promqltest: use test expression format for histograms in assertion failure messages and include reset hint in the test expression	2024-09-20 16:47:08 +02:00
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868) ... * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2024-09-10 22:32:03 +02:00
Jan Fajerski	fa318711f4	Merge branch 'main' into 3.0-main-sync-24-09-09 ... Conflicts: cmd/prometheus/main.go docs/command-line/prometheus.md docs/feature_flags.md web/ui/build_ui.sh web/web.go Resolved by dropping the UTF-8 feature flag and adding the `auto-reload-config` feature flag. For the new web ui pick all changes from `main`.	2024-09-09 15:44:22 +02:00
Owen Williams	88bb05c3e8	utf8: enable utf-8 support by default ... This change causes Prometheus to allow all UTF-8 characters in metric and label names. This means that names that were previously invalid and would have been previously rejected will be allowed through. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2024-09-06 08:48:11 -04:00
machine424	cc40b65ab4	fix(promtool): use the final database path for --sandbox-dir-root instead of the default value as it may be overridden ... add a regression test for that. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-09-05 15:25:32 +02:00
Charles Korn	4da551578c	Fix test broken by inclusion of counter_reset_hint ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2024-09-04 16:33:18 +10:00
Jan Fajerski	00315ce15e	Merge branch 'main' into 3.0-main-sync-24-08-30 ... using -Xours Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-09-02 11:27:18 +02:00
Suraj Patil	7757794bb3	[ENHANCEMENT] Promtool: Adding labels to time series while creating tsdb blocks (#14403) ... * feat: #14402 - Adding labels to time series while creating tsdb blocks Signed-off-by: Suraj Patil <patilsuraj767@gmail.com>	2024-08-28 12:12:24 +10:00
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments ... Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	2024-08-22 17:36:11 +02:00
Jan Fajerski	5138922b0d	Merge branch 'main' into 3.0-main-sync-24-08-21	2024-08-21 09:09:36 +02:00
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-08-18 12:13:25 +02:00
Bryan Boreham	2936ab80d7	[Tests] Promtool: Sort output where Prometheus does not guarantee the order. ... Previously this was working because iout-of-order chunks forced a sort and merge. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-08-14 11:19:02 +01:00
Jan Fajerski	8a96a75ba8	Merge branch 'main' into 3.0-main-sync-24-08-01 ... Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2024-08-01 15:44:47 +02:00
Kushal shukla	fe12924638	promtool: JUnit-Format XML Test Results (#14506) ... * Junit compatible output Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>	2024-07-29 21:28:08 +10:00
Seena Fallah	f253d36361	rule: allow merging labels from group level ... Support merging labels from groups to rule labels Signed-off-by: Seena Fallah <seenafallah@gmail.com>	2024-07-26 20:18:05 +02:00
Bartlomiej Plotka	a60e5ce362	[PRW 2.0] Added Sender and RW Handler support for Response Stats. (#14444) ... * [PRW 2.0] Added Sender support for Response Stats. Chained on top of https://github.com/prometheus/prometheus/pull/14427 Fixes https://github.com/prometheus/prometheus/issues/14359 Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * move write stats to it's own file Signed-off-by: Callum Styan <callumstyan@gmail.com> * Clean up header usage Signed-off-by: Callum Styan <callumstyan@gmail.com> * add missing license to new stats file Signed-off-by: Callum Styan <callumstyan@gmail.com> * Addressed all comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2024-07-19 18:53:40 +01:00
Kartikay	978fce7d3b	flags: remove "promql-at-modifier" and "promql-negative-offset" feature flags ... --------- Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in> Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>	2024-06-25 16:01:00 +02:00
Zhang Zhanpeng	debbdb8608	make matrix selection and lookback left-open and right-closed ... Signed-off-by: Zhang Zhanpeng <zhangzhanpeng.zzp@alibaba-inc.com> Signed-off-by: beorn7 <beorn@grafana.com> Co-authored-by: beorn7 <beorn@grafana.com>	2024-06-20 22:05:40 +08:00
Ben Ye	5a218708f1	tsdb: Extend compactor interface to allow compactions to create multiple output blocks (#14143) ... * add hook to allow head compaction to create multiple output blocks Signed-off-by: Ben Ye <benye@amazon.com> * change Compact interface; remove BlockPopulator changes Signed-off-by: Ben Ye <benye@amazon.com> * rebase main Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> * address feedbacks; add unit test Signed-off-by: Ben Ye <benye@amazon.com> * Apply suggestions from code review Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Update tsdb/compact_test.go Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	2024-06-12 17:31:25 -04:00
beorn7	3127a4029e	doc: Clarify the limits of dumping/backfilling via OpenMetrics ... This is about native histograms (not yet supported) and staleness markers (for which OpenMetrics support isn't even planned). Signed-off-by: beorn7 <beorn@grafana.com>	2024-05-21 14:50:06 +02:00
Björn Rabenstein	3119b8a055	Merge pull request #13218 from machine424/ro-promtool ... Make DBReadOnly more RO	2024-05-21 13:27:40 +02:00
Oleksandr Redko	f10c3454e9	Enable perfsprint linter and fix up code ... Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>	2024-05-15 17:51:05 +03:00
Bryan Boreham	8fd96241ab	test: add promqltest package references ... To packages outside of promql. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-05-08 16:08:04 +01:00
Will Hegedus	bd1878700b	promtool: Fix panic on extended tsdb analyze (#13976) ... Currently, running promtool tsdb analyze with the --extended flag will cause an 'index out of range' error if running it against a block that does not have any native histogram chunks. This change ensures that promtool won't try to display data that doesn't exist. Signed-off-by: Will Hegedus <whegedus@linode.com>	2024-04-24 11:35:34 +10:00
machine424	c5a1cc9148	chore(tsdb): add a sandboxDir to DBReadOnly, the directory can be used for transient file writes. ... use it in loadDataAsQueryable to make sure the RO Head doesn't truncate or cut new chunks in data/chunks_head/. add a -sandbox-dir-root flag to "promtool tsdb dump/dump-openmetrics" to control the root of that sandbox dirrectory. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-04-15 17:00:25 +02:00
Matthieu MOREL	6f595c6762	golangci-lint: enable whitespace linter (#13905) ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-04-11 09:27:54 +01:00
Matthieu MOREL	d496687c8e	golangci-lint: enable usestdlibvars linter ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-04-08 19:26:23 +00:00
komisan19	0249e080b4	refactor: utilize standard functions max/min ... Signed-off-by: komisan19 <18901496+komisan19@users.noreply.github.com>	2024-04-04 03:15:38 +09:00
Artur Melanchyk	44dcf02c69	TSDB: make total lock-free by using atomic ... Signed-off-by: Artur Melanchyk <artur.melanchyk@gmail.com>	2024-03-23 19:51:29 +01:00
Bryan Boreham	773170f372	Merge pull request #13822 from dgl/promtool-test-errors ... promtool: Avoid using testify for user rule tests	2024-03-23 09:42:34 +01:00
David Leadbeater	7ec4a11472	promtool: Avoid using testify for user rule tests ... Using testify outside of unit tests results in panics rather than a useful error for the user. Fixes #13703 Signed-off-by: David Leadbeater <dgl@dgl.cx>	2024-03-21 22:08:10 +11:00
heyitao	c7ca85388f	Fix yaml file format and clear ci errors ... Signed-off-by: heyitao <heyitao@uniontech.com>	2024-03-21 11:32:02 +08:00
Bryan Boreham	e79b9ed2ab	Merge pull request #13194 from machine424/open ... promtool: add a "tsdb dump-openmetrics" to dump in OpemMetrics format.	2024-02-28 17:46:59 +00:00
machine424	4b71f6ffc2	promtool: add a "tsdb dump-openmetrics" to dump in OpemMetrics format. ... This closes the loop, as the output can be fed into "tsdb create-blocks-from openmetrics" Native histograms are not supported. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-02-28 15:34:18 +01:00
machine424	f477e0539a	Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 ... Prevent adding back golang.org/x/exp/slices. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2024-02-28 14:54:53 +01:00
Bryan Boreham	eff3a13e19	model/textparse: parsers take a labels SymbolTable ... This allows strings to be interned to save memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-26 11:45:25 +00:00
Łukasz Mierzwa	5597020a60	Use github.com/klauspost/compress for gzip and zlib ... klauspost/compress is a high quality drop-in replacement for common Go compression libraries. Since Prometheus sends out a lot of HTTP requests that often return compressed output having improved compression libraries helps to save cpu & memory resources. On a test Prometheus server I was able to see cpu reduction from 31 to 30 cores. Benchmark results: name old time/op new time/op delta TargetScraperGzip/metrics=1-8 69.4µs ± 4% 69.2µs ± 3% ~ (p=0.122 n=50+50) TargetScraperGzip/metrics=100-8 84.3µs ± 2% 80.9µs ± 2% -4.02% (p=0.000 n=48+46) TargetScraperGzip/metrics=1000-8 296µs ± 1% 274µs ±14% -7.35% (p=0.000 n=47+45) TargetScraperGzip/metrics=10000-8 2.06ms ± 1% 1.66ms ± 2% -19.34% (p=0.000 n=47+45) TargetScraperGzip/metrics=100000-8 20.9ms ± 2% 17.5ms ± 3% -16.50% (p=0.000 n=49+50) name old alloc/op new alloc/op delta TargetScraperGzip/metrics=1-8 6.06kB ± 0% 6.07kB ± 0% +0.24% (p=0.000 n=48+48) TargetScraperGzip/metrics=100-8 7.04kB ± 0% 6.89kB ± 0% -2.17% (p=0.000 n=49+50) TargetScraperGzip/metrics=1000-8 9.02kB ± 0% 8.35kB ± 1% -7.49% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 18.1kB ± 1% 16.1kB ± 2% -10.87% (p=0.000 n=47+47) TargetScraperGzip/metrics=100000-8 1.21MB ± 0% 1.01MB ± 2% -16.69% (p=0.000 n=36+50) name old allocs/op new allocs/op delta TargetScraperGzip/metrics=1-8 71.0 ± 0% 72.0 ± 0% +1.41% (p=0.000 n=50+50) TargetScraperGzip/metrics=100-8 81.0 ± 0% 76.0 ± 0% -6.17% (p=0.000 n=50+50) TargetScraperGzip/metrics=1000-8 92.0 ± 0% 83.0 ± 0% -9.78% (p=0.000 n=50+50) TargetScraperGzip/metrics=10000-8 93.0 ± 0% 91.0 ± 0% -2.15% (p=0.000 n=50+50) TargetScraperGzip/metrics=100000-8 111 ± 0% 135 ± 1% +21.89% (p=0.000 n=40+50) Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2024-02-22 17:08:15 +00:00
Bryan Boreham	5a6c8f9c15	promtool: use go-cmp instead of DeepEqual ... go-cmp allows more control over unexported fields and implementation details. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-08 19:30:20 +00:00
Bryan Boreham	39af788dbd	Tests: use replacement DeepEquals using go-cmp ... Use DeepEqual replacement using go-cmp, which is more flexible. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-02-08 19:30:20 +00:00
Paweł Szulik	d5eb636a89	Refactor cmd tests to use testify. ... Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>	2024-02-01 13:51:31 +00:00
Filip Petkovski	583f3e587c	Optimize histogram iterators (#13340) ... Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2024-01-23 17:02:14 +01:00
Paulin Todev	78411d5e8b	SD Managers taking over responsibility for registration of debug metrics (#13375) ... SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2024-01-23 16:53:55 +01:00
Rewanth Tammana	102fd8cc88	Enhanced visibility for promtool test rules with JSON colored formatting (#13342) ... * Added diff flag for unit test to improvise readability & debugging Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Removed blank spaces Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed linting error Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Added cli flags to documentation Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Revert unrrelated linting fixes Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Fixed review suggestions Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Cleanup Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> * Updated flag description Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com> --------- Signed-off-by: Rewanth Tammana <22347290+rewanthtammana@users.noreply.github.com>	2024-01-18 09:49:16 -05:00