Commit graph

9687 commits

Author SHA1 Message Date
Mateusz Gozdek 83086aee00 tsdb/agent: use unique registry per tests
So tests can run in parallel.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
Mateusz Gozdek c08bb86be0 cmd/prometheus: use random listen port in TestStartupInterrupt test
So it can be run in parallel safely.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
Mateusz Gozdek 7bd7573891 cmd/prometheus/main_unix_test.go: fix unix test styling
* Formatting of error message is missing a space after ':'.
* t.Fatalf should be used instead of  t.Errorf+return.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
Mateusz Gozdek ed698c5480 .golangci.yml: sort output by default
So running linter several times will yell the same order of reported
issues, which makes it much easier to address them.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
Mateusz Gozdek e235b499b0 Makefile: consistently re-use UI_PATH variable
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
Mateusz Gozdek 2f312ff4c5 tsdb: mark TestTombstoneCleanRetentionLimitsRace test as slow
It takes over 100 seconds to execute this test, so I'd consider it as
slow.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-11 01:37:24 +01:00
David Leadbeater c0c5f3227b Update to return a float
Signed-off-by: David Leadbeater <dgl@dgl.cx>
2021-11-10 18:35:30 +01:00
David Leadbeater 89ebb3dcf2 Add parseDuration to template functions
This can be useful when generating rules, a query may use a duration,
and it may be useful to template that into a URL parameter. Therefore
this allows interfacing with systems that don't implement Prometheus
style duration parsing.

Signed-off-by: David Leadbeater <dgl@dgl.cx>
2021-11-10 18:35:30 +01:00
Augustin Husson 69b761a187
fix the different url used in the codemirror-promql readme (#9713)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2021-11-10 15:54:58 +01:00
Thomas Jackson f0003bc0ba
Don't drop ParenExpr when creating StepInvariantExpr (#9591)
* Add test case to showcase the problem in #9590

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

* Don't unwrap ParenExpr in newStepInvariantExpr

Fixes #9590

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2021-11-10 20:16:24 +05:30
Levi Harrison 7400e07fa9
Close DB in Agent tests (#9630)
* Close agent db in tests

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Close first DB before opening second

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Use seperate variables for different DBs?

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Close remote storage

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Fix closing of stuff

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Remove the build flags after a rebase

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix closing of stuff 2

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@inuits.eu>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-11-10 20:05:54 +05:30
Robert Fratto 4a83e6f453
Remove agent mode warnings when loading configs (#9622)
PR #9618 introduced failing to load the config file when agent mode is
configured to run with unspported settings. This made the block that
logs a warning on their configuration no-op, which is now removed.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
2021-11-10 19:39:30 +05:30
Jacques-Etienne Beaudet 9fae120f2a Support int and uint as datatype for template formatting
Fixes GH-9679

Signed-off-by: Jacques-Etienne Beaudet <jebeaudet@coveo.com>
2021-11-10 10:48:53 +01:00
Mateusz Gozdek 0bfef847b0 discovery/consul: fix leaking goroutine from test
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek d8561dbfd8 storage/remote: make tests use separate remote write configs
So tests can be run in parallel without races.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek 116552cc58 storage/remote: check errors from ApplyConfig in tests
So tests do not produce obscure errors when applying configuration
fails.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek 4ef8c7c1d8 discovery/xds: make each test use separate HTTP resource config
So tests can run in parallel without races.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek fedcb8999b discovery/marathon: make test use separate instances of config
So tests can be executed in parallel without producing races.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek 2cf7479622 web: prepare TestFederation_NotReady test for parallel execution
Later on in the test we override fields of handler, so we need a handler
per sub-test, rather than a global one.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek b2c933e317 web: randomize used listening ports for tests
So they can be parallelized to speed up tests execution.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek fa1b14e146 cmd/prometheus: randomize test port and isolate test data directory
Between the tests. This enables parallelizing those tests, which should
cut the test execution time.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Mateusz Gozdek 5e002dec68 util/testutil: add RandomUnprivilegedPort function
So we can randomize ports used for listening in tests, so they can run
in parallel.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-10 09:40:43 +01:00
Bryan Boreham 1ed94142fc
remote-write: slow down retries to avoid DDOS (#9634)
* remote-write: slow down retries to avoid DDOS

Increase the default max retry time from 100ms to 5 seconds.

Remote write calls are retried after a recoverable error such as the
back-end returning 500. Prometheus waits the minimum time and retries,
then doubles the wait on each subsequent retry until the maximum is
reached.

If some data is still getting through, remote-write will also increase
shards, and the default maximum is 200. 200 shards sending every 100ms
is 20 calls per second, to a back-end that is already in trouble.

5 seconds was chosen to match the default BatchSendDeadline: if we can
afford to wait that long for no response, then we can wait the same time
to retry. We will reach 5 seconds after 9 successive failures.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Update config doc for max_backoff change

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-11-09 14:08:24 -08:00
Mateusz Gozdek f4650c27e7 tsdb/wal: fix flaky TestReaderFuzz* tests
It seems sometimes you can get error like:

                Error:          Not equal:
                                expected: []byte(nil)
                                actual  : []byte{}

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,2 +1,3 @@
                                -([]uint8) <nil>
                                +([]uint8) {
                                +}

This commit does what bytes.Equal does to silence those differences. I'm
not sure if this is a correct solution or just covering up the actual bug.

Closes #9574

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-09 14:32:20 +01:00
Björn Rabenstein 4c56a193c5
Merge pull request #9478 from prometheus/beorn7/pkg-deprecation
Move packages out of deprecated pkg directory
2021-11-09 11:09:16 +01:00
曹明 a0d31c28fc tsdb: Add windows arm64 support.
Signed-off-by: 曹明 <caoming1@kingsoft.com>
2021-11-09 11:07:27 +01:00
Mateusz Gozdek b319b14431
tsdb/chunks: preallocate at least some space on non-Windows systems (#9581)
To avoid potential chunk corruption read, which I am not sure why is
happening.

Closes #9561.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-09 13:47:00 +05:30
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Niko Smeds fdcd423dfe Increase time range for PrometheusHAGroupCrashlooping alert
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
2021-11-08 15:06:42 -08:00
Julien Pivotto c564984daa
Reduce Prometheus pull requests builds (#9666)
* Reduce Prometheus builds

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-09 00:02:23 +01:00
Torbjörn Lönnemark 5e06527190 Fix incorrect PR reference in 2.25.0 changelog
Signed-off-by: Torbjörn Lönnemark <tobbez@ryara.net>
2021-11-08 23:25:45 +01:00
Julien Pivotto 9d1c1cd551 Add Robert as tsdb/agent maintainer
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-08 22:32:39 +01:00
beorn7 a1e595edac Fix two trivial lint warnings
Not sure why those show up for me locally but not if run by the CI.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-08 22:32:13 +01:00
Bryan Boreham 26d8ae0e41 Rules: simplify map key for stale series detection
The rules manager keeps a note of which series were generated by the
last run, so it can write a stale marker to those that disappeared.
Since the keys are not for human eyes, we can use a simpler format
and save the effort of quoting label values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-11-08 22:18:48 +01:00
Matthew 628211c25a
Feat UI metrics search (#9629)
* feat: add search to metrics explorer

Signed-off-by: mtfoley <mtfoley.mae@gmail.com>

* fix: ui-lint and ui-build errors

Signed-off-by: mtfoley <mtfoley.mae@gmail.com>

* feat: use @nexucis/fuzzy

Signed-off-by: mtfoley <mtfoley.mae@gmail.com>

* chore: code style and delete commented test

Signed-off-by: mtfoley <mtfoley.mae@gmail.com>

* rename Props to MetricsExplorerProps

Signed-off-by: mtfoley <mtfoley.mae@gmail.com>
2021-11-08 19:11:39 +01:00
Augustin Husson 4caae4e4a6
add a negative boost for some trigonometric functions that can overlapp other regular promQL functions (#9688)
* add a negative boost for some trigonometric functions that can overlapp other regular promQL functions

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* add comments to explain the purpose of the attribute boost

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2021-11-08 14:32:38 +01:00
Conor Evans c28b9a0574
Add datacenter to Consul service discovery logs (#9668)
* add datacenter to consul service discovery logs

Signed-off-by: Conor Evans <coevans@tcd.ie>
2021-11-08 09:34:21 +01:00
Dieter Plaetinck cda025b5b5
TSDB: demistify SeriesRefs and ChunkRefs (#9536)
* TSDB: demistify seriesRefs and ChunkRefs

The TSDB package contains many types of series and chunk references,
all shrouded in uint types.  Often the same uint value may
actually mean one of different types, in non-obvious ways.

This PR aims to clarify the code and help navigating to relevant docs,
usage, etc much quicker.

Concretely:

* Use appropriately named types and document their semantics and
  relations.
* Make multiplexing and demuxing of types explicit
  (on the boundaries between concrete implementations and generic
  interfaces).
* Casting between different types should be free.  None of the changes
  should have any impact on how the code runs.

TODO: Implement BlockSeriesRef where appropriate (for a future PR)

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* feedback

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* agent: demistify seriesRefs and ChunkRefs

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
2021-11-06 15:40:04 +05:30
johncming b882d2b7c7
tsdb/wal: Avoid writing closed channel. (#9566)
Signed-off-by: johncming <johncming@yahoo.com>
2021-11-06 15:11:06 +05:30
chenlujjj d18e42c650
refine comments of Checkpoint function (#9655)
Signed-off-by: chenlujjj <953546398@qq.com>
2021-11-06 15:09:16 +05:30
Bartlomiej Plotka 789274bf9c cmd: Fixed storage flag regression introduced in #9660
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-11-06 00:16:43 +01:00
Yijie Qin 6fce45838a
Add access function for restoration state of alerting rule (#9665) 2021-11-05 18:26:29 -04:00
Julien Pivotto b9c814fce6
Merge pull request #9681 from prometheus/release-2.31
merge back release 2.31
2021-11-05 21:21:04 +01:00
Julien Pivotto 411021ada9
Release 2.31.1 (#9669)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-05 21:13:21 +01:00
Marco Pracucci 309b094b92
Optimized MemPostings.EnsureOrder() (#9673)
* Optimizes MemPostings.EnsureOrder()

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Ignore linter warning

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-05 10:01:23 +00:00
Julien Pivotto 9621c2c0cc
Fix race with targets update during ApplyConfig (#9656)
I ended up extending the lock so refTargets remains valid for the
duration of the update.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-05 01:13:04 +01:00
Sunil Thaha 4bdaea7663
fix: storage.tsdb.path randomly initialised to data-agent/ (#9660)
Using the same variable for storage.tsdb.path and storage.agent.path
as below in main.go causes cfg.localStoragePath to be data/ or
data-agent/ at random.

  a.Flag("storage.tsdb.path", "Base path for metrics storage.").
      PreAction(serverOnlySetting()).
      Default("data/").StringVar(&cfg.localStoragePath)

  a.Flag("storage.agent.path", "Base path for metrics storage.").
      PreAction(agentOnlySetting()).
      Default("data-agent/").StringVar(&cfg.localStoragePath)
This patch fixes it by using a different variable for storage.agent.path

Signed-off-by: Sunil Thaha sthaha@redhat.com

Signed-off-by: Sunil Thaha <sthaha@redhat.com>
2021-11-04 10:08:01 +00:00
Bartlomiej Plotka e68ccc7708
Fix misleading agent-only/server-only check messages. (#9650)
* Fix misleading agent-only/server-only check messages.

Issue:

```
[root@host01 ~]# docker run -it --net=host --rm -v /root/editor/prom-agent-batcopter.yaml:/etc/prometheus/prometheus.yaml -v /root/prom-batcopter-data:/prometheus -u root --name prom-agent-batcopter quay.io/prometheus/prometheus:main --enable-feature=agent --config.file=/etc/prometheus/prometheus.yaml --storage.tsdb.path=/prometheus --web.listen-address=:9091
ts=2021-11-02T16:00:59.789Z caller=main.go:205 level=info msg="Experimental agent mode enabled."
The following flag(s) can not be used in agent mode: ["--enable-feature"]
```

Problem was that PreAction gives us all parsed flag. Context does not give us any info on what flag clause it was defined.

Also added info for flag help about being server or agent only.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* gofumpt.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-11-04 09:08:53 +00:00
Augustin Husson 17fc57948a
codemirror-promql moved to prometheus org (#9651)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2021-11-03 12:46:58 +01:00
Marco Pracucci 9f5ff5b269
Allow to disable trimming when querying TSDB (#9647)
* Allow to disable trimming when querying TSDB

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Added unit test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed TrimDisabled to DisableTrimming

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 15:38:34 +05:30