Commit graph

11115 commits

Author SHA1 Message Date
Marco Pracucci 309b094b92
Optimized MemPostings.EnsureOrder() (#9673)
* Optimizes MemPostings.EnsureOrder()

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Ignore linter warning

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-05 10:01:23 +00:00
Julien Pivotto 9621c2c0cc
Fix race with targets update during ApplyConfig (#9656)
I ended up extending the lock so refTargets remains valid for the
duration of the update.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-05 01:13:04 +01:00
Sunil Thaha 4bdaea7663
fix: storage.tsdb.path randomly initialised to data-agent/ (#9660)
Using the same variable for storage.tsdb.path and storage.agent.path
as below in main.go causes cfg.localStoragePath to be data/ or
data-agent/ at random.

  a.Flag("storage.tsdb.path", "Base path for metrics storage.").
      PreAction(serverOnlySetting()).
      Default("data/").StringVar(&cfg.localStoragePath)

  a.Flag("storage.agent.path", "Base path for metrics storage.").
      PreAction(agentOnlySetting()).
      Default("data-agent/").StringVar(&cfg.localStoragePath)
This patch fixes it by using a different variable for storage.agent.path

Signed-off-by: Sunil Thaha sthaha@redhat.com

Signed-off-by: Sunil Thaha <sthaha@redhat.com>
2021-11-04 10:08:01 +00:00
Bartlomiej Plotka e68ccc7708
Fix misleading agent-only/server-only check messages. (#9650)
* Fix misleading agent-only/server-only check messages.

Issue:

```
[root@host01 ~]# docker run -it --net=host --rm -v /root/editor/prom-agent-batcopter.yaml:/etc/prometheus/prometheus.yaml -v /root/prom-batcopter-data:/prometheus -u root --name prom-agent-batcopter quay.io/prometheus/prometheus:main --enable-feature=agent --config.file=/etc/prometheus/prometheus.yaml --storage.tsdb.path=/prometheus --web.listen-address=:9091
ts=2021-11-02T16:00:59.789Z caller=main.go:205 level=info msg="Experimental agent mode enabled."
The following flag(s) can not be used in agent mode: ["--enable-feature"]
```

Problem was that PreAction gives us all parsed flag. Context does not give us any info on what flag clause it was defined.

Also added info for flag help about being server or agent only.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* gofumpt.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-11-04 09:08:53 +00:00
Ganesh Vernekar c8b267efd6
Get histograms from TSDB to the rate() function implementation
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-11-03 19:04:18 +05:30
Augustin Husson 17fc57948a
codemirror-promql moved to prometheus org (#9651)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2021-11-03 12:46:58 +01:00
Marco Pracucci c04bab8758
Merge pull request #25 from grafana/upgrade-prometheus
Upgrade prometheus
2021-11-03 12:02:24 +01:00
Marco Pracucci 17d5a5b3df
Upgraded upstream Prometheus
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 11:23:16 +01:00
Marco Pracucci 9f5ff5b269
Allow to disable trimming when querying TSDB (#9647)
* Allow to disable trimming when querying TSDB

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Added unit test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed TrimDisabled to DisableTrimming

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 15:38:34 +05:30
Marco Pracucci 46514c6b12
Merge pull request #24 from grafana/upgrade-upstream
Upgrade upstream
2021-11-03 10:54:16 +01:00
Marco Pracucci 56f926c62c
Fixed linter issues
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 10:35:29 +01:00
Marco Pracucci 229575de0c
Upgrade upstream Prometheus
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 10:25:35 +01:00
sniper f82e56fbba
fix request bytes size and continue is useless (#9635)
Signed-off-by: kalmanzhao <kalmanzhao@tencent.com>

Co-authored-by: kalmanzhao <kalmanzhao@tencent.com>
2021-11-03 14:40:31 +05:30
Marco Pracucci edd05d7010
Add Head.AppendableMinValidTime() (#9643)
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-11-03 13:09:54 +05:30
Julien Pivotto b40e254f25
Agent: Add a boolean to the index to indicate agent mode. (#9649)
I would like to avoid extra API call's to determine if we are running in
Agent Mode, so I think we could use this approach.

This is a bootstrap of #9612

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-03 03:47:14 +00:00
Mateusz Gozdek ea924746b3
discovery/kubernetes: improve test logic for waiting for discoverers (#9584)
When running tests in parallel, 10 milliseconds may not be enough for
all discoverers to register, which will make test flaky.

This commit changes the waiting logic to wait for number of discoverers
to stop increasing during given time frame, which should be large enough
for single discoverer to register in test environment.

A following run passes with this commit:

go test -failfast -race -count 100 -v ./discovery/kubernetes/

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 22:17:32 +01:00
Mateusz Gozdek c3beca72e2 cmd/prometheus: wait for Prometheus to shutdown in tests
So temporary data directory can be successfully removed, as on Windows,
directory cannot be in used while removal.

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 20:14:19 +01:00
Mateusz Gozdek 01c5582216 .golangci.yml: enable gofumpt and goimports linters
For imports and more opinionated code formatting.

Closes #9557

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Mateusz Gozdek ce65883588 .golangci.yml: don't lint autogenerated files
So when we enable linters for formatting, they do not complain about
those files.

Refs #9557

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Mateusz Gozdek b7bdf6fab2 Fix imports formatting
According to
2829908806 (r58457095).

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Mateusz Gozdek 1a6c2283a3 Format Go source files using 'gofumpt -w -s -extra'
Part of #9557

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Julien Pivotto b1e8e8a0ca
Merge pull request #9642 from prometheus/release-2.31
Merge back release 2.31
2021-11-02 14:19:28 +01:00
Julien Pivotto 807f46a1ed
Gate agent behind a feature flag, valide mode flags (#9620)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-02 13:03:35 +00:00
Julien Pivotto 6e1d6edb33
Exclude agent from windows tests (#9645)
We are aware of the issue, but while we are working on it,
having main tests broken is an annoyance.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-02 13:58:51 +01:00
Julien Pivotto d4c83da6d2
Release 2.31 (#9639)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-11-02 11:07:15 +01:00
Björn Rabenstein b862218389
Merge pull request #9588 from darshanime/kahan
Use kahan summation for better numerical stability
2021-11-01 14:58:22 +01:00
Darshan Chaudhary a7e554b158
add check service-discovery command (#8970)
Signed-off-by: darshanime <deathbullet@gmail.com>
2021-11-01 14:42:12 +01:00
Hu Shuai 4b799c361a
Fix in typo in cmd/prometheus/main.go (#9632)
Signed-off-by: Hu Shuai <hus.fnst@cn.fujitsu.com>
2021-11-01 16:08:23 +05:30
chenlujjj 660329d5b3
add tombstoneFormatVersionSize & tombstonesCRCSize constants (#9625)
Signed-off-by: chenlujjj <953546398@qq.com>
2021-11-01 16:05:19 +05:30
Praveen Ghuge 64d9b41998
Use testing.T.TempDir() instead of ioutil.TempDir() in tsdb/wal unit tests (#9602)
Signed-off-by: Praveen Ghuge <praveen.ghuge@outlook.com>
2021-11-01 12:28:18 +05:30
darshanime 42d786f1ac use kahan summation for aggregation functions
Signed-off-by: darshanime <deathbullet@gmail.com>
2021-10-30 19:41:36 +05:30
darshanime 694b872dee address stylistic nits
Signed-off-by: darshanime <deathbullet@gmail.com>
2021-10-30 19:08:23 +05:30
darshanime a905354da3 use kahan for avg_over_time
Signed-off-by: darshanime <deathbullet@gmail.com>
2021-10-30 19:04:18 +05:30
darshanime 0a9deb9597 use kahan summation for numerical stability
Signed-off-by: darshanime <deathbullet@gmail.com>
2021-10-30 19:04:18 +05:30
Arthur Silva Sens be2599c853
config: Make remote-write required for Agent mode (#9618)
* config: Make remote-write required for Agent mode

Signed-off-by: ArthurSens <arthursens2005@gmail.com>
2021-10-30 01:41:40 +02:00
lzhfromustc 9da5382103
storage/remote: Prevent two goroutines from endless loop (#8967)
Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>
2021-10-29 16:39:02 -07:00
lzhfromustc d42be7be76
test:Fix two potential goroutine leaks (#8964)
Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>
2021-10-29 15:44:32 -07:00
Bryan Boreham 5afa606ecb
Remote-write: reuse memory for marshalling (#9412)
By holding a `proto.Buffer` per shard and passing it down to where
marshalling is done, we avoid creating a lot of garbage.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-10-29 14:44:40 -07:00
Robert Fratto bc72a718c4
Initial draft of prometheus-agent (#8785)
* Initial draft of prometheus-agent

This commit introduces a new binary, prometheus-agent, based on the
Grafana Agent code. It runs a WAL-only version of prometheus without the
TSDB, alerting, or rule evaluations. It is intended to be used to
remote_write to Prometheus or another remote_write receiver.

By default, prometheus-agent will listen on port 9095 to not collide
with the prometheus default of 9090.

Truncation of the WAL cooperates on a best-effort case with Remote
Write. Every time the WAL is truncated, the minimum timestamp of data to
truncate is determined by the lowest sent timestamp of all samples
across all remote_write endpoints. This gives loose guarantees that data
from the WAL will not try to be removed until the maximum sample
lifetime passes or remote_write starts functionining.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* add tests for Prometheus agent (#22)

* add tests for Prometheus agent

* add tests for Prometheus agent

* rearranged tests as per the review comments

* update tests for Agent

* changes as per code review comments

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* incremental changes to prometheus agent

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* changes as per code review comments

Signed-off-by: SriKrishna Paparaju <paparaju@gmail.com>

* Commit feedback from code review

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Port over some comments from grafana/agent

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Rename agent.Storage to agent.DB for tsdb consistency

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Consolidate agentMode ifs in cmd/prometheus/main.go

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* Document PreAction usage requirements better for agent mode flags

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* remove unnecessary defaultListenAddr

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* `go fmt ./tsdb/agent` and fix lint errors

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

Co-authored-by: SriKrishna Paparaju <paparaju@gmail.com>
2021-10-29 16:25:05 +01:00
David Leadbeater c91c2bbea5
promtool: Show more human readable got/exp output (#8064)
Avoid using %#v, nothing needs to parse this, so escaping " and so on
leads to hard to read output.

Add new lines, number and indentation to each alert series output.

Signed-off-by: David Leadbeater <dgl@dgl.cx>
2021-10-28 22:17:18 +11:00
DrAuYueng 69e309d202
Expose TargetsFromGroup/AlertmanagerFromGroup func and reuse this for (#9343)
static/file sd config check in promtool

Signed-off-by: DrAuYueng <ouyang1204@gmail.com>
2021-10-28 02:01:28 +02:00
Bryan Boreham c8b86f581c
Merge pull request #22 from grafana/cherry-pick-main
promql: copy data when short-circuiting (cherry pick)
2021-10-27 09:57:19 +01:00
Ben Kochie 8207b132fd
Merge pull request #9572 from prometheus/superq/cleanup_mixin_build
Use Go 1.16+ install for mixin tests
2021-10-27 10:38:45 +02:00
Bryan Boreham c2efd5bf96 promql: copy data when short-circuiting (#9552)
* promql: copy data when short-circuiting

Because the range query loop re-uses the output buffer each time round,
we must copy results into the buffer rather than using input as output.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
(cherry picked from commit a278ea4b58)
2021-10-26 23:17:28 +02:00
Xiaochao Dong c2d1c85857
close tsdb.head in test case (#9580)
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2021-10-26 11:36:25 +05:30
Julien Pivotto 25d06a9b76
Release 2.31.0-rc.1 (#9587)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-10-25 15:45:54 +02:00
Furkan Türkal 0c07663b70
fix: possible race on shared variables in test (#9470)
Fixes #9433

Signed-off-by: Furkan <furkan.turkal@trendyol.com>
2021-10-25 18:44:40 +05:30
Dieter Plaetinck d5bfbe3114
improve bstream comments and doc (#9560)
* improve bstream comments and doc

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* feedback

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
2021-10-25 18:44:15 +05:30
Julius Volz e7a12ac80a
Fix broken prefixed asset links in webpack build (#9586)
* Fix broken prefixed asset links in webpack build

Fixes https://github.com/prometheus/prometheus/issues/9585

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Use .env file for PUBLIC_URL npm build env var

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2021-10-25 12:52:13 +02:00
Furkan Türkal a6e6011d55
Add scrape_body_size_bytes metric (#9569)
Fixes #9520

Signed-off-by: Furkan <furkan.turkal@trendyol.com>
2021-10-24 23:45:31 +02:00