Commit graph

8417 commits

Author SHA1 Message Date
Julien Pivotto 8bc369bf9b
Calculate head chunk size based on actual disk usage (#8139)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-11-03 15:34:59 +05:30
Jarod Watkins 388a2d20c9
UI: Add toggle to enable/disable metric autocomplete (#8134)
* UI: Add toggle to enable/disable metric autocomplete

This change adds a toggle to enable or disable the metric autocomplete
functionality. By default it is enabled. This is a port of a change I
did in [Thanos][1].

[1]: https://github.com/thanos-io/thanos/pull/3381

Signed-off-by: Jarod Watkins <jarod@42lines.net>

* Adding full variable name

Signed-off-by: Jarod Watkins <jarod@42lines.net>
2020-11-02 16:16:54 +01:00
Julien Pivotto 7558e9d3c3
API: Make snapshot length always the same size (#8138)
Fixes #8135

Minor improvement, also we now use Int64 even on 32 bit systems.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-11-01 21:07:38 +01:00
Marco Pracucci 63be30dcee
Fixed WAL corruption on partial writes within a page (#8125)
* Fixed WAL corruption on partial writes

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed variable

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2020-10-29 16:07:03 +05:30
Julien Pivotto 318190021d
Enforce the usage of github.com/stretchr/testify/require (#8129)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 11:09:08 +01:00
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122)
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Bartlomiej Plotka 3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. (#8066)
* MultiError: Refactored MultiError for more concise and safe usage.

* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename package to merrors. (: In different PR.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed review comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix after rebase.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-28 15:24:58 +00:00
johncming 28ca0965f0
tsdb/chunkenc: fix typo of return error. (#7670)
* tsdb/chunkenc: fix typo of return error.

Signed-off-by: johncming <johncming@yahoo.com>

* tsdb: fix typo of function in markdonw.

Signed-off-by: johncming <johncming@yahoo.com>
2020-10-28 12:03:11 +00:00
Jorge Luis Betancourt 4dc755cd27
Add a metric for tracking max_samples_per_send (#8102)
Currently there is no way of tracking the value of the
`max_samples_per_send` configuration option, which is commonly tweaked
when integrating with a remote write backend.

Signed-off-by: Jorge Luis Betancourt Gonzalez <jorge-luis.betancourt@trivago.com>
2020-10-28 11:39:36 +00:00
Ganesh Vernekar 3245b3267b
Don't use returned DB to close resources on TSDB startup error (#8113)
* Don't use returned DB to close resources on TSDB startup error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add unit test and fix another panic

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comment

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-28 15:39:03 +05:30
Julien Pivotto daf382e4a9
Merge pull request #8112 from prometheus/release-2.22
Merge release 2.22 to master
2020-10-27 11:07:13 +01:00
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110)
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-27 11:06:53 +01:00
Anthony D'Atri 2cbc0f9bfe
Various doc tweaks (#8111)
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
2020-10-27 09:50:37 +00:00
Brian Brazil 3f8e51738c
More granular locking for scrapeLoop. (#8104)
Don't lock for all of Sync/stop/reload as that holds up /metrics and the
UI when they want a list of active/dropped targets. Instead take
advantage of the fact that Sync/stop/reload cannot be called
concurrently by the scrape Manager and lock just on the targets
themselves.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-10-26 14:46:20 +00:00
David Leadbeater e7e60623ff
promtool: Calculate mint and maxt per test (#8096)
* promtool: Calculate mint and maxt per test

Previously a single test that used a later eval time would make all
other tests in the file share the [mint, maxt] and potentially evaluate
far more samples than needed.

Fixes: #8019

Signed-off-by: David Leadbeater <dgl@dgl.cx>
2020-10-24 12:03:55 +01:00
Piper 712477be5a
docs: fix "timeseries" -> "time series" in README.md (#8107)
Signed-off-by: Piper <piperchester@gmail.com>
2020-10-24 10:27:24 +01:00
Harkishen Singh 9599bb9b38
Update remote_write default values in docs (#8103)
Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2020-10-23 12:12:14 +01:00
like-inspur 29b551225b
add networking.k8s.io for ingress (#8091)
* add networking.k8s.io for ingress

level=error ts=2020-10-19T08:32:30.544Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:494: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:monitoring:prometheus\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope"

Signed-off-by: root <likerj@inspur.com>

* Update rbac-setup.yml

Signed-off-by: root <likerj@inspur.com>
2020-10-22 15:08:12 -06:00
James Ranson 1cffda5de7
react updates for pathPrefix (#7979)
* dynamically determine path prefix

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* minor changes per PR review

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* use Context for apiPath and pathPrefix

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* remove unhandled "/version" path

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* only process index once instead of on every req

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* remove unneeded tag fragment

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* switch api path to const

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* revert

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* update tests

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* linter updates

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* simplify

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>

* updates per peer review

Signed-off-by: James Ranson <james_ranson@cable.comcast.com>
2020-10-22 17:22:32 +02:00
Victor Araujo 74775d7324
Add major version tag (#8026)
When the latest tag is added, also add a tag for the major version
when the version tag matches the latest release defined in VERSION.

Signed-off-by: Victor Araujo <vear91@gmail.com>
2020-10-22 17:19:44 +02:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Arthur Silva Sens c5a832b394
Close resources after failing to startup TSDB (#8031)
* Close resources after failing to startup TSDB

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Return close error instead of logging

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change named return's name

Signed-off-by: arthursens <arthursens2005@gmail.com>
2020-10-21 20:38:28 +05:30
Brian Brazil fdf1c29224
Fix metric description of prometheus_tsdb_symbol_table_size_bytes. (#8080)
This is how much memory we use to load in the on-disk
symbol tables, not the size of the tables themselves.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-10-21 14:35:40 +01:00
Ganesh Vernekar ce4b3ac282
Simplify TestHeadReadWriter_Truncate (#7437)
* Simplify TestHeadReadWriter_Truncate

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-21 18:41:39 +05:30
Ganesh Vernekar 2624d827fa
Read repair empty last file in chunks_head (#8061)
* Read repair empty file in chunks_head

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor and introduce repairLastChunkFile

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Attempt windows test fix

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-21 18:27:13 +05:30
Frederic Branczyk 3b08b2dabe
Merge pull request #8083 from metalmatze/mixin-jsonnet-absolute-import-paths
Use absolute jsonnet import paths
2020-10-21 11:39:23 +02:00
Ricky Pai fa8722be4a
add link to circleci config in README (#8089)
Signed-off-by: Ricky Pai <rickyp999+github@gmail.com>
2020-10-21 09:37:18 +02:00
Matthias Loibl 13ba013a24
Use absolute jsonnet import paths
This should be the way forward when importing libraries in jsonnet. It's
closer to how Go imports look and makes it more obvious where packages
live.

This is not breaking anything, as the old imports were already symlinks
to the now directly used directories.

Signed-off-by: Matthias Loibl <mail@matthiasloibl.com>
2020-10-20 11:42:30 +02:00
Bartlomiej Plotka 2fe1e9fa93
Create a checkpoint only at the end of Compact call (#8067)
* Create a checkpoint only at the end of Compact call

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix Bartek's offline reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Introduce TruncateInMemory and TruncateWAL

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Small enhancements and test fixing attempts

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add TestOneCheckpointPerCompactCall

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Don't truncate WAL on block compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplified the algo.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Better protection around calling truncateWAL, truncate WAL on Head compaction error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-19 20:57:08 +05:30
Ganesh Vernekar d9cd913219
Add links to TSDB blog post series in tsdb/README.md (#8078)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-19 17:17:10 +05:30
Julien Pivotto 59733b1a26
TSDB: use blocks instead of db.blocks in condition (#8068)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-19 16:51:54 +05:30
Ganesh Vernekar 5de7d15879
Consistent naming and better names for fields in TSDB format docs (#8077)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-19 14:27:19 +05:30
Julien Pivotto 8c9850c2bb
Remote: Do not collect non-initialized timestamp metrics (#8060)
* Remote: Do not collect non-initialized timestamp metrics

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-15 23:53:59 +02:00
Frederic Branczyk 0a7fdd3b76
Merge pull request #8059 from brancz/cut-2.22.0
*: Cut v2.22.0
2020-10-15 13:07:37 +02:00
Bartlomiej Plotka 9981b3f3ee
labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025)
* labels: Reduce allocated memory by Hash method in edge cases; Added tests.

Old:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 5366161	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 1700371	       767 ns/op	    2048 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	     356	   3743115 ns/op	10523442 B/op	       1 allocs/op
PASS

New:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 4758883	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 3324492	       357 ns/op	      80 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	    1087	   1083949 ns/op	    9734 B/op	       1 allocs/op
PASS

Process finished with exit code 0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Kemal's comment.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Moved to v2 xxhash for improvements.

New:

/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ -test.benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	53447894	       221 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	42341754	       326 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	   10000	   1248546 ns/op	       0 B/op	       0 allocs/op
PASS

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed old xxhash package.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Pined minor version.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-15 11:31:28 +01:00
Björn Rabenstein d49f267f76
Merge pull request #8054 from simonpasquier/improve-not-ingesting-samples-alert
documentation/prometheus-mixin: improve PrometheusNotIngestingSamples
2020-10-15 12:29:39 +02:00
Simon Pasquier f381d8a9bd documentation/prometheus-mixin: improve PrometheusNotIngestingSamples
The alert shouldn't fire when there's no target and no rule configured.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2020-10-15 11:13:17 +02:00
Tobias Schmidt 77c1da8f9d
Reflect automatic docs publishing in the release process (#8056)
No more need to do anything manually for release docs.

Signed-off-by: Tobias Schmidt <tobidt@gmail.com>
2020-10-15 11:06:42 +02:00
Ganesh Vernekar 4ed3ec7b46
More info in mmap error message (#8058)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-10-15 12:58:13 +05:30
Julien Pivotto 4596abee4d
Mixin: Ignore unset remote write timestamp (#8046)
* Mixin: Ignore unset remote write timestamp

This pull request ignores the zero value of highest_sent_timestamp_seconds
in Highest Timestamp In vs. Highest Timestamp Sent which just show that
remote write has not been successful yet.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-15 09:15:59 +02:00
Frederic Branczyk e1406c9406
*: Cut v2.22.0
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
2020-10-15 07:50:33 +02:00
frbimo 260cd84d8f
log on empty head compaction result (#8037)
Signed-off-by: frbimo <fr.bimo@gmail.com>
2020-10-14 18:59:59 +05:30
Gayathri Venkatesh 73e2ce1bd6
Do not ignore reload errors in compactHead (#8051)
* Modified unknownRefs to unknownRefs.Load()

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Modified db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Revert "Modified unknownRefs to unknownRefs.Load()"

This reverts commit 79caf595fa9b9193878dc0dd9dec17d58851ae42.

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Made changes to reload error in db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>
2020-10-14 15:05:24 +05:30
Goutham Veeramachaneni 1cc02930b6
Implement series API query hint to not hit chunks (#8050)
Fixes https://github.com/prometheus/prometheus/issues/5547

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2020-10-14 14:36:17 +05:30
Ling Jin 9145200842
tsdb: fix unkown ref in log (#8048)
Signed-off-by: JinLingChristopher <jinl1037@hotmail.com>
2020-10-13 20:03:16 +05:30
Bartlomiej Plotka dee6ac274e
labels: Extra tests and comment for WithoutLabels method. (#8024)
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-13 08:57:53 +01:00
Björn Rabenstein 71577e45eb
Merge pull request #8044 from prometheus/beorn7/metrics
Instrumentation: Report valid configs in the respective metrics from the beginning
2020-10-12 23:32:02 +02:00
Arthur Silva Sens 4f45e201cc
Promtool tsdb list now prints block sizes (#7993)
* promtool tsdb list now prints blocks' size

Signed-off-by: arthursens <arthursens2005@gmail.com>
2020-10-12 23:15:40 +02:00
beorn7 0f3c1bf6cf Report valid configs in the respective metrics from the beginning
In #7399, an early validity check of the config was introduced to
prevent the scenario where an invalid config is only detected after a
possibly very long startup procedure. However, the respective success
metrics are not updated after the initial validation so that the
success metrics suggest an invalid config. If the startup procedure,
like replaying the WAL, really takes very long, alerts about invalid
config will trigger.

This commit sets the succes metrics after initial validation. They
will be set again after the "real" config (re-)load, but that
shouldn't be a problem. The metric now truthfully represents whenever
the config was successfully loaded, no matter if the result was then
thrown away (because it was just for validation) or actually used.

Signed-off-by: beorn7 <beorn@grafana.com>
2020-10-12 21:30:59 +02:00
Jess G 90680b092c
Add a tsdb writer interface (#7950)
* add a tsdb writer interface

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in tsdbutil block creator

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* pass thru context

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in blockutil and also in tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* fix link err

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add err handling for defer

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add blockwriter test

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* mv initHead into NewBlockWriter, update tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use blocksize arg, make test more concise

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-10-12 17:04:20 +01:00