Right now the values for enqueuedSamples and enqueuedExemplars is never
subtracted leading to inflated values for failedSamples/failedExemplars
when a hard shutdown of a shard occurs.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
Channels can cause bottlenecks and tons of context switches when reading
hundreds of thousands of samples per second from a single queue.
Instead, pre-batch the samples to amortize the cost of the concurrency
overhead.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
* tsdb/agent: synchronize appender code with grafana/agent main
This commit synchronize the appender code with grafana/agent main. This
includes adding support for appending exemplars.
Closes#9610
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: fix build error
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: introduce some exemplar tests, refactor tests a little
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: address review feedback
- Re-use hash when creating a new series
- Fix typo in exemplar append error
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: remove unused AddFast method
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: close wal reader after test
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* tsdb/agent: add out-of-order tracking, change series TS in commit
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* address review feedback
Signed-off-by: Robert Fratto <robertfratto@gmail.com>
* Adapt UI for Prometheus Agent
UI is not my strongest skill, but I'd like to have something minimal for
the initial release of the agent.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Address review comments
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Add tests
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Add tests, serve only current mode paths
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Update js style, add agent test
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
The promtool check config command still uses the bearer_token_file
field which is deprecated in favour of authorization.credentials_file.
This commit modifies the command to use the new field insted.
Fixes#9874
Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>
Tests the the query being executed uses a context modified by the function.
Also modify reloadRules utility function to take time interval for the
Manager so the test runs faster. We'll work on synthetic clock later.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* track OOO-ness in tsdb into a histogram
This should help with building an understanding of customer requirements
as far as how far back samples tend to go.
Note: this has already been discussed, reviewed and approved on:
https://github.com/grafana/mimir/pull/488
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* update the test files which were not in mimir/vendor
`BufferedSeriesIterator` and `MemoizedSeriesIterator` use a method
called `Values` for exactly the purpose for which all other iterators
of the same kind use a method called `At`. That alone is confusing,
but on top of that, the `Values` method only returns a single sample,
not multiple values. I assume the naming has historical reasons. This
commit makes it more consistent. It is now easier to read, and now
`BufferedSeriesIterator` and `MemoizedSeriesIterator` implement
`chunkenc.Iterator` like many other iterators, too.
Signed-off-by: beorn7 <beorn@grafana.com>
There was a subtle and nasty bug in listSeriesIterator.Seek.
In addition, the Seek call is defined to be a no-op if the current
position of the iterator is already pointing to a suitable
sample. This commit adds fast paths for this case to several
potentially expensive Seek calls.
Another bug was in concreteSeriesIterator.Seek. It always searched the
whole series and not from the current position of the iterator.
Signed-off-by: beorn7 <beorn@grafana.com>
* remove vfsgen usages
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* web: use embed package for static assets
This requires go 1.16.
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* circleci: drop go generate in web/ui
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* Makefile: compress web assets before build
This commit add compression before (and decompression after) prometheus
is build. This ensures that gzipped assets are embeded in the prometheus
binary, if the builtinassets build tag is passed. If the build tag is
not passed this step is still executed but has no effect.
All this is executed in a subshell so that we can run the decompress
step even if the build step fails, but retain the exit code of promu.
This cleanup could also cover interrupts, but I left that out for now.
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* [FIXME]: add new module dependency on common/assets and temp replace
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* Increase time range for PrometheusHAGroupCrashlooping alert
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* Leave time range for clean restarts as-is
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* Be specific
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* build(deps): bump k8s.io/api from 0.22.3 to 0.22.4 (#9819)
Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.22.3 to 0.22.4.
- [Release notes](https://github.com/kubernetes/api/releases)
- [Commits](https://github.com/kubernetes/api/compare/v0.22.3...v0.22.4)
---
updated-dependencies:
- dependency-name: k8s.io/api
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/jest from 27.0.2 to 27.0.3 in /web/ui (#9821)
Bumps [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest) from 27.0.2 to 27.0.3.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest)
---
updated-dependencies:
- dependency-name: "@types/jest"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump github.com/aws/aws-sdk-go from 1.42.6 to 1.42.8 (#9820)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.6 to 1.42.8.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.6...v1.42.8)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @fortawesome/react-fontawesome from 0.1.15 to 0.1.16 in /web/ui (#9812)
* build(deps): bump @fortawesome/react-fontawesome in /web/ui
Bumps [@fortawesome/react-fontawesome](https://github.com/FortAwesome/react-fontawesome) from 0.1.15 to 0.1.16.
- [Release notes](https://github.com/FortAwesome/react-fontawesome/releases)
- [Changelog](https://github.com/FortAwesome/react-fontawesome/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FortAwesome/react-fontawesome/compare/0.1.15...0.1.16)
---
updated-dependencies:
- dependency-name: "@fortawesome/react-fontawesome"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* update package-lock.json
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
* build(deps): bump k8s.io/client-go from 0.22.3 to 0.22.4 (#9818)
Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.22.3 to 0.22.4.
- [Release notes](https://github.com/kubernetes/client-go/releases)
- [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kubernetes/client-go/compare/v0.22.3...v0.22.4)
---
updated-dependencies:
- dependency-name: k8s.io/client-go
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump sinon from 11.1.2 to 12.0.1 in /web/ui (#9808)
* build(deps-dev): bump sinon from 11.1.2 to 12.0.1 in /web/ui
Bumps [sinon](https://github.com/sinonjs/sinon) from 11.1.2 to 12.0.1.
- [Release notes](https://github.com/sinonjs/sinon/releases)
- [Changelog](https://github.com/sinonjs/sinon/blob/master/docs/changelog.md)
- [Commits](https://github.com/sinonjs/sinon/compare/v11.1.2...v12.0.1)
---
updated-dependencies:
- dependency-name: sinon
dependency-type: direct:development
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* update package-lock.json
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
* build(deps): bump github.com/docker/docker (#9817)
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.10+incompatible to 20.10.11+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Changelog](https://github.com/moby/moby/blob/master/CHANGELOG.md)
- [Commits](https://github.com/docker/docker/compare/v20.10.10...v20.10.11)
---
updated-dependencies:
- dependency-name: github.com/docker/docker
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* downgrade bootstrap to v4 (#9827)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
* Fix focused expression input outline styling (#9829)
This is necessary due to this change in upstream CodeMirror:
73ec1a70be (diff-0e21fb1c9519a397050defc2148a2d1b966a0982cc4cf196e1e6b007f8d095e5)
Currently the expression input shows a weird dotted outline when focused.
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* fixes wrong metric name in documentation (#9828)
* fixes wrong metric name, see https://github.com/prometheus/prometheus/blob/main/discovery/openstack/hypervisor.go#L35
Signed-off-by: teuto.net Netzdienste GmbH <github@teuto.net>
* fixes parameter doc, sorted alphabetically
Signed-off-by: teuto.net Netzdienste GmbH <github@teuto.net>
* storage/remote: use t.TempDir instead of ioutil.TempDir on tests (#9811)
Signed-off-by: Matheus Alcantara <matheusssilv97@gmail.com>
* Export rules related structs in web/api/v1 (#9831)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* documentation: align kubernetes example with the prom operator and mixins
Signed-off-by: paulfantom <pawel@krupa.net.pl>
* build(deps): bump @codemirror/view from 0.19.19 to 0.19.20 in /web/ui (#9833)
Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.19 to 0.19.20.
- [Release notes](https://github.com/codemirror/view/releases)
- [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/view/compare/0.19.19...0.19.20)
---
updated-dependencies:
- dependency-name: "@codemirror/view"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump eslint from 8.2.0 to 8.3.0 in /web/ui (#9835)
Bumps [eslint](https://github.com/eslint/eslint) from 8.2.0 to 8.3.0.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)
- [Commits](https://github.com/eslint/eslint/compare/v8.2.0...v8.3.0)
---
updated-dependencies:
- dependency-name: eslint
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump nock from 13.1.3 to 13.2.1 in /web/ui (#9836)
Bumps [nock](https://github.com/nock/nock) from 13.1.3 to 13.2.1.
- [Release notes](https://github.com/nock/nock/releases)
- [Changelog](https://github.com/nock/nock/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nock/nock/compare/v13.1.3...v13.2.1)
---
updated-dependencies:
- dependency-name: nock
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @codemirror/search from 0.19.2 to 0.19.3 in /web/ui (#9838)
Bumps [@codemirror/search](https://github.com/codemirror/search) from 0.19.2 to 0.19.3.
- [Release notes](https://github.com/codemirror/search/releases)
- [Changelog](https://github.com/codemirror/search/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/search/compare/0.19.2...0.19.3)
---
updated-dependencies:
- dependency-name: "@codemirror/search"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/node from 16.11.7 to 16.11.9 in /web/ui (#9839)
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.7 to 16.11.9.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)
---
updated-dependencies:
- dependency-name: "@types/node"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/react from 17.0.35 to 17.0.36 in /web/ui (#9837)
Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.35 to 17.0.36.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react)
---
updated-dependencies:
- dependency-name: "@types/react"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @codemirror/state from 0.19.5 to 0.19.6 in /web/ui (#9840)
Bumps [@codemirror/state](https://github.com/codemirror/state) from 0.19.5 to 0.19.6.
- [Release notes](https://github.com/codemirror/state/releases)
- [Changelog](https://github.com/codemirror/state/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/state/compare/0.19.5...0.19.6)
---
updated-dependencies:
- dependency-name: "@codemirror/state"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* web:Fix a potential goroutine leak by channel (#8965)
Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>
* build(deps): bump github.com/aws/aws-sdk-go from 1.42.8 to 1.42.10 (#9845)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/jquery from 3.5.8 to 3.5.9 in /web/ui (#9844)
Bumps [@types/jquery](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jquery) from 3.5.8 to 3.5.9.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jquery)
---
updated-dependencies:
- dependency-name: "@types/jquery"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* cmd: use t.TempDir instead of ioutil.TempDir on tests (#9852)
* Check postings bytes length when decoding (#9766)
Added validation to expected postings length compared to the bytes slice
length. With 32bit postings, we expect to have 4 bytes per each posting.
If the number doesn't add up, we know that the input data is not
compatible with our code (maybe it's cut, or padded with trash, or even
written in a different coded).
This is needed in downstream projects to correctly identify cached
postings written with an unknown codec, but it's also a good idea to
validate it here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* update email
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
* promql: Remove unused Point.MarshalJSON method (#9860)
JSON marshaling is only needed for the HTTP API. Since Point is such a
frequently marshaled type, it gets an optimized treatment directly in
web/api/v1/api.go. The MarshalJSON method still provided in the promql
package is therefore unused and its existence is confusing.
Signed-off-by: beorn7 <beorn@grafana.com>
* Fix panic, out of order chunks, and race warning during WAL replay (#9856)
* Fix panic on WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Refactor: introduce walSubsetProcessor
walSubsetProcessor packages up the `processWALSamples()` function and
its input and output channels, helping to clarify how these things
relate.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Refactor: extract more methods onto walSubsetProcessor
This makes the main logic easier to follow.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Fix race warning by locking processWALSamples
Although we have waited for the processor to finish, we still get a
warning from the race detector because it doesn't know how the different
parts relate.
Add a lock round each batch of samples, so the race detector can see
that we never access series owned by the processor outside of a lock.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Added test to reproduce issue 9859
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Remove redundant unit test
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix out of order chunks during WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix nits
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Niko Smeds <nikosmeds@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <beorn@grafana.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: Julius Volz <julius.volz@gmail.com>
Co-authored-by: teuto.net Netzdienste GmbH <github@teuto.net>
Co-authored-by: Matheus Alcantara <msalcantara@protonmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: paulfantom <pawel@krupa.net.pl>
Co-authored-by: Frederic Branczyk <fbranczyk@gmail.com>
Co-authored-by: lzhfromustc <43191155+lzhfromustc@users.noreply.github.com>
Co-authored-by: Matheus Alcantara <matheusssilv97@gmail.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: jessicagreben <jessicagrebens@gmail.com>
Co-authored-by: Jessica G <14338270+JessicaGreben@users.noreply.github.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix panic on WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Refactor: introduce walSubsetProcessor
walSubsetProcessor packages up the `processWALSamples()` function and
its input and output channels, helping to clarify how these things
relate.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Refactor: extract more methods onto walSubsetProcessor
This makes the main logic easier to follow.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Fix race warning by locking processWALSamples
Although we have waited for the processor to finish, we still get a
warning from the race detector because it doesn't know how the different
parts relate.
Add a lock round each batch of samples, so the race detector can see
that we never access series owned by the processor outside of a lock.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Added test to reproduce issue 9859
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Remove redundant unit test
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix out of order chunks during WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix nits
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
JSON marshaling is only needed for the HTTP API. Since Point is such a
frequently marshaled type, it gets an optimized treatment directly in
web/api/v1/api.go. The MarshalJSON method still provided in the promql
package is therefore unused and its existence is confusing.
Signed-off-by: beorn7 <beorn@grafana.com>
* Add log to debug out-of-order chunks when compacting
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Revert test changes
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Do not use golang 1.17 specific functions
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Added validation to expected postings length compared to the bytes slice
length. With 32bit postings, we expect to have 4 bytes per each posting.
If the number doesn't add up, we know that the input data is not
compatible with our code (maybe it's cut, or padded with trash, or even
written in a different coded).
This is needed in downstream projects to correctly identify cached
postings written with an unknown codec, but it's also a good idea to
validate it here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
