* write chunks via queue, predicting chunkrefs
* fixes
* linter
* cleanup
* better test coverage
* deduplicate operations when cutting file
* better comments
* fix race
* improvements based on PR feedback
* comments
* concurrent correctness test of write queue
* linter
* use isStarted flag in chunk write queue
* separate mutex for isStarted
* fixing tests
* fix race in test
* PR feedback
* add license headers
* PR feedback
* pr feedback
* dont recreate workerCtrl channel
* address PR feedback in high concurrency read and write test
* refactor the high concurrency write queue test
* pr feedback
* use require.Eventually
* typo
* Fixing comment
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
* use buffered chan for chunk write queue
* fix races
* address feedback on PR
* better comment
* less type casts
* dont reeimplement varint
* move mtx above property it protects
* improve tests by using wg instead of inspecting queue
* add comment
* add comment
* writeChunkF comment
* rename isStarted -> isRunning
* prevent panic when double stopping
* fix variable name in comment
* delete outdated comment
* naming and comment fixes in TestChunkWriteQueue_WrappingAroundSizeLimit
* use require.False instead of require.Equal(t, false
* always enable chunk write queue
* dont call callback with lock held
* undo rename of curFileSequence to curFileSeq
* fix accidental change
* fix comment
* Better syntax
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
* Better wording in comment
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
* better comments to explain the use of coordinator chans in test
* fix test which broke to the async writing of chunks
* undo previous renaming of shadowing variable
* rename var threadID to workerID
* rename variables based on PR feedback
* rename pos -> ts
* better implementation of whileNotCanceled
* update querySeriesRef before using it
* rename labels -> lbls
* initialize the chunk write queue operations metric with all possible labels
* dont return error from CutNewFile
* recreate chunk ref map regularly to free
* limit how often we recreate the chunk ref map
* fix typo in comment
* better comment
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
* Tool for CLI compactions.
* Use concurrency when populating symbols for multiple blocks.
* Use concurrency when writing to multiple output blocks.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Tests the the query being executed uses a context modified by the function.
Also modify reloadRules utility function to take time interval for the
Manager so the test runs faster. We'll work on synthetic clock later.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* track OOO-ness in tsdb into a histogram
This should help with building an understanding of customer requirements
as far as how far back samples tend to go.
Note: this has already been discussed, reviewed and approved on:
https://github.com/grafana/mimir/pull/488
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* update the test files which were not in mimir/vendor
* Increase time range for PrometheusHAGroupCrashlooping alert
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* Leave time range for clean restarts as-is
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* Be specific
Signed-off-by: Niko Smeds <nikosmeds@gmail.com>
* build(deps): bump k8s.io/api from 0.22.3 to 0.22.4 (#9819)
Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.22.3 to 0.22.4.
- [Release notes](https://github.com/kubernetes/api/releases)
- [Commits](https://github.com/kubernetes/api/compare/v0.22.3...v0.22.4)
---
updated-dependencies:
- dependency-name: k8s.io/api
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/jest from 27.0.2 to 27.0.3 in /web/ui (#9821)
Bumps [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest) from 27.0.2 to 27.0.3.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest)
---
updated-dependencies:
- dependency-name: "@types/jest"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump github.com/aws/aws-sdk-go from 1.42.6 to 1.42.8 (#9820)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.6 to 1.42.8.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.6...v1.42.8)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @fortawesome/react-fontawesome from 0.1.15 to 0.1.16 in /web/ui (#9812)
* build(deps): bump @fortawesome/react-fontawesome in /web/ui
Bumps [@fortawesome/react-fontawesome](https://github.com/FortAwesome/react-fontawesome) from 0.1.15 to 0.1.16.
- [Release notes](https://github.com/FortAwesome/react-fontawesome/releases)
- [Changelog](https://github.com/FortAwesome/react-fontawesome/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FortAwesome/react-fontawesome/compare/0.1.15...0.1.16)
---
updated-dependencies:
- dependency-name: "@fortawesome/react-fontawesome"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* update package-lock.json
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
* build(deps): bump k8s.io/client-go from 0.22.3 to 0.22.4 (#9818)
Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.22.3 to 0.22.4.
- [Release notes](https://github.com/kubernetes/client-go/releases)
- [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kubernetes/client-go/compare/v0.22.3...v0.22.4)
---
updated-dependencies:
- dependency-name: k8s.io/client-go
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump sinon from 11.1.2 to 12.0.1 in /web/ui (#9808)
* build(deps-dev): bump sinon from 11.1.2 to 12.0.1 in /web/ui
Bumps [sinon](https://github.com/sinonjs/sinon) from 11.1.2 to 12.0.1.
- [Release notes](https://github.com/sinonjs/sinon/releases)
- [Changelog](https://github.com/sinonjs/sinon/blob/master/docs/changelog.md)
- [Commits](https://github.com/sinonjs/sinon/compare/v11.1.2...v12.0.1)
---
updated-dependencies:
- dependency-name: sinon
dependency-type: direct:development
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* update package-lock.json
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
* build(deps): bump github.com/docker/docker (#9817)
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.10+incompatible to 20.10.11+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Changelog](https://github.com/moby/moby/blob/master/CHANGELOG.md)
- [Commits](https://github.com/docker/docker/compare/v20.10.10...v20.10.11)
---
updated-dependencies:
- dependency-name: github.com/docker/docker
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* downgrade bootstrap to v4 (#9827)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
* Fix focused expression input outline styling (#9829)
This is necessary due to this change in upstream CodeMirror:
73ec1a70be (diff-0e21fb1c9519a397050defc2148a2d1b966a0982cc4cf196e1e6b007f8d095e5)
Currently the expression input shows a weird dotted outline when focused.
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* fixes wrong metric name in documentation (#9828)
* fixes wrong metric name, see https://github.com/prometheus/prometheus/blob/main/discovery/openstack/hypervisor.go#L35
Signed-off-by: teuto.net Netzdienste GmbH <github@teuto.net>
* fixes parameter doc, sorted alphabetically
Signed-off-by: teuto.net Netzdienste GmbH <github@teuto.net>
* storage/remote: use t.TempDir instead of ioutil.TempDir on tests (#9811)
Signed-off-by: Matheus Alcantara <matheusssilv97@gmail.com>
* Export rules related structs in web/api/v1 (#9831)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* documentation: align kubernetes example with the prom operator and mixins
Signed-off-by: paulfantom <pawel@krupa.net.pl>
* build(deps): bump @codemirror/view from 0.19.19 to 0.19.20 in /web/ui (#9833)
Bumps [@codemirror/view](https://github.com/codemirror/view) from 0.19.19 to 0.19.20.
- [Release notes](https://github.com/codemirror/view/releases)
- [Changelog](https://github.com/codemirror/view/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/view/compare/0.19.19...0.19.20)
---
updated-dependencies:
- dependency-name: "@codemirror/view"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump eslint from 8.2.0 to 8.3.0 in /web/ui (#9835)
Bumps [eslint](https://github.com/eslint/eslint) from 8.2.0 to 8.3.0.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)
- [Commits](https://github.com/eslint/eslint/compare/v8.2.0...v8.3.0)
---
updated-dependencies:
- dependency-name: eslint
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump nock from 13.1.3 to 13.2.1 in /web/ui (#9836)
Bumps [nock](https://github.com/nock/nock) from 13.1.3 to 13.2.1.
- [Release notes](https://github.com/nock/nock/releases)
- [Changelog](https://github.com/nock/nock/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nock/nock/compare/v13.1.3...v13.2.1)
---
updated-dependencies:
- dependency-name: nock
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @codemirror/search from 0.19.2 to 0.19.3 in /web/ui (#9838)
Bumps [@codemirror/search](https://github.com/codemirror/search) from 0.19.2 to 0.19.3.
- [Release notes](https://github.com/codemirror/search/releases)
- [Changelog](https://github.com/codemirror/search/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/search/compare/0.19.2...0.19.3)
---
updated-dependencies:
- dependency-name: "@codemirror/search"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/node from 16.11.7 to 16.11.9 in /web/ui (#9839)
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 16.11.7 to 16.11.9.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)
---
updated-dependencies:
- dependency-name: "@types/node"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/react from 17.0.35 to 17.0.36 in /web/ui (#9837)
Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 17.0.35 to 17.0.36.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react)
---
updated-dependencies:
- dependency-name: "@types/react"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump @codemirror/state from 0.19.5 to 0.19.6 in /web/ui (#9840)
Bumps [@codemirror/state](https://github.com/codemirror/state) from 0.19.5 to 0.19.6.
- [Release notes](https://github.com/codemirror/state/releases)
- [Changelog](https://github.com/codemirror/state/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codemirror/state/compare/0.19.5...0.19.6)
---
updated-dependencies:
- dependency-name: "@codemirror/state"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* web:Fix a potential goroutine leak by channel (#8965)
Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>
* build(deps): bump github.com/aws/aws-sdk-go from 1.42.8 to 1.42.10 (#9845)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps-dev): bump @types/jquery from 3.5.8 to 3.5.9 in /web/ui (#9844)
Bumps [@types/jquery](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jquery) from 3.5.8 to 3.5.9.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jquery)
---
updated-dependencies:
- dependency-name: "@types/jquery"
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* cmd: use t.TempDir instead of ioutil.TempDir on tests (#9852)
* Check postings bytes length when decoding (#9766)
Added validation to expected postings length compared to the bytes slice
length. With 32bit postings, we expect to have 4 bytes per each posting.
If the number doesn't add up, we know that the input data is not
compatible with our code (maybe it's cut, or padded with trash, or even
written in a different coded).
This is needed in downstream projects to correctly identify cached
postings written with an unknown codec, but it's also a good idea to
validate it here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* update email
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
* promql: Remove unused Point.MarshalJSON method (#9860)
JSON marshaling is only needed for the HTTP API. Since Point is such a
frequently marshaled type, it gets an optimized treatment directly in
web/api/v1/api.go. The MarshalJSON method still provided in the promql
package is therefore unused and its existence is confusing.
Signed-off-by: beorn7 <beorn@grafana.com>
* Fix panic, out of order chunks, and race warning during WAL replay (#9856)
* Fix panic on WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Refactor: introduce walSubsetProcessor
walSubsetProcessor packages up the `processWALSamples()` function and
its input and output channels, helping to clarify how these things
relate.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Refactor: extract more methods onto walSubsetProcessor
This makes the main logic easier to follow.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Fix race warning by locking processWALSamples
Although we have waited for the processor to finish, we still get a
warning from the race detector because it doesn't know how the different
parts relate.
Add a lock round each batch of samples, so the race detector can see
that we never access series owned by the processor outside of a lock.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Added test to reproduce issue 9859
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Remove redundant unit test
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix out of order chunks during WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix nits
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Niko Smeds <nikosmeds@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <beorn@grafana.com>
Co-authored-by: Augustin Husson <husson.augustin@gmail.com>
Co-authored-by: Julius Volz <julius.volz@gmail.com>
Co-authored-by: teuto.net Netzdienste GmbH <github@teuto.net>
Co-authored-by: Matheus Alcantara <msalcantara@protonmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: paulfantom <pawel@krupa.net.pl>
Co-authored-by: Frederic Branczyk <fbranczyk@gmail.com>
Co-authored-by: lzhfromustc <43191155+lzhfromustc@users.noreply.github.com>
Co-authored-by: Matheus Alcantara <matheusssilv97@gmail.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: jessicagreben <jessicagrebens@gmail.com>
Co-authored-by: Jessica G <14338270+JessicaGreben@users.noreply.github.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix panic on WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Refactor: introduce walSubsetProcessor
walSubsetProcessor packages up the `processWALSamples()` function and
its input and output channels, helping to clarify how these things
relate.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Refactor: extract more methods onto walSubsetProcessor
This makes the main logic easier to follow.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Fix race warning by locking processWALSamples
Although we have waited for the processor to finish, we still get a
warning from the race detector because it doesn't know how the different
parts relate.
Add a lock round each batch of samples, so the race detector can see
that we never access series owned by the processor outside of a lock.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Added test to reproduce issue 9859
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Remove redundant unit test
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix out of order chunks during WAL replay
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix nits
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
JSON marshaling is only needed for the HTTP API. Since Point is such a
frequently marshaled type, it gets an optimized treatment directly in
web/api/v1/api.go. The MarshalJSON method still provided in the promql
package is therefore unused and its existence is confusing.
Signed-off-by: beorn7 <beorn@grafana.com>
* Add log to debug out-of-order chunks when compacting
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Revert test changes
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Do not use golang 1.17 specific functions
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Added validation to expected postings length compared to the bytes slice
length. With 32bit postings, we expect to have 4 bytes per each posting.
If the number doesn't add up, we know that the input data is not
compatible with our code (maybe it's cut, or padded with trash, or even
written in a different coded).
This is needed in downstream projects to correctly identify cached
postings written with an unknown codec, but it's also a good idea to
validate it here.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
