Commit graph

9683 commits

Author SHA1 Message Date
Augustin Husson 74f497ac18 Update web/ui/react-app/src/pages/serviceDiscovery/Services.tsx
Co-authored-by: Julius Volz <julius.volz@gmail.com>
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2022-01-19 14:02:39 +01:00
Augustin Husson b9bc8e7c33 create a component to handle the search bar with debounce
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2022-01-12 16:23:38 +01:00
Björn Rabenstein 933f50bcda
Merge pull request #10121 from prometheus/beorn7/promql
PromQL: Promote negative offset and @ modifer to stable
2022-01-12 14:09:46 +01:00
Abhishek 2f4289a3bf
add limit to the rules api (#10152) 2022-01-11 22:44:22 -05:00
Chris Marchbanks eb7c10dbbd
Merge pull request #9274 from bboreham/improve-rw-sharding
remote_write: shard up more when backlogged
2022-01-11 13:23:23 -07:00
beorn7 c2b80d8643 PromQL: Test negative offset in PromQL tests
Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-11 18:23:40 +01:00
beorn7 b39f2739e5 PromQL: Always enable negative offset and @ modifier
This follows the line of argument that the invariant of not looking
ahead of the query time was merely emerging behavior and not a
documented stable feature. Any query that looks ahead of the query
time was simply invalid before the introduction of the negative offset
and the @ modifier.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-11 17:08:55 +01:00
dependabot[bot] 076109fa19
build(deps): bump github.com/aws/aws-sdk-go from 1.42.28 to 1.42.31 (#10146)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.28 to 1.42.31.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.28...v1.42.31)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-11 15:56:25 +01:00
Ganesh Vernekar 129ed4ec8b
Fix Example() function in TSDB (#10153)
* Fix Example() function in TSDB

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-01-11 17:24:03 +05:30
beorn7 61509fc840 PromQL: Promote negative offset and @ modifer to stable
Following the argument that breaking the invariant that PromQL does
not look ahead of the evaluation time implies a breaking change, we
still need to keep the feature flag around, but at least we can
communicate that the feature is considered stable, and that the
feature flags will be ignored from v3 on.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-11 00:34:33 +01:00
dependabot[bot] 714bc3ff29
build(deps): bump google.golang.org/api from 0.63.0 to 0.64.0 (#10135)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.63.0 to 0.64.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.63.0...v0.64.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-10 23:36:26 +01:00
Augustin Husson dd02675467
move dependabot frequency running to monthly (#10144)
Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2022-01-10 23:26:24 +01:00
Augustin Husson bff9d06874
Service Discovery Page rework (#10131)
* rework the target page

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* put back the URL of the endpoint

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* replace old code by the new one and change function style

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* align filter and search bar on the same row

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* remove unnecessary return

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* upgrade kvsearch to v0.3.0

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix unit test

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* add missing style on column

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* add placeholder and autofocus

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* put back the previous table design

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix issue relative to the position of the tooltip

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix health filter

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix test on label tooltip

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* simplify filter condition

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* rework service discovery page

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* introduced generic custom infinite scroll component

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* adjust the placeholder in discovery page

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* ignore returning type missing

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* apply fix required by the review

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* index discoveredLabels

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2022-01-10 17:39:51 +01:00
Björn Rabenstein 0f4a1e6eac
Merge pull request #10119 from prometheus/beorn7/remote
API: Promote remote-write-receiver to stable
2022-01-10 15:55:10 +01:00
Augustin Husson dd9f96b893
rework the target page (#10103)
* rework the target page

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* put back the URL of the endpoint

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* replace old code by the new one and change function style

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* align filter and search bar on the same row

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* remove unnecessary return

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* upgrade kvsearch to v0.3.0

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix unit test

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* add missing style on column

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* add placeholder and autofocus

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* put back the previous table design

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix issue relative to the position of the tooltip

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix health filter

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* fix test on label tooltip

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>

* simplify filter condition

Signed-off-by: Augustin Husson <husson.augustin@gmail.com>
2022-01-10 15:53:14 +01:00
Mauro Stettler 0df3489275
Write chunks via queue, predicting the refs (#10051)
* Write chunks via queue, predicting the refs

Our load tests have shown that there is a latency spike in the
remote write handler whenever the head chunks need to be written,
because chunkDiskMapper.WriteChunk() blocks until the chunks are written
to disk.

This adds a queue to the chunk disk mapper which makes the WriteChunk()
method non-blocking unless the queue is full. Reads can still be served
from the queue.

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* address PR feeddback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* initialize metrics without .Add(0)

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* change isRunningMtx to normal lock

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* do not re-initialize chunkrefmap

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update metric outside of lock scope

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add benchmark for adding job to chunk write queue

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary "success" var

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* gofumpt -extra

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* avoid WithLabelValues call in addJob

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* format comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* addressing PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* rename cutExpectRef to cutAndExpectRef

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* use head.Init() instead of .initTime()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* address PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* PR feedback

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update test according to PR feedback

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* replace callbackWg -> awaitCb

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* better test of truncation with empty files

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* replace callbackWg -> awaitCb

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2022-01-10 13:36:45 +00:00
Bryan Boreham 954c0e8020 remote_write: round desired shards up before check
Previously we would reject an increase from 2 to 2.5 as being
within 30%; by rounding up first we see this as an increase from 2 to 3.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-01-10 09:57:37 +00:00
Bryan Boreham 6d01ce8c4d remote_write: shard up more when backlogged
Change the coefficient from 1% to 5%, so instead of targetting to clear
the backlog in 100s we target 20s.

Update unit test to reflect the new behaviour.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-01-10 09:57:37 +00:00
Bryan Boreham d588b14d9c remote_write: detailed test for shard calculation
Drive the input parameters to `calculateDesiredShards()` very precisely,
to illustrate some questionable behaviour marked with `?!`.

See https://github.com/prometheus/prometheus/issues/9178,
https://github.com/prometheus/prometheus/issues/9207,

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-01-10 09:57:37 +00:00
chenlujjj 2ce94ac196
Add '--weight' flag to 'promtool check metrics' command (#10045) 2022-01-07 16:58:28 -05:00
Victor Häggqvist 931acc3ee8
fix overlapping click-targets of alert state checkboxes (#10136)
using the same margin as checkboxes on top of /graph

Signed-off-by: Victor Häggqvist <victor@snilius.com>
2022-01-07 10:38:01 +01:00
Björn Rabenstein 671bb0d5b8
Merge pull request #10128 from prometheus/beorn7/release
Update Go dependencies in preparation for v2.33
2022-01-06 18:48:20 +01:00
Björn Rabenstein 366cba876d
Merge pull request #10110 from prometheus/beorn7/release2
release: Extend instructions for the release shepherd
2022-01-06 18:03:00 +01:00
beorn7 4727c41a4b Update Go dependencies
Note that go.mod has been manually modified to use the following
versions:

- k8s.io/api v0.22.4
- k8s.io/apimachinery v0.22.4
- k8s.io/client-go v0.22.4

The current version of those modules requires Go1.17, which we only
intend to require once Go1.18 is out.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-06 16:22:33 +01:00
Oleg Zaytsev a83d46ee9c
Tidy postingsWithIndexHeap (#10123)
Unexported postingsWithIndexHeap's methods that don't need to be
exported, and added detailed comments.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2022-01-06 16:03:44 +05:30
Bryan Boreham 82860a770c
tsdb: use simpler map key to improve exemplar ingest performance (#10111)
* tsdb: fix exemplar benchmarks

Go benchmarks are expected to do an amount of work that varies with
the `b.N` parameter. Previously these benchmarks would report a result
like 0.01 ns/op, which is nonsense.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* tsdb: use simpler map key to improve exemplar perf

Prometheus holds an index of exemplars so it can discard the oldest one
for a series when a new one is added.
Since the keys are not for human eyes, we can use a simpler format
and save the effort of quoting label values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Exemplars: allocate index map with estimated size

This avoids Go having to re-size the map several times as it grows.
16 exemplars per series is a guess; if it is too low then the map will
be sparse, while if it is too high then the map will have to resize once
or twice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-01-06 15:58:58 +05:30
Chris Marchbanks ba03f7fc23
Merge pull request #10102 from prometheus/update-metrics-on-rw-fails
Update sent timestamp when write irrecoverably fails
2022-01-05 10:46:09 -07:00
beorn7 8fdfa52976 API: Promote remote-write-receiver to stable
Since `/api/v1/write` is a mutating endpoint, we should still activate
the remote-write-receiver explicitly. But we should do it in the same
way as the other mutating endpoints, i.e. via a flag
`--web.enable-remote-write-receiver`.

This commit marks the feature flag as deprecated, i.e. it still works
but logs a warning on startup. This enables users to seamlessly
migrate. With the next minor release, we can start ignoring the
feature flag (but still warn a user that is trying to use it).

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-05 15:36:07 +01:00
Shihao Xia d26fd5c97b
fix potential goroutine leaks (#10057)
Signed-off-by: Shihao Xia <charlesxsh@hotmail.com>
2022-01-05 16:21:06 +05:30
Oleg Zaytsev 701545286d
Pop intersected postings heap without popping (#10092)
See this comment for detailed explanation:

https://github.com/prometheus/prometheus/pull/9907#issuecomment-1002189932

TL;DR: if we don't call Pop() on the heap implementation, we don't need
to return our param as an `interface{}` so we save an allocation.
This would be popped for every label value, so it can be thousands of
saved allocations here (see benchmarks).

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2022-01-05 16:16:43 +05:30
beorn7 3be33de316 release: Extend instructions for the release shepherd
This adds the following:
- Find a new volunteer for the release shepherd if needed.
- Consider experimental features for promotion/demotion.

The latter could need more detailed explanation, but I guess we can
add those once we have the planned metrics for feature flags. For now,
I just wanted to have that point in so that it is not completely
forgotten.

Signed-off-by: beorn7 <beorn@grafana.com>

Co-authored-by: Levi Harrison <git@leviharrison.dev>
2022-01-05 11:41:07 +01:00
Björn Rabenstein 5a68a41f13
Merge pull request #10109 from prometheus/beorn7/release
release: Volunteer @beorn7 and @csmarchbanks for 2.33 and 2.34
2022-01-05 11:26:09 +01:00
Peter Štibraný e51a17b501
CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108)
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
2022-01-05 15:10:00 +05:30
beorn7 b75843543c release: Volunteer @beorn7 and @csmarchbanks for 2.33 and 2.34
In a meeting at Grafana Labs, we had the shocking revelation that the
next release is scheduled to happen soon, but there is no release
shepherd for it yet.

(Which reminds me: Perhaps we should make it a responsibility of the
release shepherd to make sure there is one for the next release, too?)

Given my current part-time commitment, I would actually try to avoid
being the release shepherd, but on the other hand, it's about time
that a Grafanista to do the job again, and the only one who could do
it at all in the room was me. @csmarchbanks volunteered for the next
release. So here we are.

Of course, if any other team member would like to jump in and do this
release or the next, we'll happily insert you into the list.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-04 16:51:25 +01:00
Goutham Veeramachaneni 6696b7a5f0
Don't update metrics on context cancellation
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2022-01-04 10:46:52 +01:00
Chris Marchbanks dfa5cb7462
Merge pull request #10038 from charlesxsh/fix-TestReshardRaceWithStop
add proper exit for loop
2022-01-03 09:02:45 -07:00
Goutham Veeramachaneni 1af81dc5c9
Update sent timestamp when write irrecoverably fails.
We have an alert that fires when prometheus_remote_storage_highest_timestamp_in_seconds - prometheus_remote_storage_queue_highest_sent_timestamp_seconds
becomes too high. But we have an agent that fires this when the remote "rate-limits" the user.

This is because prometheus_remote_storage_queue_highest_sent_timestamp_seconds doesn't get updated
when the remote sends a 429.

I think we should update the metrics, and the change I made makes sense. Because if the requests fails
because of connectivity issues, etc. we will never exit the `sendWriteRequestWithBackoff` function. It only
exits the function when there is a non-recoverable error, like a bad status code, and in that case, I think
the metric needs to be updated.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2022-01-03 11:13:48 +01:00
David Leadbeater a961062c37
Disable time based retention in tests (#8818)
Fixes #7699.

Signed-off-by: David Leadbeater <dgl@dgl.cx>
2022-01-02 23:46:03 +01:00
Drew DeVault 8b0e1efd4f
Merge pull request #10100 from ddevault/main
codemirror-promql: use npm to run lezer-generator
2022-01-02 11:41:47 +01:00
Ben Kochie b277571b24
Merge pull request #10097 from prometheus/superq/bump_go
Update minimum Go version
2022-01-02 08:56:55 +01:00
Jessica G 174a1147d5
Merge pull request #9861 from JessicaGreben/minor-prom-improvements
Add exit code constants in promtool
2021-12-31 12:07:02 -08:00
SuperQ a78a3cea02
Rollback k8s.io to 0.22.4.
Signed-off-by: SuperQ <superq@gmail.com>
2021-12-31 17:39:40 +01:00
SuperQ 41b43c3b87
Update minimum Go version
Update minimum required Go version to 1.16 due to use of standard
library package `io/fs`.

Signed-off-by: SuperQ <superq@gmail.com>
2021-12-31 17:29:00 +01:00
jessicagreben 4b03fa3100 replace config exit code with failure exit code
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-12-30 05:37:57 -08:00
Shihao Xia c3e7bfb813 add proper exit for loop
Signed-off-by: Shihao Xia <charlesxsh@hotmail.com>
2021-12-29 23:48:11 -05:00
dependabot[bot] 51b0826925
build(deps): bump github.com/aws/aws-sdk-go from 1.42.24 to 1.42.25 (#10093)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-28 16:56:50 -05:00
dependabot[bot] e0cec62025
build(deps): bump github.com/Azure/azure-sdk-for-go from 60.2.0+incompatible to 60.3.0+incompatible (#10086)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-28 12:46:04 -05:00
dependabot[bot] 440974f403
build(deps): bump github.com/Azure/go-autorest/autorest/adal from 0.9.17 to 0.9.18 (#10089)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-28 11:56:21 -05:00
dependabot[bot] d95d588d0e
build(deps): bump github.com/miekg/dns from 1.1.44 to 1.1.45 (#10088)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-28 11:55:15 -05:00
Oleg Zaytsev 3947238ce0
Label values with matchers by intersecting postings (#9907)
* LabelValues w/matchers by intersecting postings

Instead of iterating all matched series to find the values, this
checks if each one of the label values is present in the matched series
(postings).

Pending to be benchmarked.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Benchmark labelValuesWithMatchers

name                                                                   old time/op    new time/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                         157ms ± 0%        48ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                      1.80s ± 0%       0.46s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"                144ms ± 0%        57ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"      304ms ± 0%       111ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                      761ms ± 0%       164ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        6.11µs ± 0%      6.62µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                        117ms ± 0%        62ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     1.44s ± 0%       0.24s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              92.1ms ± 0%      70.3ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     196ms ± 0%       115ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     1.23s ± 0%       0.21s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       1.06ms ± 0%      0.88ms ± 0%

name                                                                   old alloc/op   new alloc/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                        29.5MB ± 0%      26.9MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                     46.8MB ± 0%     251.5MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"               29.5MB ± 0%      22.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     46.8MB ± 0%      23.9MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                     10.3kB ± 0%  138535.2kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        5.54kB ± 0%      7.09kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                       39.1MB ± 0%      28.5MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     287MB ± 0%       253MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              34.3MB ± 0%      23.9MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    51.6MB ± 0%      25.5MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     144MB ± 0%       139MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       6.43kB ± 0%      8.66kB ± 0%

name                                                                   old allocs/op  new allocs/op
Querier/Head/labelValuesWithMatchers/i_with_n="1"                          104k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                       204k ± 0%        600k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"                 104k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"       204k ± 0%        500k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                       66.0 ± 0%       255.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                          61.0 ± 0%       205.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                         304k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                     5.20M ± 0%       0.70M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"                204k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"      304k ± 0%        600k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                     3.00M ± 0%       0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                         61.0 ± 0%       247.0 ± 0%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Don't expand postings to intersect them

Using a min heap we can check whether matched postings intersect with
each one of the label values postings. This avoid expanding postings
(and thus having all of them in memory at any point).

Slightly slower than the expanding postings version for some cases, but
definitely pays the price once the cardinality grows.

Still offers 10x latency improvement where previous latencies were
reaching 1s.

Benchmark results:

name \ time/op                                                         old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                       157ms ± 0%        48ms ± 0%              110ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                    1.80s ± 0%       0.46s ± 0%              0.18s ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"              144ms ± 0%        57ms ± 0%              125ms ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    304ms ± 0%       111ms ± 0%              177ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                    761ms ± 0%       164ms ± 0%              134ms ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                      6.11µs ± 0%      6.62µs ± 0%             4.29µs ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                      117ms ± 0%        62ms ± 0%              120ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   1.44s ± 0%       0.24s ± 0%              0.15s ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"            92.1ms ± 0%      70.3ms ± 0%            125.4ms ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"   196ms ± 0%       115ms ± 0%              170ms ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   1.23s ± 0%       0.21s ± 0%              0.14s ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                     1.06ms ± 0%      0.88ms ± 0%             0.92ms ± 0%

name \ alloc/op                                                        old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                      29.5MB ± 0%      26.9MB ± 0%             19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                   46.8MB ± 0%     251.5MB ± 0%             36.3MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"             29.5MB ± 0%      22.3MB ± 0%             19.1MB ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"   46.8MB ± 0%      23.9MB ± 0%             20.7MB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                   10.3kB ± 0%  138535.2kB ± 0%              6.4kB ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                      5.54kB ± 0%      7.09kB ± 0%             4.30kB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                     39.1MB ± 0%      28.5MB ± 0%             20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   287MB ± 0%       253MB ± 0%               38MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"            34.3MB ± 0%      23.9MB ± 0%             20.7MB ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"  51.6MB ± 0%      25.5MB ± 0%             22.3MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   144MB ± 0%       139MB ± 0%                0MB ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                     6.43kB ± 0%      8.66kB ± 0%             5.86kB ± 0%

name \ allocs/op                                                       old.txt      intersect.txt    intersect_noexpand.txt
Querier/Head/labelValuesWithMatchers/i_with_n="1"                        104k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="^.+$"                     204k ± 0%        600k ± 0%               400k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",j!="foo"               104k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"     204k ± 0%        500k ± 0%               300k ± 0%
Querier/Head/labelValuesWithMatchers/n_with_j!="foo"                     66.0 ± 0%       255.0 ± 0%              139.0 ± 0%
Querier/Head/labelValuesWithMatchers/n_with_i="1"                        61.0 ± 0%       205.0 ± 0%               87.0 ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1"                       304k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="^.+$"                   5.20M ± 0%       0.70M ± 0%              0.50M ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",j!="foo"              204k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/i_with_n="1",i=~"^.*$",j!="foo"    304k ± 0%        600k ± 0%               400k ± 0%
Querier/Block/labelValuesWithMatchers/n_with_j!="foo"                   3.00M ± 0%       0.00M ± 0%              0.00M ± 0%
Querier/Block/labelValuesWithMatchers/n_with_i="1"                       61.0 ± 0%       247.0 ± 0%              129.0 ± 0%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Apply comment suggestions from the code review

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* Change else { if } to else if

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Remove sorting of label values

We were not sorting them before, so no need to sort them now

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-12-28 15:59:03 +01:00