prometheus/storage
Robert Fratto a09465baee
storage/remote: disable resharding during active retry backoffs (#13562)
* storage/remote: disable resharding during active retry backoffs

Today, remote_write reshards based on pure throughput. This is
problematic if throughput has been diminished because of HTTP 429s;
increasing the number of shards due to backpressure will only exacerbate
the problem.

This commit disables resharding for twice the retry backoff, ensuring
that resharding will never occur during an active backoff, and that
resharding does not become enabled again until enough time has elapsed
to allow any pending requests to be retried.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* storage/remote: test that resharding is disabled on retry

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* storage/remote: address review feedback

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

* storage/remote: track time where resharding initially got disabled

This change introduces a second atomic int64 to roughly track when
resharding got disabled. This int64 is only updated after updating the
disabled timestamp if resharding was previously enabled.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

---------

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
2024-02-28 14:28:39 -08:00
..
remote storage/remote: disable resharding during active retry backoffs (#13562) 2024-02-28 14:28:39 -08:00
buffer.go Optimize histogram iterators (#13340) 2024-01-23 17:02:14 +01:00
buffer_test.go Optimize histogram iterators (#13340) 2024-01-23 17:02:14 +01:00
fanout.go storage: skip merging when no remote storage configured 2024-01-18 17:50:06 +00:00
fanout_test.go golangci-lint: enable testifylint linter (#13254) 2023-12-07 11:35:01 +00:00
generic.go Add warnings (and annotations) to PromQL query results (#12152) 2023-09-14 18:57:31 +02:00
interface.go Add ShardedPostings() support to TSDB (#10421) 2024-01-29 11:57:27 +00:00
lazy.go Add warnings (and annotations) to PromQL query results (#12152) 2023-09-14 18:57:31 +02:00
memoized_iterator.go Optimize histogram iterators (#13340) 2024-01-23 17:02:14 +01:00
memoized_iterator_test.go golangci-lint: enable testifylint linter (#13254) 2023-12-07 11:35:01 +00:00
merge.go Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 2024-02-28 14:54:53 +01:00
merge_test.go Optimize histogram iterators (#13340) 2024-01-23 17:02:14 +01:00
noop.go Add warnings (and annotations) to PromQL query results (#12152) 2023-09-14 18:57:31 +02:00
secondary.go Add warnings (and annotations) to PromQL query results (#12152) 2023-09-14 18:57:31 +02:00
series.go Optimize histogram iterators (#13340) 2024-01-23 17:02:14 +01:00
series_test.go golangci-lint: enable testifylint linter (#13254) 2023-12-07 11:35:01 +00:00