prometheus/storage
Bryan Boreham 60804c5a09
remote_write: reduce blocking from garbage-collect of series (#9109)
* Refactor: pass segment-reading function as param

To allow a different implementation to be used when garbage-collecting.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* remote_write: reduce blocking from GC of series

Add a method `UpdateSeriesSegment()` which is used together with
`SeriesReset()` to garbage-collect old series. This allows us to
split the lock around queueManager series data and avoid blocking
`Append()` while reading series from the last checkpoint.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Cosmetic: review feedback on comments

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* remote-write benchmark: include GC of series

Reduce the total number of samples per iteration from 5000*5000
(25 million) which is too big for my laptop, to 1*10000.

Extend `createTimeseries()` to add additional labels, so that the
queue manager is doing more realistic work.

Move the Append() call to a background goroutine - this works because
TestWriteClient uses a WaitGroup to signal completion.

Call `StoreSeries()` and `SeriesReset()` while adding samples, to
simulate the garbage-collection that wal.Watcher does.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Change BenchmarkSampleDelivery to call UpdateSeriesSegment

This matches what Watcher.garbageCollectSeries() is doing now

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-07-27 13:21:48 -07:00
..
remote remote_write: reduce blocking from garbage-collect of series (#9109) 2021-07-27 13:21:48 -07:00
buffer.go storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005) 2020-03-24 20:15:47 +00:00
buffer_test.go Optimized vector selector 2021-03-11 14:32:56 +01:00
fanout.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
fanout_test.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
generic.go tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069) 2020-07-31 16:03:02 +01:00
interface.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
lazy.go tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069) 2020-07-31 16:03:02 +01:00
memoized_iterator.go Update storage/memoized_iterator.go 2021-03-15 09:27:20 +01:00
memoized_iterator_test.go Optimized vector selector 2021-03-11 14:32:56 +01:00
merge.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
merge_test.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
noop.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
secondary.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
series.go expose seriesToChunkEncoder (#8845) 2021-05-19 13:01:35 +01:00