mirror of
https://github.com/prometheus/prometheus.git
synced 2024-09-21 00:07:36 -07:00
0e202dacb4
This will fix issue #1035 and will also help to make issue #1264 less bad. The fundamental problem in the current code: In the preload phase, we quite accurately determine which chunks will be used for the query being executed. However, in the subsequent step of creating series iterators, the created iterators are referencing _all_ in-memory chunks in their series, even the un-pinned ones. In iterator creation, we copy a pointer to each in-memory chunk of a series into the iterator. While this creates a certain amount of allocation churn, the worst thing about it is that copying the chunk pointer out of the chunkDesc requires a mutex acquisition. (Remember that the iterator will also reference un-pinned chunks, so we need to acquire the mutex to protect against concurrent eviction.) The worst case happens if a series doesn't even contain any relevant samples for the query time range. We notice that during preloading but then we will still create a series iterator for it. But even for series that do contain relevant samples, the overhead is quite bad for instant queries that retrieve a single sample from each series, but still go through all the effort of series iterator creation. All of that is particularly bad if a series has many in-memory chunks. This commit addresses the problem from two sides: First, it merges preloading and iterator creation into one step, i.e. the preload call returns an iterator for exactly the preloaded chunks. Second, the required mutex acquisition in chunkDesc has been greatly reduced. That was enabled by a side effect of the first step, which is that the iterator is only referencing pinned chunks, so there is no risk of concurrent eviction anymore, and chunks can be accessed without mutex acquisition. To simplify the code changes for the above, the long-planned change of ValueAtTime to ValueAtOrBefore time was performed at the same time. (It should have been done first, but it kind of accidentally happened while I was in the middle of writing the series iterator changes. Sorry for that.) So far, we actively filtered the up to two values that were returned by ValueAtTime, i.e. we invested work to retrieve up to two values, and then we invested more work to throw one of them away. The SeriesIterator.BoundaryValues method can be removed once #1401 is fixed. But I really didn't want to load even more changes into this PR. Benchmarks: The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times faster) and allocate 95% fewer bytes. The reason for that is that the benchmark reads one sample after another from the time series and creates a new series iterator for each sample read. To find out how much these improvements matter in practice, I have mirrored a beefy Prometheus server at SoundCloud that suffers from both issues #1035 and #1264. To reach steady state that would be comparable, the server needs to run for 15d. So far, it has run for 1d. The test server currently has only half as many memory time series and 60% of the memory chunks the main server has. The 90th percentile rule evaluation cycle time is ~11s on the main server and only ~3s on the test server. However, these numbers might get much closer over time. In addition to performance improvements, this commit removes about 150 LOC.
97 lines
4.5 KiB
Go
97 lines
4.5 KiB
Go
// Copyright 2014 The Prometheus Authors
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package local
|
|
|
|
import (
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
"github.com/prometheus/common/model"
|
|
|
|
"github.com/prometheus/prometheus/storage/metric"
|
|
)
|
|
|
|
// Storage ingests and manages samples, along with various indexes. All methods
|
|
// are goroutine-safe. Storage implements storage.SampleAppender.
|
|
type Storage interface {
|
|
prometheus.Collector
|
|
// Append stores a sample in the Storage. Multiple samples for the same
|
|
// fingerprint need to be submitted in chronological order, from oldest
|
|
// to newest. When Append has returned, the appended sample might not be
|
|
// queryable immediately. (Use WaitForIndexing to wait for complete
|
|
// processing.) The implementation might remove labels with empty value
|
|
// from the provided Sample as those labels are considered equivalent to
|
|
// a label not present at all.
|
|
Append(*model.Sample) error
|
|
// NeedsThrottling returns true if the Storage has too many chunks in memory
|
|
// already or has too many chunks waiting for persistence.
|
|
NeedsThrottling() bool
|
|
// NewPreloader returns a new Preloader which allows preloading and pinning
|
|
// series data into memory for use within a query.
|
|
NewPreloader() Preloader
|
|
// MetricsForLabelMatchers returns the metrics from storage that satisfy the given
|
|
// label matchers. At least one label matcher must be specified that does not
|
|
// match the empty string.
|
|
MetricsForLabelMatchers(...*metric.LabelMatcher) map[model.Fingerprint]metric.Metric
|
|
// LastSamplePairForFingerprint returns the last sample pair for the
|
|
// provided fingerprint. If the respective time series does not exist or
|
|
// has an evicted head chunk, nil is returned.
|
|
LastSamplePairForFingerprint(model.Fingerprint) *model.SamplePair
|
|
// Get all of the label values that are associated with a given label name.
|
|
LabelValuesForLabelName(model.LabelName) model.LabelValues
|
|
// Get the metric associated with the provided fingerprint.
|
|
MetricForFingerprint(model.Fingerprint) metric.Metric
|
|
// Drop all time series associated with the given fingerprints.
|
|
DropMetricsForFingerprints(...model.Fingerprint)
|
|
// Run the various maintenance loops in goroutines. Returns when the
|
|
// storage is ready to use. Keeps everything running in the background
|
|
// until Stop is called.
|
|
Start() error
|
|
// Stop shuts down the Storage gracefully, flushes all pending
|
|
// operations, stops all maintenance loops,and frees all resources.
|
|
Stop() error
|
|
// WaitForIndexing returns once all samples in the storage are
|
|
// indexed. Indexing is needed for FingerprintsForLabelMatchers and
|
|
// LabelValuesForLabelName and may lag behind.
|
|
WaitForIndexing()
|
|
}
|
|
|
|
// SeriesIterator enables efficient access of sample values in a series. Its
|
|
// methods are not goroutine-safe. A SeriesIterator iterates over a snapshot of
|
|
// a series, i.e. it is safe to continue using a SeriesIterator after or during
|
|
// modifying the corresponding series, but the iterator will represent the state
|
|
// of the series prior the modification.
|
|
type SeriesIterator interface {
|
|
// Gets the value that is closest before the given time. In case a value
|
|
// exist at precisely the given time, that value is returned. If no
|
|
// applicable value exists, a SamplePair with timestamp model.Earliest
|
|
// and value 0.0 is returned.
|
|
ValueAtOrBeforeTime(model.Time) model.SamplePair
|
|
// Gets the boundary values of an interval: the first and last value
|
|
// within a given interval.
|
|
BoundaryValues(metric.Interval) []model.SamplePair
|
|
// Gets all values contained within a given interval.
|
|
RangeValues(metric.Interval) []model.SamplePair
|
|
}
|
|
|
|
// A Preloader preloads series data necessary for a query into memory, pins it
|
|
// until released via Close(), and returns an iterator for the pinned data. Its
|
|
// methods are generally not goroutine-safe.
|
|
type Preloader interface {
|
|
PreloadRange(
|
|
fp model.Fingerprint,
|
|
from model.Time, through model.Time,
|
|
) (SeriesIterator, error)
|
|
// Close unpins any previously requested series data from memory.
|
|
Close()
|
|
}
|