This commit is the first of several and should not be regarded as the
desired end state for these cleanups. What this one does it, however,
is wrap the query index writing behind an interface type that can be
injected into the storage stack and have its lifecycle managed
separately as needed. It also would mean we can swap out underlying
implementations to support remote indexing, buffering, no-op indexing
very easily.
In the future, most of the individual index interface members in the
tiered storage will go away in favor of agents that can query and
resolve what they need from the datastore without the user knowing
how and why they work.
There are too many parameters to constructing a LevelDB storage
instance for a construction method, so I've opted to take an
idiomatic approach of embedding them in a struct for easier
mediation and versioning.
When samples get flushed to disk, they lose sub-second precision anyways. By
already dropping sub-second precision, data fetched from memory vs. disk will
behave the same. Later, we should consider also storing a more compact
representation than time.Time in memory if we're not going to use its full
precision.
Current series always get watermarks written out upon append now. This
drops support for old series without any watermarks by always reporting
them as too old (stale) during queries.
This also short-circuits optimize() for now, since it is complex to implement
for the new operator, and ops generated by the query layer already fulfill the
needed invariants. We should still investigate later whether to completely
delete operator optimization code or extend it to support
getValueRangeAtIntervalOp operators.
An design question was open for me in the beginning was whether to
serialize other types to disk, but Protocol Buffers quickly won out,
which allows us to drop support for other types. This is a good
start to cleaning up a lot of cruft in the storage stack and
can let us eventually decouple the various moving parts into
separate subsystems for easier reasoning.
This commit is not strictly required, but it is a start to making
the rest a lot more enjoyable to interact with.
This adds timers around several query-relevant code blocks. For now, the
query timer stats are only logged for queries initiated through the UI.
In other cases (rule evaluations), the stats are simply thrown away.
My hope is that this helps us understand where queries spend time,
especially in cases where they sometimes hang for unusual amounts of
time.
This commit conditionalizes the creation of the diskFrontier and
seriesFrontier along with the iterator such that they are provisioned
once something is actually required from disk.
This is mainly a small performance improvement, since we skip past the last
extracted time immediately if it was also the last sample in the chunk, instead
of trying to extract non-existent values before the chunk end again and again
and only gradually approaching the end of the chunk.
The current behavior only adds those samples to the view that are extracted by
the last pass of the last processed op and throws other ones away. This is a
bug. We need to append all samples that are extracted by each op pass.
This also makes view.appendSamples() take an array of samples.
The previous implementation spawned N goroutines to group samples
together and would not start work until the semaphore unblocked.
While this didn't leak, it polluted the scheduling space. Thusly,
the routine only starts after a semaphore has been acquired.
The one-off keys have been replaced with ``model.LabelPair``, which is
indexable. The performance impact is negligible, but it represents
a cognitive simplification.
The reality is that if we ever try to encode a Protocol Buffer and it
fails, it's likely that such an error is ultimately not a runtime error
and should be fixed forthwith. Thusly, we should rename
``Encoder.Encode`` to ``Encoder.MustEncode`` and drop the error return
value.
Some users of GetMetricForFingerprint() end up modifying the returned metric
labelset. Since the memory storage's implementation of
GetMetricForFingerprint() returned a pointer to the metric (and maps are
reference types anyways), the external mutation propagated back into the memory
storage.
The fix is to make a copy of the metric before returning it.
- only the data extracted in the last loop iteration of ExtractSamples() was
emitted as output
- if e.g. op interval < sample interval, there were situations where the same
sample was added multiple times to the output
This commit updates the documentation, Makefiles, formatting, and
code semantics to support the 1.1. runtime, which includes ...
1. ``make advice``,
2. ``make format``, and
3. ``go fix`` on various targets.
This commit simplifies the way that compactions across a database's
keyspace occur due to reading the LevelDB internals. Secondarily it
introduces the database size estimation mechanisms.
Include database health and help interfaces.
Add database statistics; remove status goroutines.
This commit kills the use of Go routines to expose status throughout
the web components of Prometheus. It also dumps raw LevelDB status
on a separate /databases endpoint.
This commit simplifies the way that compactions across a database's
keyspace occur due to reading the LevelDB internals. Secondarily it
introduces the database size estimation mechanisms.
This commit introduces the long-tail deletion mechanism, which will
automatically cull old sample values. It is an acceptable
hold-over until we get a resampling pipeline implemented.
Kill legacy OS X documentation, too.
This does two things:
1) Make TieredStorage.AppendSamples() write directly to memory instead of
buffering to a channel first. This is needed in cases where a rule might
immediately need the data generated by a previous rule.
2) Replace the single storage mutex by two new ones:
- memoryMutex - needs to be locked at any time that two concurrent
goroutines could be accessing (via read or write) the
TieredStorage memoryArena.
- memoryDeleteMutex - used to prevent any deletion of samples from
memoryArena as long as renderView is running and
assembling data from it.
The LevelDB disk storage does not need to be protected by a mutex when
rendering a view since renderView works off a LevelDB snapshot.
The rationale against adding memoryMutex directly to the memory storage: taking
a mutex does come with a small inherent time cost, and taking it is only
required in few places. In fact, no locking is required for the memory storage
instance which is part of a view (and not the TieredStorage).
This commit extracts the model.Values truncation behavior into the actual
tiered storage, which uses it and behaves in a peculiar way—notably the
retention of previous elements if the chunk were to ever go empty. This is
done to enable interpolation between sparse sample values in the evaluation
cycle. Nothing necessarily new here—just an extraction.
Now, the model.Values TruncateBefore functionality would do what a user
would expect without any surprises, which is required for the
DeletionProcessor, which may decide to split a large chunk in two if it
determines that the chunk contains the cut-off time.
This commit introduces three background compactors, which compact
sparse samples together.
1. Older than five minutes is grouped together into chunks of 50 every 30
minutes.
2. Older than 60 minutes is grouped together into chunks of 250 every 50
minutes.
3. Older than one day is grouped together into chunks of 5000 every 70
minutes.
This commit drops the Storage interface and just replaces it with a
publicized TieredStorage type. Storage had been anticipated to be
used as a wrapper for testability but just was not used due to
practicality. Merely overengineered. My bad. Anyway, we will
eventually instantiate the TieredStorage dependencies in main.go and
pass them in for more intelligent lifecycle management.
These changes will pave the way for managing the curators without
Law of Demeter violations.
