This commit introduces the long-tail deletion mechanism, which will
automatically cull old sample values. It is an acceptable
hold-over until we get a resampling pipeline implemented.
Kill legacy OS X documentation, too.
This does two things:
1) Make TieredStorage.AppendSamples() write directly to memory instead of
buffering to a channel first. This is needed in cases where a rule might
immediately need the data generated by a previous rule.
2) Replace the single storage mutex by two new ones:
- memoryMutex - needs to be locked at any time that two concurrent
goroutines could be accessing (via read or write) the
TieredStorage memoryArena.
- memoryDeleteMutex - used to prevent any deletion of samples from
memoryArena as long as renderView is running and
assembling data from it.
The LevelDB disk storage does not need to be protected by a mutex when
rendering a view since renderView works off a LevelDB snapshot.
The rationale against adding memoryMutex directly to the memory storage: taking
a mutex does come with a small inherent time cost, and taking it is only
required in few places. In fact, no locking is required for the memory storage
instance which is part of a view (and not the TieredStorage).
This commit extracts the model.Values truncation behavior into the actual
tiered storage, which uses it and behaves in a peculiar way—notably the
retention of previous elements if the chunk were to ever go empty. This is
done to enable interpolation between sparse sample values in the evaluation
cycle. Nothing necessarily new here—just an extraction.
Now, the model.Values TruncateBefore functionality would do what a user
would expect without any surprises, which is required for the
DeletionProcessor, which may decide to split a large chunk in two if it
determines that the chunk contains the cut-off time.
This commit reduces the general compile time dependencies to omit
the Protocol Buffer compiler and the Go Protocol Buffer generator
tool. The build steps to furnish them still remain, but they can
optionally be called if data.proto or config.proto are under work.
This also removes the now obsolete scalar count() function and corrects the
expressions test naming (broken in
2202cd71c9 (L6R59))
so that the expression tests will actually run.
This commit introduces three background compactors, which compact
sparse samples together.
1. Older than five minutes is grouped together into chunks of 50 every 30
minutes.
2. Older than 60 minutes is grouped together into chunks of 250 every 50
minutes.
3. Older than one day is grouped together into chunks of 5000 every 70
minutes.
This commit drops the Storage interface and just replaces it with a
publicized TieredStorage type. Storage had been anticipated to be
used as a wrapper for testability but just was not used due to
practicality. Merely overengineered. My bad. Anyway, we will
eventually instantiate the TieredStorage dependencies in main.go and
pass them in for more intelligent lifecycle management.
These changes will pave the way for managing the curators without
Law of Demeter violations.
