Commit graph

26 commits

Author SHA1 Message Date
Brian Brazil 74094947ea effect -> affect 2016-05-12 15:14:48 +01:00
eliothedeman 1543ef92b2 Adds holt-winters query function 2016-03-28 15:42:27 -04:00
beorn7 dad302144d Make a naked return less naked 2016-03-09 15:06:00 +01:00
beorn7 c740789ce3 Improve predict_linear
Fixes https://github.com/prometheus/prometheus/issues/1401

This remove the last (and in fact bogus) use of BoundaryValues.

Thus, a whole lot of unused (and arguably sub-optimal / ugly) code can
be removed here, too.
2016-02-25 12:10:55 +01:00
beorn7 0e202dacb4 Streamline series iterator creation
This will fix issue #1035 and will also help to make issue #1264 less
bad.

The fundamental problem in the current code:

In the preload phase, we quite accurately determine which chunks will
be used for the query being executed. However, in the subsequent step
of creating series iterators, the created iterators are referencing
_all_ in-memory chunks in their series, even the un-pinned ones. In
iterator creation, we copy a pointer to each in-memory chunk of a
series into the iterator. While this creates a certain amount of
allocation churn, the worst thing about it is that copying the chunk
pointer out of the chunkDesc requires a mutex acquisition. (Remember
that the iterator will also reference un-pinned chunks, so we need to
acquire the mutex to protect against concurrent eviction.) The worst
case happens if a series doesn't even contain any relevant samples for
the query time range. We notice that during preloading but then we
will still create a series iterator for it. But even for series that
do contain relevant samples, the overhead is quite bad for instant
queries that retrieve a single sample from each series, but still go
through all the effort of series iterator creation. All of that is
particularly bad if a series has many in-memory chunks.

This commit addresses the problem from two sides:

First, it merges preloading and iterator creation into one step,
i.e. the preload call returns an iterator for exactly the preloaded
chunks.

Second, the required mutex acquisition in chunkDesc has been greatly
reduced. That was enabled by a side effect of the first step, which is
that the iterator is only referencing pinned chunks, so there is no
risk of concurrent eviction anymore, and chunks can be accessed
without mutex acquisition.

To simplify the code changes for the above, the long-planned change of
ValueAtTime to ValueAtOrBefore time was performed at the same
time. (It should have been done first, but it kind of accidentally
happened while I was in the middle of writing the series iterator
changes. Sorry for that.) So far, we actively filtered the up to two
values that were returned by ValueAtTime, i.e. we invested work to
retrieve up to two values, and then we invested more work to throw one
of them away.

The SeriesIterator.BoundaryValues method can be removed once #1401 is
fixed. But I really didn't want to load even more changes into this
PR.

Benchmarks:

The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times
faster) and allocate 95% fewer bytes. The reason for that is that the
benchmark reads one sample after another from the time series and
creates a new series iterator for each sample read.

To find out how much these improvements matter in practice, I have
mirrored a beefy Prometheus server at SoundCloud that suffers from
both issues #1035 and #1264. To reach steady state that would be
comparable, the server needs to run for 15d. So far, it has run for
1d. The test server currently has only half as many memory time series
and 60% of the memory chunks the main server has. The 90th percentile
rule evaluation cycle time is ~11s on the main server and only ~3s on
the test server. However, these numbers might get much closer over
time.

In addition to performance improvements, this commit removes about 150
LOC.
2016-02-19 16:24:38 +01:00
Brian Brazil c77c3a8c56 promql: Limit extrapolation of delta/rate/increase
The new implementation detects the start and end of a series by
looking at the average sample interval within the range. If the first
(last) sample in the range is more than 1.1*interval distant from the
beginning (end) of the range, it is considered the first (last) sample
of the series as a whole, and extrapolation is limited to half the
interval (rather than all the way to the beginning (end) of the
range). In addition, if the extrapolated starting point of a counter
(where it is zero) is within the range, it is used as the starting
point of the series.

Fixes #581
2016-01-08 15:32:43 +01:00
Brian Brazil bac1f28cad Similar to topk/bottomk, have sort/sort_desc put NaN at end.
This makes topk and bottomk consistent with the sorting functions,
as per #1271.
2015-12-31 14:52:48 +00:00
Brian Brazil 88ca82304c Make topk/bottomk prefer returning real numbers over NaN. 2015-12-22 13:53:43 +00:00
James Sanford 5b53262b7a promql: Add clamp_max/clamp_min functions. 2015-11-26 13:38:06 -08:00
Brian Brazil 5740a8fade promql: Remove deprecated 2nd argument to delta()
This change is breaking, use increase() instead.

I'm not cleaning up the function in this PR, as my solution to #581 will
rewrite and simplify increase/rate/delta.
2015-10-10 15:41:23 +01:00
Brian Brazil f08abdb48b promql: Add irate() function
irate is a rate function that only looks at the most
recent two data points, and calucaltes a per-second value
from that. This produces much more granular graphs for
fast moving data, and works sanely across many scrape intervals.

It doesn't do so well for slowly moving data.
2015-10-09 21:44:35 +01:00
Fabian Reinartz e005f939fd Fix scalar construction in function 2015-09-18 16:49:32 +02:00
Fabian Reinartz eca41f5319 Run gofmt 2015-09-16 14:33:12 +02:00
Brian Brazil 9b382647b5 Remove optional vector() 2nd argument 2015-09-13 09:13:22 +01:00
Brian Brazil 69f5fa0c1e promql: Add vector function.
Currently the only way to convert a scalar to a vector is to
use absent(), which isn't very clean. This adds a vector()
function that's the inverse of scalar() and lets your optionally
set labels.

Example usage would be
vector(time() % 86400) < 3600
to filter to only the first hour of the day.
2015-09-11 12:09:34 +01:00
Julius Volz 61c42c8da0 Change relabel_replace() to do full-string matches.
THIS IS A BREAKING CHANGE.

Fixes part of https://github.com/prometheus/prometheus/issues/996
2015-09-01 15:49:28 +02:00
Fabian Reinartz d6b8da8d43 Switch promql types to common/model 2015-08-25 13:49:14 +02:00
Fabian Reinartz 1535ef1457 Replace metric.SamplePair with model.SamplePair 2015-08-22 14:52:35 +02:00
Fabian Reinartz 306e8468a0 Switch from client_golang/model to common/model 2015-08-21 13:33:38 +02:00
Julius Volz 27ed874358 Implement label_replace()
Implements part of https://github.com/prometheus/prometheus/issues/959.
2015-08-18 14:20:07 +02:00
Brian Brazil d6a80c2b76 promql: Add support for predict_linear(my_timeseries[1h], 7200)
This will give a prediction for the value of my_timeseries in 2 hours,
based on the last hour of data.
2015-08-05 15:16:49 +01:00
Brian Brazil f34de493d5 Add increase() function, to replace delta(..., 1).
This calculates how much a counter increases over
a given period of time, which is the area under the curve
of it's rate.

increase(x[5m]) is equivilent to rate(x[5m]) * 300.
2015-05-26 22:49:21 +01:00
Julius Volz d44a89c6e8 Implement changes() function.
changes() takes a range vector and returns the number of times a value
has changed in the given time window for each time series as an instant
vector.
2015-05-26 19:06:22 +02:00
Julius Volz 6f33ed9e59 Add resets() function to count counter resets.
resets() returns for every range vector element how many counter
resets there have been in the specified range.
2015-05-26 17:56:52 +02:00
Fabian Reinartz 5602328c7c Refactor query evaluation.
This copies the evaluation logic from the current rules/ package.
The new engine handles the execution process from query string to final result.
It provides query timeout and cancellation and general flexibility for
future changes.

functions.go: Add evaluation implementation. Slight changes to in/out data but
	not to the processing logic.
quantile.go: No changes.
analyzer.go: No changes.
engine.go: Actually new part. Mainly consists of evaluation methods
	which were not changed.
setup_test.go: Copy of rules/helpers_test.go to setup test storage.
promql_test.go: Copy of rules/rules_test.go.
2015-04-28 14:19:05 +02:00
Fabian Reinartz 32b7595c47 Create promql package with lexer/parser.
This commit creates a (so far unused) package. It contains the a custom
lexer/parser for the query language.

ast.go: New AST that interacts well with the parser.
lex.go: Custom lexer (new).
lex_test.go: Lexer tests (new).
parse.go: Custom parser (new).
parse_test.go: Parser tests (new).
functions.go: Changed function type, dummies for parser testing (barely changed/dummies).
printer.go: Adapted from rules/ and adjusted to new AST (mostly unchanged, few additions).
2015-04-23 16:04:50 +02:00