Commit graph

4586 commits

Author SHA1 Message Date
Björn Rabenstein f8cb25e932 Merge pull request #559 from prometheus/beorn7/fix
Fix the embarrassing bug introduced in commit 0851945.
2015-02-26 19:25:57 +01:00
beorn7 edd716e63c Fix the embarrassing bug introduced in commit 0851945.
In that commit, the 'maintainSeries' call was accidentally removed.

This commit refactors things a bit so that there is now a clean
'maintainMemorySeries' and a 'maintainArchivedSeries' call.

Straighten the nomenclature a bit (consistently use 'drop' for
chunks and 'purge' for series/metrics).

Remove the annoying 'Completed maintenance sweep through archived
fingerprints' message if there were no archived fingerprints to do
maintenance on.
2015-02-26 18:30:33 +01:00
Brian Brazil 93edd930a6 Better handling of edge conditions in humanizeDuration 2015-02-26 17:30:03 +00:00
Björn Rabenstein 041aa59623 Merge pull request #553 from prometheus/beorn7/release
Version 0.11.0!
2015-02-23 20:21:59 +01:00
beorn7 5f8d7a9dbd Cut release 0.11.0. 2015-02-23 20:21:00 +01:00
beorn7 a4f6edf496 Vendor client_golang v0.2.0. 2015-02-23 18:11:42 +01:00
Björn Rabenstein 136cf661b7 Merge pull request #545 from prometheus/beorn7/quantile
Add the histogram_quantile function.
2015-02-23 15:30:24 +01:00
beorn7 1a61bcae07 Fix plural of 'histogram'.
Actually, 'histogram' is Ancient Greek and 3rd declension... ;-)
2015-02-23 15:29:26 +01:00
Julius Volz 03191cfc8e Merge pull request #550 from atombender/fix_makefile
Makefile: Calculate $(GOROOT), etc. relative to Makefile
2015-02-23 12:42:22 +01:00
Alexander Staubo 8a47f8a44d Makefile: Calculate $(GOROOT), etc. relative to Makefile instead of using $(PWD), which is not always available (eg., in debuild). 2015-02-22 23:14:11 -05:00
beorn7 9827e1ccec Update vendoring of client_golang. 2015-02-22 01:04:52 +01:00
beorn7 17443d288b Avoid copying of the COWMetric if we already have the metric available. 2015-02-22 01:04:52 +01:00
beorn7 9e7c3e3bcd Add the histogram_quantile function.
Since we are now getting really deep into floating point calculation,
the tests had to take into account the precision loss. Since the rule
tests are based on direct line matching in the output, implementing
the "almost equal" semantics was pretty cumbersome, but here we are.
2015-02-22 01:04:51 +01:00
Julius Volz 452c88964a Merge pull request #546 from prometheus/fix-label-grouping
Fix aggregation grouping key calculation.
2015-02-21 17:55:26 +01:00
Julius Volz 42601acfde Replace labelsToKey() with metric Fingerprint (fixes grouping bug). 2015-02-21 17:45:47 +01:00
Julius Volz 7fefccd929 Write() directly into hash and use model.SeparatorByte. 2015-02-21 17:19:13 +01:00
Julius Volz 645cf57bed Fix aggregation grouping key calculation. 2015-02-21 14:05:50 +01:00
Julius Volz 79834edcb5 Merge pull request #544 from q3k/feature/hide-instance-auth
Hide HTTP auth parts from instance label
2015-02-20 17:34:04 +01:00
Sergiusz 'q3k' Bazański 0d0bb3c030 Change instance identifiers to be host:port
This changes the PublicURL function into InstanceIdentifier, which now
returns a simple <host>:<port> string instead of a full URL.
2015-02-20 16:21:13 +01:00
Sergiusz 'q3k' Bazański bb69a3d284 Hide HTTP auth parts from URL
This  instroduces an extra function in the Target interface (PublicURL)
which is used to populate the instance field in scraped metrics.
2015-02-19 18:58:47 +01:00
Brian Brazil 79bf5a278e Merge pull request #543 from kormat/cpucount
Add cpu count to rhs table on node cpu console
2015-02-19 11:19:44 +00:00
Stephen Shirley 05a746bf95 Add cpu count to rhs table
Also fix formatting of bounded values to be more readable.
2015-02-19 12:07:52 +01:00
Julius Volz 2dd70b8c13 Merge pull request #542 from prometheus/I-cant-believe-its-not-federation
Proof of concept for federation via console templates.
2015-02-19 00:23:39 +01:00
Brian Brazil 5adcec3018 Proof of concept for federation via console templates.
This exposes samples via the console templates in the
text exposition format, which the parser will fall back to.

This is not a proper federation solution, but should tide us
over for now. Extenions could include passing in a query or queries in
the url parameters.
2015-02-18 23:21:33 +00:00
Julius Volz c069c0dafa Merge pull request #536 from prometheus/offset
Implement offset operator.
2015-02-18 14:48:56 +01:00
Brian Brazil 40b0a1ac0d Merge pull request #537 from kormat/master
Fix available memory calculation.
2015-02-18 10:36:08 +00:00
Stephen Shirley fbcbb6a635 Fix available memory calculation.
Also account for buffers, making the value match the output of free(1)
2015-02-18 11:26:29 +01:00
Julius Volz 15b2b5aa66 Add tests for invalid uses of "offset". 2015-02-18 02:56:40 +01:00
Julius Volz 67e20acc6c Lower-case some package-internal names. 2015-02-18 02:45:54 +01:00
Julius Volz 72d7b325a1 Implement offset operator.
This allows changing the time offset for individual instant and range
vectors in a query.

For example, this returns the value of `foo` 5 minutes in the past
relative to the current query evaluation time:

    foo offset 5m

Note that the `offset` modifier always needs to follow the selector
immediately. I.e. the following would be correct:

    sum(foo offset 5m) // GOOD.

While the following would be *incorrect*:

    sum(foo) offset 5m // INVALID.

The same works for range vectors. This returns the 5-minutes-rate that
`foo` had a week ago:

    rate(foo[5m] offset 1w)

This change touches the following components:

* Lexer/parser: additions to correctly parse the new `offset`/`OFFSET`
  keyword.
* AST: vector and matrix nodes now have an additional `offset` field.
  This is used during their evaluation to adjust query and result times
  appropriately.
* Query analyzer: now works on separate sets of ranges and instants per
  offset. Isolating different offsets from each other completely in this
  way keeps the preloading code relatively simple.

No storage engine changes were needed by this change.

The rules tests have been changed to not probe the internal
implementation details of the query analyzer anymore (how many instants
and ranges have been preloaded). This would also become too cumbersome
to test with the new model, and measuring the result of the query should
be sufficient.

This fixes https://github.com/prometheus/prometheus/issues/529
This fixed https://github.com/prometheus/promdash/issues/201
2015-02-18 02:41:27 +01:00
Julius Volz 79a4a6d8e8 Merge pull request #532 from prometheus/beorn7/ingestion-tweaks
Improve persisting chunks to disk.
2015-02-17 17:31:00 +01:00
Julius Volz b0c8b56603 Merge pull request #527 from prometheus/nogodep-build
Copy vendored deps manually instead of using Godeps.
2015-02-17 17:17:45 +01:00
beorn7 af91fb8e31 Improve persisting chunks to disk.
This is done by bucketing chunks by fingerprint. If the persisting to
disk falls behind, more and more chunks are in the queue. As soon as
there are "double hits", we will now persist both chunks in one go,
doubling the disk throughput (assuming it is limited by disk
seeks). Should even more pile up so that we end wit "triple hits", we
will persist those first, and so on.

Even if we have millions of time series, this will still help,
assuming not all of them are growing with the same speed. Series that
get many samples and/or are not very compressable will accumulate
chunks faster, and they will soon get double- or triple-writes.

To improve the chance of double writes,
-storage.local.persistence-queue-capacity could be set to a higher
value. However, that will slow down shutdown a lot (as the queue has
to be worked through). So we leave it to the user to set it to a
really high value. A more fundamental solution would be to checkpoint
not only head chunks, but also chunks still in the persist queue. That
would be quite complicated for a rather limited use-case (running many
time series with high ingestion rate on slow spinning disks).
2015-02-17 16:02:09 +01:00
Julius Volz 13048b7468 Simplify GOPATH/dependency setup. 2015-02-17 02:20:16 +01:00
Julius Volz 464524fa44 Make config/Makefile use user Go installation.
Otherwise it will fail to "go get" the "github.com/golang/protobuf"
package because its dir already exists in the vendored packages,
but the copy isn't a git repository.
2015-02-17 02:20:16 +01:00
Julius Volz 5643587cc9 Also vendor Prometheus-internal deps and update existing ones. 2015-02-17 02:20:12 +01:00
Julius Volz b85c72bc50 Get rid of unnecessary tabs in Makefile.INCLUDE. 2015-02-17 02:08:56 +01:00
Julius Volz 740a99a9ac Remove Google Code files, add GitHub files for protobuf lib. 2015-02-17 02:08:56 +01:00
Julius Volz af627bb2b9 Copy vendored deps manually instead of using Godeps.
We were using Godep incorrectly (cloning repos from the internet during
build time instead of including Godeps/_workspace in the GOPATH via
"godep go"). However, to avoid even having to fetch "godeps" from the
internet during build, this now just copies the vendored files into the
GOPATH.

Also, the protocol buffer library moved from Google Code to GitHub,
which is reflected in these updates.

This fixes https://github.com/prometheus/prometheus/issues/525
2015-02-17 02:08:56 +01:00
juliusv 4d135b8b36 Merge pull request #530 from prometheus/fix-mac-build
Fix embedded files generation on OS X.
2015-02-16 20:27:49 +01:00
Julius Volz 5917a04554 Fix embedded files generation on OS X. 2015-02-16 16:52:11 +01:00
juliusv 0ae2c1fc18 Merge pull request #526 from prometheus/lightening-embedding
Dramatically decrease resources for file embedding.
2015-02-13 17:55:31 +01:00
Julius Volz 8cb2c802a0 Dramatically decrease resources for file embedding.
This dramatically decreases the needed time and memory for building the
blob files. The memory numbers are measured via the
memory.max_usage_in_bytes value from cgroups.

* generating files.go:
OLD: 466MB   19s
NEW:  80MB    1s

* building files.go:
OLD: 1210MB  2.25s
NEW:    7MB  0.05s
2015-02-13 17:16:44 +01:00
Björn Rabenstein 498e1b5154 Merge pull request #524 from prometheus/beorn7/ingestion-tweaks
Improve performance of ingestion.
2015-02-13 14:49:16 +01:00
beorn7 e22f26bc58 Move to a queue model for appending samples after all.
Starting a goroutine takes 1-2µs on my laptop. From the "numbers every
Go programmer should know", I had 300ns for a channel send in my
mind. Turns out, on my laptop, it takes only 60ns. That's fast enough
to warrant the machinery of yet another channel with a fixed set of
worker goroutines feeding from it. The number chosen (8 for now) is
low enough to not really afflict a measurable overhead (a big
Prometheus server has >1000 goroutines running), but high enough to
not make sample ingestion a bottleneck.
2015-02-13 14:26:54 +01:00
beorn7 fe518fdb28 Simplify AppendSamples by allowing it to be goroutine-unsafe. 2015-02-13 12:13:22 +01:00
beorn7 8a1c195b54 Move emptiness check to the receivers. 2015-02-12 19:47:24 +01:00
beorn7 5d3cd65a5d Improve performance of ingestion.
- Parallelize AppendSamples as much as possible without breaking the
  contract about temporal order.

- Allocate more fingerprint locker slots.

- Do not run early checkpoints if we are behind on chunk persistence.

- Increase fpMinWaitDuration to give the disk more time for more
  important things.

Also, switch math.MaxInt64 and math.MinInt64 to the new constants.
2015-02-12 18:12:37 +01:00
Johannes 'fish' Ziemke 3d0fb51648 Merge pull request #521 from sammcj/patch-1
Clean fetched package cache
2015-02-11 11:21:56 +01:00
Sam ddc065b943 Clean fetched package cache
To further reduce image size
2015-02-11 11:23:35 +11:00