Commit graph

1542 commits

Author SHA1 Message Date
beorn7 9406afad72 Do not double-count non-persisted head chunks on loading. 2015-02-27 00:06:16 +01:00
beorn7 dbc22b972c Check last time in head chunk for head chunk timeout, not first. 2015-02-26 23:40:42 +01:00
Björn Rabenstein db49fd965f Merge pull request #562 from prometheus/beorn7/vendoring
Update vendoring.
2015-02-26 22:49:25 +01:00
beorn7 f74f7f2fb1 Re-rewrite import paths.
Another run of 'godep save' fixed the import path.

So you first have to run 'godep update <packages>' and then a 'godep save'.

Let's see if this satisfies Travis.
2015-02-26 22:08:45 +01:00
beorn7 08acf744f7 Update client_golang. 2015-02-26 19:26:44 +01:00
beorn7 784176d572 Update vendoring. 2015-02-26 19:26:43 +01:00
Björn Rabenstein f8cb25e932 Merge pull request #559 from prometheus/beorn7/fix
Fix the embarrassing bug introduced in commit 0851945.
2015-02-26 19:25:57 +01:00
beorn7 edd716e63c Fix the embarrassing bug introduced in commit 0851945.
In that commit, the 'maintainSeries' call was accidentally removed.

This commit refactors things a bit so that there is now a clean
'maintainMemorySeries' and a 'maintainArchivedSeries' call.

Straighten the nomenclature a bit (consistently use 'drop' for
chunks and 'purge' for series/metrics).

Remove the annoying 'Completed maintenance sweep through archived
fingerprints' message if there were no archived fingerprints to do
maintenance on.
2015-02-26 18:30:33 +01:00
Brian Brazil 93edd930a6 Better handling of edge conditions in humanizeDuration 2015-02-26 17:30:03 +00:00
Björn Rabenstein 041aa59623 Merge pull request #553 from prometheus/beorn7/release
Version 0.11.0!
2015-02-23 20:21:59 +01:00
beorn7 5f8d7a9dbd Cut release 0.11.0. 2015-02-23 20:21:00 +01:00
beorn7 a4f6edf496 Vendor client_golang v0.2.0. 2015-02-23 18:11:42 +01:00
Björn Rabenstein 136cf661b7 Merge pull request #545 from prometheus/beorn7/quantile
Add the histogram_quantile function.
2015-02-23 15:30:24 +01:00
beorn7 1a61bcae07 Fix plural of 'histogram'.
Actually, 'histogram' is Ancient Greek and 3rd declension... ;-)
2015-02-23 15:29:26 +01:00
Julius Volz 03191cfc8e Merge pull request #550 from atombender/fix_makefile
Makefile: Calculate $(GOROOT), etc. relative to Makefile
2015-02-23 12:42:22 +01:00
Alexander Staubo 8a47f8a44d Makefile: Calculate $(GOROOT), etc. relative to Makefile instead of using $(PWD), which is not always available (eg., in debuild). 2015-02-22 23:14:11 -05:00
beorn7 9827e1ccec Update vendoring of client_golang. 2015-02-22 01:04:52 +01:00
beorn7 17443d288b Avoid copying of the COWMetric if we already have the metric available. 2015-02-22 01:04:52 +01:00
beorn7 9e7c3e3bcd Add the histogram_quantile function.
Since we are now getting really deep into floating point calculation,
the tests had to take into account the precision loss. Since the rule
tests are based on direct line matching in the output, implementing
the "almost equal" semantics was pretty cumbersome, but here we are.
2015-02-22 01:04:51 +01:00
Julius Volz 452c88964a Merge pull request #546 from prometheus/fix-label-grouping
Fix aggregation grouping key calculation.
2015-02-21 17:55:26 +01:00
Julius Volz 42601acfde Replace labelsToKey() with metric Fingerprint (fixes grouping bug). 2015-02-21 17:45:47 +01:00
Julius Volz 7fefccd929 Write() directly into hash and use model.SeparatorByte. 2015-02-21 17:19:13 +01:00
Julius Volz 645cf57bed Fix aggregation grouping key calculation. 2015-02-21 14:05:50 +01:00
Julius Volz 79834edcb5 Merge pull request #544 from q3k/feature/hide-instance-auth
Hide HTTP auth parts from instance label
2015-02-20 17:34:04 +01:00
Sergiusz 'q3k' Bazański 0d0bb3c030 Change instance identifiers to be host:port
This changes the PublicURL function into InstanceIdentifier, which now
returns a simple <host>:<port> string instead of a full URL.
2015-02-20 16:21:13 +01:00
Sergiusz 'q3k' Bazański bb69a3d284 Hide HTTP auth parts from URL
This  instroduces an extra function in the Target interface (PublicURL)
which is used to populate the instance field in scraped metrics.
2015-02-19 18:58:47 +01:00
Brian Brazil 79bf5a278e Merge pull request #543 from kormat/cpucount
Add cpu count to rhs table on node cpu console
2015-02-19 11:19:44 +00:00
Stephen Shirley 05a746bf95 Add cpu count to rhs table
Also fix formatting of bounded values to be more readable.
2015-02-19 12:07:52 +01:00
Julius Volz 2dd70b8c13 Merge pull request #542 from prometheus/I-cant-believe-its-not-federation
Proof of concept for federation via console templates.
2015-02-19 00:23:39 +01:00
Brian Brazil 5adcec3018 Proof of concept for federation via console templates.
This exposes samples via the console templates in the
text exposition format, which the parser will fall back to.

This is not a proper federation solution, but should tide us
over for now. Extenions could include passing in a query or queries in
the url parameters.
2015-02-18 23:21:33 +00:00
Julius Volz c069c0dafa Merge pull request #536 from prometheus/offset
Implement offset operator.
2015-02-18 14:48:56 +01:00
Brian Brazil 40b0a1ac0d Merge pull request #537 from kormat/master
Fix available memory calculation.
2015-02-18 10:36:08 +00:00
Stephen Shirley fbcbb6a635 Fix available memory calculation.
Also account for buffers, making the value match the output of free(1)
2015-02-18 11:26:29 +01:00
Julius Volz 15b2b5aa66 Add tests for invalid uses of "offset". 2015-02-18 02:56:40 +01:00
Julius Volz 67e20acc6c Lower-case some package-internal names. 2015-02-18 02:45:54 +01:00
Julius Volz 72d7b325a1 Implement offset operator.
This allows changing the time offset for individual instant and range
vectors in a query.

For example, this returns the value of `foo` 5 minutes in the past
relative to the current query evaluation time:

    foo offset 5m

Note that the `offset` modifier always needs to follow the selector
immediately. I.e. the following would be correct:

    sum(foo offset 5m) // GOOD.

While the following would be *incorrect*:

    sum(foo) offset 5m // INVALID.

The same works for range vectors. This returns the 5-minutes-rate that
`foo` had a week ago:

    rate(foo[5m] offset 1w)

This change touches the following components:

* Lexer/parser: additions to correctly parse the new `offset`/`OFFSET`
  keyword.
* AST: vector and matrix nodes now have an additional `offset` field.
  This is used during their evaluation to adjust query and result times
  appropriately.
* Query analyzer: now works on separate sets of ranges and instants per
  offset. Isolating different offsets from each other completely in this
  way keeps the preloading code relatively simple.

No storage engine changes were needed by this change.

The rules tests have been changed to not probe the internal
implementation details of the query analyzer anymore (how many instants
and ranges have been preloaded). This would also become too cumbersome
to test with the new model, and measuring the result of the query should
be sufficient.

This fixes https://github.com/prometheus/prometheus/issues/529
This fixed https://github.com/prometheus/promdash/issues/201
2015-02-18 02:41:27 +01:00
Julius Volz 79a4a6d8e8 Merge pull request #532 from prometheus/beorn7/ingestion-tweaks
Improve persisting chunks to disk.
2015-02-17 17:31:00 +01:00
Julius Volz b0c8b56603 Merge pull request #527 from prometheus/nogodep-build
Copy vendored deps manually instead of using Godeps.
2015-02-17 17:17:45 +01:00
beorn7 af91fb8e31 Improve persisting chunks to disk.
This is done by bucketing chunks by fingerprint. If the persisting to
disk falls behind, more and more chunks are in the queue. As soon as
there are "double hits", we will now persist both chunks in one go,
doubling the disk throughput (assuming it is limited by disk
seeks). Should even more pile up so that we end wit "triple hits", we
will persist those first, and so on.

Even if we have millions of time series, this will still help,
assuming not all of them are growing with the same speed. Series that
get many samples and/or are not very compressable will accumulate
chunks faster, and they will soon get double- or triple-writes.

To improve the chance of double writes,
-storage.local.persistence-queue-capacity could be set to a higher
value. However, that will slow down shutdown a lot (as the queue has
to be worked through). So we leave it to the user to set it to a
really high value. A more fundamental solution would be to checkpoint
not only head chunks, but also chunks still in the persist queue. That
would be quite complicated for a rather limited use-case (running many
time series with high ingestion rate on slow spinning disks).
2015-02-17 16:02:09 +01:00
Julius Volz 13048b7468 Simplify GOPATH/dependency setup. 2015-02-17 02:20:16 +01:00
Julius Volz 464524fa44 Make config/Makefile use user Go installation.
Otherwise it will fail to "go get" the "github.com/golang/protobuf"
package because its dir already exists in the vendored packages,
but the copy isn't a git repository.
2015-02-17 02:20:16 +01:00
Julius Volz 5643587cc9 Also vendor Prometheus-internal deps and update existing ones. 2015-02-17 02:20:12 +01:00
Julius Volz b85c72bc50 Get rid of unnecessary tabs in Makefile.INCLUDE. 2015-02-17 02:08:56 +01:00
Julius Volz 740a99a9ac Remove Google Code files, add GitHub files for protobuf lib. 2015-02-17 02:08:56 +01:00
Julius Volz af627bb2b9 Copy vendored deps manually instead of using Godeps.
We were using Godep incorrectly (cloning repos from the internet during
build time instead of including Godeps/_workspace in the GOPATH via
"godep go"). However, to avoid even having to fetch "godeps" from the
internet during build, this now just copies the vendored files into the
GOPATH.

Also, the protocol buffer library moved from Google Code to GitHub,
which is reflected in these updates.

This fixes https://github.com/prometheus/prometheus/issues/525
2015-02-17 02:08:56 +01:00
juliusv 4d135b8b36 Merge pull request #530 from prometheus/fix-mac-build
Fix embedded files generation on OS X.
2015-02-16 20:27:49 +01:00
Julius Volz 5917a04554 Fix embedded files generation on OS X. 2015-02-16 16:52:11 +01:00
juliusv 0ae2c1fc18 Merge pull request #526 from prometheus/lightening-embedding
Dramatically decrease resources for file embedding.
2015-02-13 17:55:31 +01:00
Julius Volz 8cb2c802a0 Dramatically decrease resources for file embedding.
This dramatically decreases the needed time and memory for building the
blob files. The memory numbers are measured via the
memory.max_usage_in_bytes value from cgroups.

* generating files.go:
OLD: 466MB   19s
NEW:  80MB    1s

* building files.go:
OLD: 1210MB  2.25s
NEW:    7MB  0.05s
2015-02-13 17:16:44 +01:00
Björn Rabenstein 498e1b5154 Merge pull request #524 from prometheus/beorn7/ingestion-tweaks
Improve performance of ingestion.
2015-02-13 14:49:16 +01:00