Commit graph

10024 commits

Author SHA1 Message Date
Fabian Reinartz 2c3e778d90 Compactor interface 2017-03-02 09:17:30 +01:00
Fabian Reinartz 9304179ef7 Merge branch 'master' into dev-2.0 2017-03-02 08:16:58 +01:00
Derek Marcotte 0a7fb56b16 Expose PromConsole.Graph.buildQueryUrl, refactor dispatch to use (#2461)
Expose buildQueryUrl, refactor dispatch to use

buildQueryUrl will allow users to execute queries over the range of an
existing graph.  This will be helpful to select data series they wish to
annotate the graph with, for example.
2017-03-01 22:37:50 +00:00
Fabian Reinartz 2c3b56350a Write meta.json file atomically 2017-03-01 17:19:57 +01:00
Fabian Reinartz 6c4217276f Periodically trigger compaction
This addresses an issue where the compaction triggered on cutting
a new block doesn't find anything as the writers are still active on the
block that should be ready for compaction.
2017-03-01 16:52:21 +01:00
Julius Volz 7e14533b32 Merge pull request #2396 from lightpriest/fuzzy-fix-lookup
Fix fuzzy search lookup issues
2017-02-28 19:06:30 +01:00
Erdem Agaoglu 8809735d7f Setting User-Agent header (#2447) 2017-02-28 09:59:33 -04:00
Fabian Reinartz 097a2c1e59 vendor: re-vendor storage 2017-02-28 09:34:30 +01:00
Fabian Reinartz 4397b4d508 *: pass Prometheus registry into storage 2017-02-28 09:33:14 +01:00
Fabian Reinartz db5c88ea9a Misc compaction fixes 2017-02-28 08:40:51 +01:00
Fabian Reinartz b281e4e39b Accept prometheus.Registerer in constructor 2017-02-28 07:26:24 +01:00
Julius Volz 07491c9f1b Merge pull request #2452 from prometheus/swappable-post
notifier: Allow swapping out HTTP Doer
2017-02-27 22:15:24 +01:00
Julius Volz f152ac5e23 notifier: Allow swapping out HTTP Doer
We need to be able to modify the HTTP POST in Weave Cortex to add
multitenancy information to a notification. Since we only really need a
special header in the end, the other option would be to just allow
passing in headers to the notifier. But swapping out the whole Doer is
more general and allows others to swap out the network-talky bits of the
notifier for their own use. Doing this via contexts here wouldn't work
well, due to the decoupled flow of data in the notifier.

There was no existing interface containing the ctxhttp.Post() or
ctxhttp.Do() methods, so I settled on just using Do() as a swappable
function directly (and with a more minimal signature than Post).
2017-02-27 20:36:22 +01:00
Fabian Reinartz 306831f151 Add per-block state ULID 2017-02-27 10:46:15 +01:00
Fabian Reinartz 9a5dfadb09 Tweak IndexWriter interface, serialize directly into dir 2017-02-25 07:24:20 +01:00
Tom Wilkie 1ab893c6ec Limit 'discarding sample' logs to 1 every 10s (#2446)
* Limit 'discarding sample' logs to 1 every 10s

* Include the vendored library

* Review feedback
2017-02-23 19:20:39 +01:00
Fabian Reinartz 78780cd2ba Segment chunk file
This adds write path support for segmented chunk data files.
Files of 512MB are pre-allocated and written to. If the file size
is exceeded, the next file is started. On completion, files
are truncated to their final size.
2017-02-23 10:50:22 +01:00
Julius Volz 16bd5c8ebe Merge pull request #2423 from prometheus/multiple-remote-writers
Re-add multiple remote writers
2017-02-23 09:48:40 +01:00
Fabian Reinartz cc0ff26f1f retrieval: handle GZIP compression ourselves
The automatic GZIP handling of net/http does not preserve
buffers across requests and thus generates a lot of garbage.
We handle GZIP ourselves to circumvent this.t
2017-02-22 13:25:25 +01:00
Julius Volz 2f39dbc8b3 Rename StorageQueueManager -> QueueManager 2017-02-21 21:45:43 +01:00
Julius Volz e9476b35d5 Re-add multiple remote writers
Each remote write endpoint gets its own set of relabeling rules.

This is based on the (yet-to-be-merged)
https://github.com/prometheus/prometheus/pull/2419, which removes legacy
remote write implementations.
2017-02-20 13:23:12 +01:00
Julius Volz a6d81a9f88 Update vendoring for github.com/prometheus/common/model 2017-02-20 13:23:12 +01:00
Björn Rabenstein 8cd3be5c5f Merge pull request #2433 from prometheus/beorn7/meta
Replace AUTHORS.md by an updated MAINTAINERS.md
2017-02-20 11:46:17 +01:00
beorn7 a1596f9c79 Replace AUTHORS.md by an updated MAINTAINERS.md 2017-02-20 11:45:22 +01:00
Fabian Reinartz 311e7b5069 storage/vendor: update to latest fabxc/tsdb 2017-02-20 11:11:44 +01:00
Fabian Reinartz a3b47c4929 Create default logger for DB 2017-02-19 16:04:37 +01:00
Fabian Reinartz f734773214 Replace per-file locking with single PID lock file
File locks have a multitude of problems that make them hard to use
correctly. As they are just advisory, they are only meaningful to
prevent accidents like running the same process twice.
A simple PID file lock works reliably in those cases and is simpler.
2017-02-19 13:01:19 +01:00
Fabian Reinartz c808928b90 SeriesWriter -> ChunkWriter
This commit simplifies a SeriesWriter into a ChunkWriter and detaches
it entirely from the notion of a series and the series index.
2017-02-19 11:27:31 +01:00
Fabian Reinartz a3d042b54e Support multiple chunk files in read path 2017-02-18 17:33:20 +01:00
Derek Marcotte 74a3b1051a Add rickshawGraph proerty to the Graph object. (#2438)
Add rickshawGraph property to the Graph object.

This allows further changes to be made to the graph, for example,
annotations.
2017-02-18 10:03:32 +00:00
Björn Rabenstein 089dc1076b Merge pull request #2435 from jmeulemans/open-chunks-gauge
Adding gauge for number of open head chunks.
2017-02-17 16:02:06 +01:00
Jeremy Meulemans 025c828976 Changed to open_head_chunks to address review.
Now incrementing numHeadChunks directly.
2017-02-17 07:10:13 -06:00
Frederic Branczyk e14caf83e7 Merge pull request #2434 from stephen-soltesz/master
Fix kubernetes host:port relabel regex.
2017-02-17 10:51:05 +00:00
Jeremy Meulemans 074050b8c0 Updating for failed codeclimate check. 2017-02-16 18:04:28 -06:00
Jeremy Meulemans f70b52d0b6 Adding gauge for number of open head chunks.
Fixes #1710
2017-02-16 17:56:45 -06:00
Stephen Soltesz 3f29324e04 Fix kubernetes host:port relabel regex.
This change corrects a bug introduced by PR
https://github.com/prometheus/prometheus/pull/2427

The regex uses three groups: the hostname, an optional port, and the
prefered port from a kubernetes annotation.

Previously, the second group should have been ignored if a :port was not
present in the input. However, making the port group optional with the
"?" had the unintended side-effect of allowing the hostname regex "(.+)"
to match greedily, which included the ":port" patterns up to the ";"
separating the hostname from the kubernetes port annotation.

This change updates the regex for the hostname to match any non-":"
characters. This forces the regex to stop if a ":port" is present and
allow the second group to match the optional port.
2017-02-16 14:46:04 -05:00
Fabian Reinartz afa084920c Use Castagnoli polynomial for CRC32s; cache them 2017-02-15 21:13:17 -08:00
Fabian Reinartz 9c7a88223e Add full encode/decode WAL cycle test 2017-02-14 21:55:50 -08:00
Tobias Schmidt 0436e982cb Merge pull request #2427 from stephen-soltesz/master
Kubernetes port annotations - match addresses with or without declared ports
2017-02-14 22:45:30 -05:00
Stephen Soltesz 0b1790ee44 Match addresses with or without declared ports.
This change updates port relabeling for pod and service discovery so the
relabeling regex matches addresses with or without declared ports. As
well, this change uses a consistent style in the replacement pattern
for the two expressions.

Previously, for both services or pods that did not have declared ports, the
relabel config regex would fail to match:

    __meta_kubernetes_service_annotation_prometheus_io_port
    regex: (.+)(?::\d+);(\d+)

    __meta_kubernetes_pod_annotation_prometheus_io_port
    regex: (.+):(?:\d+);(\d+)

Both regexes expected a <host>:<port> pattern.

The new regex matches addresses with or without declared ports by making
the :<port> pattern optional.

    __meta_kubernetes_service_annotation_prometheus_io_port
    __meta_kubernetes_pod_annotation_prometheus_io_port
    regex: (.+)(?::\d+)?;(\d+)
2017-02-14 20:12:38 -05:00
Fabian Reinartz 2c97428a79 Add basic WAL tests 2017-02-14 16:12:56 -08:00
Björn Rabenstein a37d39c463 Merge pull request #2425 from prometheus/beorn7/vendoring
vendoring: Do not ignore appengine build tags
2017-02-14 22:14:27 +01:00
Julius Volz 980586d183 Merge pull request #2419 from prometheus/remove-legacy-remotes
Remove legacy remote storage implementations
2017-02-14 18:35:39 +01:00
beorn7 b4ce5ca925 vendoring: Do not ignore appengine build tags
For weird reasons, this excluded github.com/miekg/dns/udp_other.go
from vendoring.
2017-02-14 18:32:07 +01:00
Julius Volz beb3c4b389 Remove legacy remote storage implementations
This removes legacy support for specific remote storage systems in favor
of only offering the generic remote write protocol. An example bridge
application that translates from the generic protocol to each of those
legacy backends is still provided at:

documentation/examples/remote_storage/remote_storage_bridge

See also https://github.com/prometheus/prometheus/issues/10

The next step in the plan is to re-add support for multiple remote
storages.
2017-02-14 17:52:05 +01:00
Björn Rabenstein 0f412e7e26 Merge pull request #2422 from levenlabs/nameList
discovery: Instead of looping over conf.Search, use NameList()
2017-02-14 17:48:06 +01:00
Fabian Reinartz 84e8027a8e Close consumed WAL segments, verify checksums on read 2017-02-14 00:42:06 -08:00
Fabian Reinartz eeb03e97e2 Add headers and format flag for WAL segments 2017-02-14 00:24:53 -08:00
Fabian Reinartz 79944a5912 Break out WAL into segment files 2017-02-13 23:53:19 -08:00
James Hartig 865f28bb15 discovery: Instead of looping over conf.Search, use NameList() 2017-02-13 15:48:51 -05:00