Commit graph

289 commits

Author SHA1 Message Date
Brian Brazil 93145b960a retrieval: Reduce flakiness of target tests
Bump timeouts of tests where we don't want I/O timeouts.

Adjust the full channel test to be much more reliable,
by reducing the ingestion timeout from 1ms to 0.
2015-09-22 19:23:36 +01:00
Fabian Reinartz cac6eea434 Merge pull request #1105 from prometheus/consulnil
Fix nil panic on consul error
2015-09-22 14:55:31 +02:00
Fabian Reinartz 327152862c Update expfmt.NewDecoder usage 2015-09-22 12:11:28 +02:00
Fabian Reinartz 1ce89a4a0b Fix nil panic on consul error 2015-09-22 09:04:31 +02:00
Julius Volz af513468eb Fix some dead code, missing error checks, shadowings.
I applied
https://medium.com/@jgautheron/quality-pipeline-for-go-projects-497e34d6567
and was greeted with a deluge of warnings, most of which were not
applicable or really fixable realistically. These are some of the first
ones I decided to fix.
2015-09-14 12:21:34 +02:00
Jimmi Dyson 7ef9399920 Clean up kubernetes http response bodies 2015-09-11 11:44:28 +01:00
Anders Daljord Morken 9fb65a91af Close HTTP connections on HTTP errors too.
Move defer resp.Body.Close() up to make sure it's called even when the
HTTP request returns something other than 200 or Decoder construction
fails. This avoids leaking and eventually running out of file descriptors.
2015-09-10 22:41:05 +02:00
Fabian Reinartz 8456b7e12f Use go1.5.1 2015-09-10 12:11:44 +02:00
Jimmi Dyson a1574aa2b3 Move TLS options to scrape config
Fixes #1013, fixes #989
2015-09-09 09:52:21 +01:00
Julius Volz b7b7b2e883 Merge pull request #1050 from fabric8io/kubernetes-discovery
Kubernetes SD improvements
2015-09-04 14:58:11 +02:00
Jimmi Dyson d7a7fd4589 Kubernetes SD improvements
* Support multiple masters with retries against each master as required.
* Scrape masters' metrics.
* Add role meta label for node/service/master to make it easier for relabeling.
2015-09-04 11:31:20 +01:00
Fabian Reinartz cc1a2a2061 Remove attachment of global labels upon ingestion 2015-09-03 14:16:23 +02:00
Fabian Reinartz ebf417a282 Fix map initialization 2015-09-01 18:06:22 +02:00
Julius Volz f63a899744 Change config regexes to full-string matches.
This anchors all regular expressions entered via the config to match a
full string vs. a substring.

THIS IS A BREAKING CHANGE!

Fixes part of https://github.com/prometheus/prometheus/issues/996
2015-09-01 15:46:41 +02:00
Fabian Reinartz 542da6774e Fix draining of file watcher events 2015-08-28 12:17:22 +02:00
Daniel Lundin 4abf54b747 serverset: extract shard number from serverset data 2015-08-27 16:26:00 +02:00
Julius Volz 29eaa8c7cf Merge pull request #1030 from prometheus/fix-flakey-filesd
Fix flakey FileSD test.
2015-08-26 13:25:00 +02:00
Julius Volz 3fd5826589 Fix flakey FileSD test.
When the test ends, all files matching the watcher's glob are removed
via defer. In that moment, the draining goroutine may still be running
and then detect no files matching the configured glob just before the
test exits.

This is now solved by waiting for the draining goroutine to finish
before leaving the test function and thus causing the deferred file
removal.
2015-08-26 13:06:34 +02:00
Julius Volz 744d5d5a7a Merge pull request #1029 from prometheus/vet-fixes
Fix "go vet" errors.
2015-08-26 12:50:18 +02:00
Julius Volz 995d3b831d Fix most golint warnings.
This is with `golint -min_confidence=0.5`.

I left several lint warnings untouched because they were either
incorrect or I felt it was better not to change them at the moment.
2015-08-26 12:44:46 +02:00
Julius Volz 963ad82dcb Fix "go vet" errors.
I ignored all errors of the type "composite literal uses unkeyed
fields". Most of them are wrong because of
https://github.com/golang/go/issues/9171.
2015-08-26 02:05:04 +02:00
Fabian Reinartz 6664b77f36 Merge pull request #1021 from prometheus/appenders
move metric modifications into SampleAppenders
2015-08-25 17:47:55 +02:00
Fabian Reinartz 01834fa528 Move metric modifications into SampleAppenders 2015-08-25 15:32:37 +02:00
Fabian Reinartz d6d88f8950 Add missing license headers 2015-08-24 19:19:21 +02:00
Julius Volz d36a7f4e6f Fix busylooping in case of no target providers.
merge() closes the channel that handleUpdates() reads from when there
are zero configured target providers in the configuration. In that case,
the for-select loop in handleUpdates() entered a busy loop. It should
exit when the upstream channel is closed.
2015-08-24 16:42:28 +02:00
Fabian Reinartz 3a0145c09e Reenable blocked appending tests 2015-08-22 09:47:57 +02:00
Fabian Reinartz 438e232c9b Fix grouping of import blocks 2015-08-22 09:42:45 +02:00
Fabian Reinartz 6d0f58dcf3 sanitize scrape health recording code 2015-08-21 23:01:08 +02:00
Fabian Reinartz 25bf5fdaf5 Timeout sample appends 2015-08-21 18:04:35 +02:00
Fabian Reinartz 11a577fcd0 Switch to common/expfmt for extraction 2015-08-21 13:33:38 +02:00
Fabian Reinartz 306e8468a0 Switch from client_golang/model to common/model 2015-08-21 13:33:38 +02:00
Sharif Nassar 6cb519fe82 Add Consul ServiceID to the discovery meta labels. 2015-08-20 14:04:42 -07:00
Fabian Reinartz 0f5022c091 Add missing Kubernetes doc strings 2015-08-18 14:37:28 +02:00
Fabian Reinartz f592740bac Only exit static target provider on done 2015-08-18 11:51:53 +02:00
Julius Volz b4adf2723d Merge pull request #994 from robbiet480/consul-datacenter-name
Pass through current agent Consul datacenter name
2015-08-18 01:09:24 +02:00
Robbie Trencheny 48e461f7db Pass through current agent Consul datacenter name
Instead of only filling __meta_consul_dc when datacenter is set in
consul_sd_config this change fills the label based on what the agent
reports it's current data center is, if datacenter isn't manually set,
otherwise it uses whatever datacenter was set to.
2015-08-17 16:00:26 -07:00
Fabian Reinartz d0a90964c1 Fix license header 2015-08-17 19:51:12 +02:00
Fabian Reinartz eabbdc6603 Add missing license headers 2015-08-17 19:49:10 +02:00
Julius Volz 47a96bff1a Update constant names in comments. 2015-08-17 15:05:06 +02:00
Brian Brazil e1d5eb52f2 retrieval: Don't include unmatched source of regex in replacement.
ReplaceAllString only replaces the matching part of the regex,
the unmatched bits around it are left in place. This is not the
expected or desired behaviour as the replacement string should
be everything.

This may break users dependant on this behaviour, but
what they're doing is still possible.
2015-08-17 00:31:56 +01:00
Fabian Reinartz 3c6dd161d7 Scrape all services on empty services list. 2015-08-14 17:39:41 +02:00
Fabian Reinartz 9b9ff66212 Merge pull request #977 from prometheus/fabxc/target-dedup
Improve target discovery pipeline
2015-08-14 16:38:16 +02:00
Fabian Reinartz 8fa9ec278b Add application labels as meta labels
Removes built-in conditional scraping based on application's
'prometheus' label.
2015-08-14 15:34:02 +02:00
Fabian Reinartz f269943950 Adjust Kubernetes SD to pipeline changes 2015-08-14 13:30:27 +02:00
Fabian Reinartz 4e84b86510 Improve target discovery pipeline
Replace the TargetProvider Stop method with done channels
that ensure properly broadcasted shutdown of the whole pipeline.
2015-08-14 13:30:27 +02:00
Fabian Reinartz 15b4115a25 Merge pull request #986 from prometheus/fabxc/tpdoc
Clarify docs of TargetProvider
2015-08-14 12:07:15 +02:00
Fabian Reinartz 625374ee36 Clarify docs of TargetProvider 2015-08-14 12:02:22 +02:00
Fabian Reinartz f7e3722388 Rename __meta_dns_srv_name to __meta_dns_name
This is change potentially breaking relabeling rules.
2015-08-13 17:02:56 +02:00
Fabian Reinartz b964da4b75 Merge pull request #905 from fabric8io/kubernetes-discovery
Kubernetes discovery
2015-08-13 15:08:32 +02:00
Fabian Reinartz 24e91720ad Merge pull request #980 from prometheus/map-labels
Retrieval: Add relabel action to map labels names with a regex.
2015-08-13 14:36:59 +02:00
Brian Brazil 4e70a0a14e Retrieval: Add relabel action to map label names with a regex.
The intended use case is where a user has tags/labels coming
from metadata in Kubernetes or EC2, and wants to make
some subset of them into target labels.
2015-08-13 13:19:11 +01:00
Jimmi Dyson 923f8111d4 Initial Kubernetes discovery
Fixes #904
2015-08-13 10:38:52 +01:00
Miek Gieben caaa3de4ff Make HashMod use MD5 instead of FNV
MD5 will will distribute the inputs more uniformly over the output
space than FNV; leading to more evenly balanced load when using HashMod.
2015-08-13 09:42:07 +01:00
Fabian Reinartz 0138d37458 Improve unique target group sources.
Include position of same SD mechanisms within the same scrape configuration.
Move unique prefixing out of SD implementations and target manager into
its own interface.
2015-08-10 11:29:09 +02:00
Fabian Reinartz 54202bc5a8 Merge pull request #902 from xperimental/feature/marathon-discovery
retrieval/discovery: Service discovery using marathon API
2015-08-10 01:43:37 +02:00
Robert Jacob 4d0f974c42 Add service discovery using Marathon API. 2015-08-10 01:36:24 +02:00
Will Rouesnel 7810448dbe Add proxy_url parameter to allow specifying per-job HTTP proxy servers
Allow scrape_configs to have an optional proxy_url option which specifies
a proxy to be used for all connections to hosts in that config.

Internally this modifies the various client functions to take a *url.URL pointer
which currently must point to an HTTP proxy (but has been left open-ended to
allow the url format to be extended to support others, such as maybe SOCKS if
needed).
2015-08-08 04:29:27 +10:00
Jimmi Dyson da4c50a6cf Make scheme relabelable via discovery 2015-08-06 12:00:33 +01:00
Jimmi Dyson 52cf6b3e6e Configuration options for bearer tokens, client certs & CA certs
Fixes #918, fixes #917
2015-08-04 17:18:46 +01:00
Florian Pfitzer 1fa0b0f253 fix consul port label 2015-07-31 16:20:17 +00:00
Brian Brazil adf7f16d1a Merge pull request #934 from prometheus/query-params
Retrieval: Make it possible to relabel query params
2015-07-31 11:01:45 +01:00
Brian Brazil d8875d17d8 Retrieval: Make it possible to relabel query params
This only allows relabelling the first value
for a given parameter, this should be sufficient in practice.
2015-07-31 10:09:28 +01:00
Johannes 'fish' Ziemke 6e7d743cd4 Merge pull request #946 from prometheus/add-sd-dns-a
Add support for A record based DNS SD
2015-07-30 16:01:47 +02:00
Johannes 'fish' Ziemke 9ab340e95e Add support for A record based DNS SD
If using A records, the user needs to specify "port" and set "type" to
"A".
2015-07-30 15:55:38 +02:00
beorn7 645f6772e5 Add Consul Address, ServicePort, and ServiceAddress to the meta labels.
In setups where the ServiceAddress is the relevant address for
scraping, users can relabel the `__address__` label to ServiceAddress
+ ":" + ServicePort.

This needs to be documented, of course. Will do once this is LGTM'd.
2015-07-22 18:19:13 +02:00
Julius Volz 9d98910fca Revert "Use Consul ServiceAddress instead of Address when set"
This reverts commit 0ac7e7217e.

See discussion on https://github.com/prometheus/prometheus/pull/812 for
reasoning. While fixing one use case, it breaks others, and we need a
more generic way of handling this.
2015-07-22 13:04:29 +02:00
Fabian Reinartz d53cc7935d retrieval: avoid race conditions 2015-07-08 21:27:52 +02:00
Brian Brazil 3d268d681e retrieval: Handle serverset node not existing.
This stops configuration loading hanging if
the Znode doesn't exist, and retries until the
node does exist.
2015-07-01 13:56:31 +01:00
Fabian Reinartz 080e067601 Merge pull request #832 from prometheus/fabxc/target-test
retrieval: double timeout in target scrape test.
2015-06-25 17:23:52 +02:00
Brian Brazil 52859b8033 Merge pull request #836 from prometheus/shard
Add 'hashmod' relabel action.
2015-06-24 21:40:10 +01:00
Brian Brazil 682f949ab1 Add 'hashmod' relabel action.
This takes the modulus of a hash of some labels.
Combined with a keep relabel action, this allows
for sharding of targets across multiple prometheus
servers.
2015-06-24 21:14:53 +01:00
Fabian Reinartz 23862c92c4 retrieval/discovery: refresh services in Consul to recover from missing events. 2015-06-24 17:48:27 +02:00
Fabian Reinartz c292979374 retrieval: double timeout in target scrape test. 2015-06-23 21:59:55 +02:00
Julius Volz d868264bb8 Improve UI of /alerts page.
Changes to the UI:
- "Active Since" timestamps are now human-readable.
- Alerting rules are now pretty-printed better.
- Labels are no longer just strings, but alert bubbles (like we do on
  the status page for base labels).
- Alert states and target health states are now capitalized in the
  presentation layer rather than at the source.
2015-06-23 18:48:45 +02:00
Fabian Reinartz 53b9d5917d web: improve target URL handling and display. 2015-06-23 13:45:15 +02:00
Fabian Reinartz dc7d27ab9a retrieval: add honor label handling and parametrized querying.
This commit adds the honor_labels and params arguments to the scrape
config. This allows to specify query parameters used by the scrapers
and handling scraped labels with precedence.
2015-06-23 13:45:14 +02:00
Fabian Reinartz 459d18cf18 Merge pull request #812 from Marmelatze/consul_services
Use Consul ServiceAddress instead of Address when set
2015-06-17 20:10:52 +02:00
Florian Pfitzer 0ac7e7217e Use Consul ServiceAddress instead of Address when set 2015-06-17 15:39:42 +02:00
Brian Brazil 4d895242f9 Add support for Zookeeper Serversets for SD.
It can discover an entire tree of serversets, or just one.
2015-06-16 11:02:08 +01:00
Brian Brazil 0dbae36d36 Allow ingested metrics to be relabeled.
The main purpose of this is to allow for blacklisting
of expensive metrics as a tactical option.
It could also find uses for renaming and removing labels
from federation.
2015-06-13 15:18:27 +01:00
Brian Brazil 58ceae82bc Revert "Allow ingested metrics to be relabeled."
This reverts commit f2f26ca08f.

Was accidentally pushed to master instead of a branch for PR.
2015-06-12 22:12:26 +01:00
Brian Brazil f2f26ca08f Allow ingested metrics to be relabeled.
The main purpose of this is to allow for blacklisting
of expensive metrics as a tactical option.
It could also find uses for renaming and removing labels
from federation.
2015-06-12 22:06:30 +01:00
Fabian Reinartz b5fe2e9afe Merge pull request #773 from prometheus/fabxc/simple-cfg
config: simplify default config handling.
2015-06-08 16:22:06 +02:00
Brian Brazil b8b1d3cbac Web: Add pre-relabel labels to status page.
Figuring out what's going on with the new service discovery
and labels is difficult. Add a popover with the labels
to the target table to make things simpler, and help
discovery of potentially useful labels.
2015-06-08 12:19:01 +01:00
Fabian Reinartz 0af1cff8af config: simplify default config handling. 2015-06-06 09:04:04 +02:00
Fabian Reinartz 8214b4ee78 retrieval/discovery: surround __meta_consul_tags value with tag seperators. 2015-06-05 19:18:34 +02:00
Fabian Reinartz 280d11dca8 main: exit on invalid rule files on startup. 2015-06-02 18:44:41 +02:00
Fabian Reinartz 0de6edbdfc Move pkg/ to util/ 2015-06-01 21:12:32 +02:00
Fabian Reinartz dfaf31a1da Move web/httputils to pkg/httputil and add DeadlineClient to it 2015-06-01 21:12:31 +02:00
Fabian Reinartz a4f179230a Merge pull request #744 from prometheus/fabxc/fix-labels
Fix discarding of labels in file target groups
2015-05-27 19:57:15 +02:00
Fabian Reinartz e9b344abee Fix discarding of labels in file target groups 2015-05-27 18:52:44 +02:00
Fabian Reinartz 8b7e5f9184 Stop holding TargetManager lock when stopping components.
TargetProviders may flush some last changes to the target manager
before actually stopping. To properly read those form the channel
the target manager must not be locked while stopping a provider.
2015-05-27 12:41:37 +02:00
Brian Brazil f34de493d5 Add increase() function, to replace delta(..., 1).
This calculates how much a counter increases over
a given period of time, which is the area under the curve
of it's rate.

increase(x[5m]) is equivilent to rate(x[5m]) * 300.
2015-05-26 22:49:21 +01:00
Fabian Reinartz efb39cfd4e Fix file SD test 2015-05-23 21:20:39 +02:00
Julius Volz 267fd34156 Switch Prometheus to use github.com/prometheus/log.
This change is conceptually very simple, although the diff is large. It
switches logging from "github.com/golang/glog" to
"github.com/prometheus/log", while not actually changing any log
messages. V(1)-style logging has been changed to be log.Debug*().
2015-05-20 18:19:32 +02:00
Fabian Reinartz 7143dff02f Add initial implementation for SD via Consul.
This commit adds service discovery using Consul's HTTP API and watches
(long polling) to retrieve target updates.
2015-05-20 11:46:24 +02:00
Fabian Reinartz b0c181dc0d Add Consul SD configuration. 2015-05-20 11:46:24 +02:00
Fabian Reinartz ff832d2e03 Attach __meta_filepath label to file SD targets. 2015-05-19 15:49:38 +02:00
Fabian Reinartz 8de50619f1 Increase target test wait times
On slow systems such as Travis CI occasionally the tests fail
because the wait times are too short.
2015-05-19 12:06:52 +02:00
Fabian Reinartz 385919a65a Avoid inter-component blocking if ingestion/scraping blocks.
Appending to the storage can block for a long time. Timing out
scrapes can also cause longer blocks. This commit avoids that those
blocks affect other compnents than the target itself.
Also the Target interface was removed.
2015-05-18 17:58:51 +02:00