Commit graph

115 commits

Author SHA1 Message Date
Krasi Georgiev b75428ec19 rename package retrieve to scrape
no fucnctinal changes just renaming retrieval to scrape
2018-02-01 09:55:07 +00:00
Frederic Branczyk 5cea27f06a
Merge pull request #3655 from Conorbro/dropped-target-fix
Fix dropped target list growing forever
2018-01-17 11:03:39 +01:00
Adam f64014f70b Fix the incrementing of prometheus_target_scrapes_exceeded_sample_limit_total (#3669)
Once a scrape target has reached the samle_limit, any further samples
from that target will continue to result in errSampleLimit.

This means that after the completion of the append loop `err` must still
be errSampleLimit, and so the metric would not have been incremented.
2018-01-09 15:43:28 +00:00
conorbroderick 658914ba27 Fix dropped target list growing forever 2018-01-05 11:08:37 +00:00
Shubheksha Jalan ec94df49d4 Refactor SD configuration to remove config dependency (#3629)
* refactor: move targetGroup struct and CheckOverflow() to their own package

* refactor: move auth and security related structs to a utility package, fix import error in utility package

* refactor: Azure SD, remove SD struct from config

* refactor: DNS SD, remove SD struct from config into dns package

* refactor: ec2 SD, move SD struct from config into the ec2 package

* refactor: file SD, move SD struct from config to file discovery package

* refactor: gce, move SD struct from config to gce discovery package

* refactor: move HTTPClientConfig and URL into util/config, fix import error in httputil

* refactor: consul, move SD struct from config into consul discovery package

* refactor: marathon, move SD struct from config into marathon discovery package

* refactor: triton, move SD struct from config to triton discovery package, fix test

* refactor: zookeeper, move SD structs from config to zookeeper discovery package

* refactor: openstack, remove SD struct from config, move into openstack discovery package

* refactor: kubernetes, move SD struct from config into kubernetes discovery package

* refactor: notifier, use targetgroup package instead of config

* refactor: tests for file, marathon, triton SD - use targetgroup package instead of config.TargetGroup

* refactor: retrieval, use targetgroup package instead of config.TargetGroup

* refactor: storage, use config util package

* refactor: discovery manager, use targetgroup package instead of config.TargetGroup

* refactor: use HTTPClient and TLS config from configUtil instead of config

* refactor: tests, use targetgroup package instead of config.TargetGroup

* refactor: fix tagetgroup.Group pointers that were removed by mistake

* refactor: openstack, kubernetes: drop prefixes

* refactor: remove import aliases forced due to vscode bug

* refactor: move main SD struct out of config into discovery/config

* refactor: rename configUtil to config_util

* refactor: rename yamlUtil to yaml_config

* refactor: kubernetes, remove prefixes

* refactor: move the TargetGroup package to discovery/

* refactor: fix order of imports
2017-12-29 21:01:34 +01:00
Krasi Georgiev 60ef2016d5 add a cancel func to the scrape pool as it is needed in the scrape loop select block 2017-12-18 17:29:00 +00:00
Krasi Georgiev 9c61f0e8a0 scrape pool doesn't rely on context as Stop() needs to be blocking to prevent Scrape loops trying to write to a closed TSDB storage. 2017-12-18 17:22:49 +00:00
conorbroderick 8b6f3a1bd3 Added service discovery view showing labels before and after relabelling 2017-12-13 15:46:11 +00:00
Julius Volz 099df0c5f0 Migrate "golang.org/x/net/context" -> "context" (#3333)
In some places, where ctxhttp or gRPC are concerned, we still need to use the
old contexts.
2017-10-24 21:21:42 -07:00
Marc Sluiter 6a633eece1 Added go-conntrack for monitoring http connections (#3241)
Added metrics for in- and outgoing traffic with go-conntrack.
2017-10-06 11:22:19 +01:00
Tobias Schmidt 40c278ee2d Send a HTTP Accept header when scraping 2017-09-25 14:51:29 +02:00
Fabian Reinartz 249d69b513 Merge pull request #3186 from prometheus/startweb
web: start web handler while TSDB is starting up
2017-09-21 09:53:03 +02:00
Fabian Reinartz 7b02bfee0a web: start web handler while TSDB is starting up 2017-09-20 15:03:19 +02:00
Fabian Reinartz 437f51a85f Fix cache maintenance on changing metric representations
We were not properly maintaining the scrape cache when the same metric
was exposed with a different string representation.
This overall reduces the scraping cache's complexity, which fixes the
issue and saves about 10% of memory in a scraping-only Prometheus
instance.
2017-09-19 15:03:27 +02:00
Goutham Veeramachaneni 3f0267c548 Merge branch 'dev-2.0' into go-kit/log
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-15 23:15:27 +05:30
Fabian Reinartz 1121b9f7d4 retrieval: cache dropped series, mutate labels in place 2017-09-14 08:36:19 +02:00
Fabian Reinartz d21f149745 *: migrate to go-kit/log 2017-09-08 22:01:51 +05:30
Fabian Reinartz 5bed8af4cb retrieval: pool scrape buffers
This adds a bucketed buffer pool to the scrapers so we don't have to
allocate a new buffer on each scrape or hold it fixed to the scrape
loop.

The latter can consume significant amounts of unused memory, e.g. 4GB
when scraping 2MB /metrics from 2000 targets.
2017-09-07 14:43:21 +02:00
Fabian Reinartz a8887f46dc Merge branch 'dev-2.0' of github.com:prometheus/prometheus into dev-2.0 2017-09-07 14:15:12 +02:00
Fabian Reinartz 0efecea6d4 Adapt storage APIs to uint64 references 2017-09-07 14:14:41 +02:00
Krasi Georgiev 153cb0cbe3 scraping errors will show in the log when debug mode is enabled (#3135)
Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>
2017-09-05 11:55:14 +01:00
Fabian Reinartz 9516d04472 util: Add idle timeout for scrape connections 2017-08-10 14:47:51 +02:00
Fabian Reinartz 4d3d8ee229 Merge pull request #2850 from tomwilkie/dev-2.0-remote
Remote APIs for v2
2017-08-03 13:39:09 +02:00
Edward Marshall c490725ac9 Additional targetScrapeSample metrics (#3018) 2017-08-02 13:10:18 +01:00
Tom Wilkie 1f3b59ccf5 s/met/lset/ 2017-07-18 11:42:29 +01:00
Tom Wilkie 014bd31a86 Remove unnecessary whitespace changes, add comment. 2017-07-13 11:26:46 +01:00
Tom Wilkie 2ac1809a5b Get label set from cache in addReportSample. 2017-07-12 22:09:16 +01:00
Tom Wilkie 240feb313b Don't regenerate label set for cached values. 2017-07-12 15:54:38 +01:00
Tom Wilkie db8128ceeb Add label set as first parameter to AddFast, ingored by TSDB adapter. 2017-07-12 15:20:12 +01:00
Goutham Veeramachaneni 243419c007 Return tsdb.ErrOutOfBounds as storage.ErrOutOfBounds
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-06 14:18:31 +02:00
Goutham Veeramachaneni 643c5837a0 Stop metrics that are 10mins ahead from now
Fixes #2893

Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>
2017-07-04 15:34:08 +02:00
Goutham Veeramachaneni 3069bd3996 Handle scrapes with OutOfBounds metrics better
fixes #2894

Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>
2017-07-04 11:24:13 +02:00
Fabian Reinartz 9ea748e745 Don't reallocate label set if still known
If the storage deprecates a ref, we have to re-insert with the full
label set. Typically that doesn't correlate with a new series being
created.
We can still use the allocated label set from before.
2017-06-26 14:38:57 +02:00
Fabian Reinartz 2368d2c45b retrieval: fix memory leak in scrape cache 2017-06-26 00:24:54 +02:00
Fabian Reinartz 98c2d8477a Merge pull request #2844 from Gouthamve/cobra
Move CLI commander to cobra
2017-06-19 11:59:52 +02:00
Goutham Veeramachaneni 507790a357
Rework logging to use explicitly passed logger
Mostly cleaned up the global logger use. Still some uses in discovery
package.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-16 15:52:44 +05:30
Julius Volz 6f66125809 retrieval: Fix "up" reporting for failed scrapes 2017-06-14 22:22:12 -04:00
Fabian Reinartz 669075c6b9 Merge branch 'master' into dev-2.0 2017-06-06 09:36:51 +02:00
Fabian Reinartz eb651233ac Merge pull request #2787 from prometheus/limit2
Rework sample limit to work for 2.0
2017-06-06 08:21:12 +02:00
Brian Brazil 37bc607e96 Rework sample limit to work for 2.0
Correctly update reported series.
Increment prometheus_target_scrapes_exceeded_sample_limit_total.
Add back unittests.
Ignore stale markers when calculating sample limit.

Fixes #2770
2017-05-31 15:41:51 +01:00
Fabian Reinartz bc7aff8cef retrieval: extract scrape cache 2017-05-30 09:37:23 -07:00
Fabian Reinartz a83014f53c retrieval: fix memory leak and consumption for caches 2017-05-26 08:44:24 +02:00
Fabian Reinartz 3d8661b8d5 Add comment 2017-05-24 17:05:42 +02:00
Fabian Reinartz 43ca652217 retrieval: Don't allocate map on every scrape 2017-05-24 16:23:48 +02:00
Fabian Reinartz d3f662f15e Merge branch 'dev-2.0' into grobie/reduce-noisy-append-errors 2017-05-24 15:29:30 +02:00
Fabian Reinartz d289dc55c3 storage: update TSDB 2017-05-22 11:53:08 +02:00
Brian Brazil 0920972f79 Initilise scraped sample map, and rename to series map. 2017-05-16 18:33:51 +01:00
Brian Brazil bf38963118 Plumb through logger with target field to scrape loop. 2017-05-16 18:33:51 +01:00
Brian Brazil d657d722dc Log count of dupliates/out of order samples as warnings.
Keep log of each sample as debug log.
2017-05-16 18:33:51 +01:00
Brian Brazil 8b9d3e7547 Put end of run staleness handler in seperate function.
Improve log message.
2017-05-16 18:33:51 +01:00