Commit graph

2815 commits

Author SHA1 Message Date
beorn7 c13b1ecfe9 Make chunk iterators more DRY
This finally extracts all the common code of the two chunk iterators
into one. Any future chunk encodings with fast access by index can use
the same iterator by simply providing an indexAccessor. Other future
chunk encodings without fast index access (like Gorilla-style) can
still implement the chunkIterator interface as usual.
2016-03-07 20:23:14 +01:00
beorn7 32f280a3cd Slim down the chunkIterator interface
For one, remove unneeded methods.

Then, instead of using a channel for all values, use a
bufio.Scanner-like interface. This removes the need for creating a
goroutine and avoids the (unnecessary) locking performed by channel
sending and receiving.

This will make it much easier to write new chunk implementations (like
Gorilla-style encoding).
2016-03-07 19:50:13 +01:00
Björn Rabenstein 1bd4c92e1f Merge pull request #1457 from prometheus/beorn7/promtool
Add a command to promtool that dumps metadata of heads.db
2016-03-07 17:22:48 +01:00
beorn7 b6fdb355d7 Move dump-heads into its own tool 2016-03-07 16:30:19 +01:00
beorn7 f193f2b8ef Add a command to promtool that dumps metadata of heads.db
I needed this today for debugging. It can certainly be improved, but
it's already quite helpful.

I refactored the reading of heads.db files out of persistence, which
is an improvement, too.

I made minor changes to the cli package to allow outputting via the
io.Writer interface.
2016-03-07 16:21:57 +01:00
Fabian Reinartz 6bbb4af837 Merge pull request #1465 from prometheus/beorn7/fix-test2
Fix flaky file-sd test
2016-03-07 15:46:18 +01:00
beorn7 d44b83690e Fix flaky file-sd test 2016-03-07 15:39:18 +01:00
Björn Rabenstein 2a2cc52828 Merge pull request #1405 from prometheus/beorn7/storage
Streamline series iterator creation
2016-03-07 13:30:56 +01:00
Fabian Reinartz 5b9e85e556 Merge pull request #1404 from prometheus/scraperef2
Retrieval refactoring
2016-03-06 22:17:00 +01:00
Fabian Reinartz 6ceb7e7887 Merge pull request #1463 from mischief/linuxisms
scripts: drop -f from hostname, openbsd does not support it
2016-03-05 08:57:18 +01:00
Nick Owens 53777e7bc4 scripts: drop -f from hostname, openbsd does not support it 2016-03-04 19:59:28 -08:00
Fabian Reinartz 8d2a73aff0 Merge pull request #1451 from pdbogen/origin/1446
rewrite operator balancing to be recursive
2016-03-03 19:42:17 +01:00
Patrick Bogen 250344b344 use short variable assignment 2016-03-03 09:46:50 -08:00
beorn7 75a6b460ef Give TestEvictAndLoadChunkDescs more time to actually evict
Obviously, it's really bad to depend on timing here. The proper fix
would be to have something like WaitForIndexing for other things to
wait for, too.

For now, let's see if the wait time increase fixes the issue.
2016-03-03 13:29:39 +01:00
beorn7 fc7de5374a Quarantine series upon problem writing to the series file
This fixes https://github.com/prometheus/prometheus/issues/1059 , but
not in the obvious way (simply not updating the persist watermark,
because that's actually not that simple - we don't really know what
has gone wrong exactly). As any errors relevant here are most likely
caused by severe and unrecoverable problems with the series file,
Using the now quarantine feature is the right step. We don't really
have to be worried about any inconsistent state of the series because
it will be removed for good ASAP. Another plus is that we don't have
to declare the whole storage dirty anymore.
2016-03-03 13:15:02 +01:00
Fabian Reinartz 29e31dc3c6 Merge pull request #1452 from prometheus/fix-style-checker
Detect code style violations in deeply nested files
2016-03-03 09:22:56 +01:00
Tobias Schmidt d7889e61bb Detect code style violations in deeply nested files
So far the style check did not recognize issues in files in deeply
nested directories, e.g. retrieval/discovery/kubernetes/discovery.go.
2016-03-03 02:21:16 -05:00
Patrick Bogen 2062fbae0f rewrite operator balancing to be recursive 2016-03-02 15:56:40 -08:00
beorn7 0ea5801e47 Handle errors caused by data corruption more gracefully
This requires all the panic calls upon unexpected data to be converted
into errors returned. This pollute the function signatures quite
lot. Well, this is Go...

The ideas behind this are the following:

- panic only if it's a programming error. Data corruptions happen, and
  they are not programming errors.

- If we detect a data corruption, we "quarantine" the series,
  essentially removing it from the database and putting its data into
  a separate directory for forensics.

- Failure during writing to a series file is not considered corruption
  automatically. It will call setDirty, though, so that a
  crashrecovery upon the next restart will commence and check for
  that.

- Series quarantining and setDirty calls are logged and counted in
  metrics, but are hidden from the user of the interfaces in
  interface.go, whith the notable exception of Append(). The reasoning
  is that we treat corruption by removing the corrupted series, i.e. a
  query for it will return no results on its next call anyway, so
  return no results right now. In the case of Append(), we want to
  tell the user that no data has been appended, though.

Minor side effects:

- Now consistently using filepath.* instead of path.*.

- Introduced structured logging where I touched it. This makes things
  less consistent, but a complete change to structured logging would
  be out of scope for this PR.
2016-03-02 23:02:34 +01:00
beorn7 8766f99085 Merge branch 'beorn7/storage2' into beorn7/storage3 2016-03-02 23:02:06 +01:00
beorn7 162f6fa6f6 Merge branch 'beorn7/storage' into beorn7/storage2 2016-03-02 23:01:26 +01:00
beorn7 79a2ae2d2e Add missing test file 2016-03-02 23:00:23 +01:00
Fabian Reinartz 7a0c0c3ca2 Remove noise from CHANGELOG 2016-03-02 17:59:23 +01:00
Fabian Reinartz 1e7ce3ffdb Bump version to 0.17.0 2016-03-02 17:59:10 +01:00
Fabian Reinartz 2bfb86d77c Update changelog for 0.17.0 release 2016-03-02 17:58:55 +01:00
beorn7 b6840997a7 Merge branch 'beorn7/storage2' into beorn7/storage3 2016-03-02 16:11:25 +01:00
beorn7 ce58fd357b Merge branch 'beorn7/storage' into beorn7/storage2
Conflicts:
	storage/local/chunk.go
	storage/local/interface.go
2016-03-02 16:09:32 +01:00
beorn7 2581648f70 Separate iterators by offset
Add test that exposes the problem.
2016-03-02 16:01:03 +01:00
Fabian Reinartz 6adf77e411 Merge pull request #1447 from prometheus/fabxc/alertfix
Make copying alerting state safer.
2016-03-02 12:25:19 +01:00
Fabian Reinartz d89c254849 Make copying alerting state safer.
This considers static labels in the equality of alerts to
avoid falsely copying state from a different alert definition with
the same name across reloads.

To be safe, it also copies the state map rather than just its pointer
so that remaining collisions disappear after one evaluation interval.
2016-03-02 12:21:54 +01:00
Fabian Reinartz 95c9706d2d Fix missing comment period. 2016-03-02 09:16:56 +01:00
Fabian Reinartz ddc74f712b Add sortable target list 2016-03-02 09:10:20 +01:00
Julius Volz 9ea2465b99 Fix typo in lexer test. 2016-03-02 01:13:27 +01:00
Brian Brazil ca31d36382 Merge pull request #1444 from prometheus/add-tests-for-string-parsing
Add tests to specify the string escaping behavior
2016-03-01 22:31:08 +00:00
Tobias Schmidt 907b1380a7 Add tests to specify the string escaping behavior 2016-03-01 17:23:18 -05:00
Fabian Reinartz 5b78fdd6b7 Merge pull request #1439 from prometheus/fabxc/notifier
Rename notification to notifier
2016-03-01 20:14:52 +01:00
Fabian Reinartz 2c931950c6 Merge pull request #1441 from prometheus/scraperef9
Next iteration of retrieval refactoring
2016-03-01 17:16:43 +01:00
Fabian Reinartz 499f4af4aa Test target URL 2016-03-01 14:49:57 +01:00
Fabian Reinartz 50c2f20756 Add targetScraper tests 2016-03-01 14:33:28 +01:00
Fabian Reinartz 1ede7b9d72 Consolidate TargetStatus into Target.
This commit simplifies the TargetHealth type and moves the target
status into the target itself. This also removes a race where error
and last scrape time could have been out of sync.
2016-03-01 14:33:21 +01:00
Fabian Reinartz 2060a0a15b Turn target group members into plain lists.
As the scrape pool deduplicates targets now, it is no longer necessary
to store a hash map for members of each group.
2016-03-01 14:33:12 +01:00
Fabian Reinartz 0d7105abee Remove scrape config from Target.
This commit removes the scrapeConfig entirely from Target.
All identity defining parameters are thus immutable now and the mutex
can be removed..

Target identity is now correctly defined by the labels and the full URL.
This in particular includes URL parameters that are not specified in the
label set.

Fingerprint is also removed from hash to remove an unnecessary tight coupling
to the common/model package.
2016-03-01 14:32:57 +01:00
Fabian Reinartz 75681b691a Extract HTTP client from Target.
The HTTP client is the same across all targets with the same
scrape configuration. Thus, this commit moves it into the scrape
pool.
2016-03-01 14:31:57 +01:00
Fabian Reinartz cf56e33030 Merge pull request #1429 from prometheus/scraperef7
Next iteration of retrieval refactoring
2016-03-01 14:04:27 +01:00
Fabian Reinartz 9bea27ae8a Add scraping tests 2016-03-01 14:00:48 +01:00
Fabian Reinartz 76a8c6160d Deduplicate targets in scrape pool.
With this commit the scrape pool deduplicates incoming
targets before scraping them. This way multiple target providers
can produce the same target but it will be scraped only once.
2016-03-01 13:50:51 +01:00
Fabian Reinartz 84f74b9a84 Apply new scrape config on reload.
This commit updates a target set's scrape configuration
on reload. This will cause all running scrape loops to be
stopped and started again with new parameters.
2016-03-01 13:50:51 +01:00
Fabian Reinartz 02f635dc24 Remove interval/timeout from Target internals 2016-03-01 13:50:51 +01:00
Fabian Reinartz 775316f8d2 Move appender construction from Target to scrapePool 2016-03-01 13:50:51 +01:00
Fabian Reinartz fbe251c2df Fix scrape interval length calculation 2016-03-01 13:48:36 +01:00