Commit graph

1456 commits

Author SHA1 Message Date
Julius Volz 6717f6b2f4 Add architecture overview diagram.
Change-Id: I860a748be534b2b9b0b60a9dd2e4436b78cc3907
2014-12-07 01:22:03 +01:00
juliusv 5b942982b7 Fix another README.md typo. 2014-12-07 00:09:09 +01:00
juliusv cc83fdc9c3 Make README.md introduction less esoteric. 2014-12-07 00:05:10 +01:00
juliusv a0457829ee Some more README.md updates. 2014-12-06 23:52:19 +01:00
juliusv c1b0237d4b Fix typo in README.md 2014-12-06 23:46:33 +01:00
Björn Rabenstein f71c215454 Merge pull request #415 from peterbourgon/fix-dockerfile
Fix Dockerfile for new repo layout
2014-12-06 13:26:32 +01:00
Bjoern Rabenstein 89bb376bce Reduce lock-protected area during scrape.
Change-Id: Iaa7faa7c916b1890b568d05bd8bfff6299b6767d
2014-12-05 19:40:41 +01:00
Peter Bourgon 88019e852a Use ENTRYPOINT and CMD 2014-12-04 20:47:17 +01:00
Bjoern Rabenstein fee88a7a77 Remove the remaining races, new and old.
Also, resolve a few other TODOs.

Change-Id: Icb39b5a5e8ca22ebcb48771cd8951c5d9e112691
2014-12-03 18:07:23 +01:00
Peter Bourgon ec4182c8c7 Remove newline at end of Dockerfile 2014-12-03 15:02:37 +01:00
Peter Bourgon dde4209144 Fix Dockerfile for new repo layout 2014-12-03 14:53:35 +01:00
Bjoern Rabenstein c0e5ac35f0 Remove spurious quotes in Makefile.INCLUDE.
Change-Id: Ic75f00e73dbfdeaa8419f5b22789fe24c84ec606
2014-11-28 16:16:29 +01:00
Bjoern Rabenstein 7c3bb145e1 Bump VERSION to 0.9.0rc1.
Change-Id: Ie1a02d026290e9ecee83990b7baa2cd730df6869
2014-11-28 16:11:49 +01:00
Björn Rabenstein ee9b32cd13 Merge pull request #413 from prometheus/cleanup/do-some-todos
Cleanup/do some todos
2014-11-28 12:53:20 +01:00
Bjoern Rabenstein 66c80b5ebd Fix typo.
Change-Id: I72608c7841c00145458807d3c3ee29db7b5ac2bc
2014-11-28 12:50:19 +01:00
Bjoern Rabenstein 674624f1c8 Completed more TODOs.
- Documented checkpoint file format.
- High-level description of series sanitation.
- Replace fp.LoadFromString panic with an error.
  (Change in client_golang already submitted.)
- Introduced checks for series file size where appropriate.
- Removed two Law of Demeter violations.

Change-Id: I555d97a2c8f4769820c2fc8bf5d6f4e160222abc
2014-11-27 20:46:45 +01:00
Bjoern Rabenstein 7d11019aa2 Squash a few trivial TODOs.
- Delete unneeded file view_adapter.go.
- Assessed that we still need the fingerprints in nodes
  (to create iterators).
- Turned numMemChunkDescs into a metric.

Change-Id: I29be963c795a075ec00c095f76bf26405535609d
2014-11-27 18:26:06 +01:00
Bjoern Rabenstein 49683c0c20 Avoid test flags in normal binary.
Change-Id: If1fba813a73bf93ea5918dcda326e3ffa81a797d
2014-11-27 18:04:48 +01:00
Bjoern Rabenstein ea86f7e8f8 Fix weird things after merge.
And I swear I'll never use 'rebase' to 'clean something up' ever agin,
even if Julius tells me to do so...

Change-Id: Ifeabab20445279bf693c95f062da769b60fe195f
2014-11-25 17:39:02 +01:00
Bjoern Rabenstein 3a17aeabfd Merge branch 'beorn/storage-ng-with-commit-history-cleaned-up'
Conflicts:
	Makefile
	Makefile.INCLUDE
	VERSION
	main.go
	notification/notification.go
	retrieval/target.go
	retrieval/target_test.go
	retrieval/targetmanager.go
	retrieval/targetmanager_test.go
	retrieval/targetpool.go
	retrieval/targetpool_test.go
	rules/ast/functions.go
	rules/rules_test.go
	storage/metric/interface.go
	storage/metric/tiered/curator.go
	storage/metric/tiered/end_to_end_test.go
	storage/metric/tiered/leveldb.go
	storage/metric/tiered/memory.go
	storage/metric/tiered/memory_test.go
	storage/metric/tiered/tiered.go
	storage/remote/queue_manager.go
	templates/templates.go
	templates/templates_test.go
	web/api/query.go
	web/consoles.go
	web/web.go

Change-Id: I96e6312b51e877d4434fe96c494e9558fe2e1d16
2014-11-25 17:36:17 +01:00
Bjoern Rabenstein 9bc05052ad Add line that has mysteriously disappeared after rebase.
Change-Id: I3612eb0b626e66e607b363e9801f187d2ba637a3
2014-11-25 17:15:56 +01:00
Julius Volz 1491fb9370 Update GeneratorURL field name in notification tests.
Change-Id: Ic4357999b6ebcf54008869a395e56d12a0ead211
2014-11-25 17:13:04 +01:00
Julius Volz af8c6df94d Eliminate modal alerts in graphing UI.
This shows errors in a pane under the expression input instead.

Change-Id: Iec209e1628a3b102cce9f34b2467621772dfb8ff
2014-11-25 17:13:04 +01:00
Julius Volz 5e8d57bec1 Make tabular view the default (vs. graphing view).
Change-Id: I9f0961f2c474e8cce5e376ce4e20040644f89370
2014-11-25 17:13:04 +01:00
Julius Volz 6eecee55b7 Fix acronym caps in GeneratorURL.
Change-Id: Ib18c1f617dcde1039e848059545a6d8831d9bf66
2014-11-25 17:13:04 +01:00
Bjoern Rabenstein 0ae1d8889a Fix tests after merge.
Change-Id: Ia90da9a3e48ed780ec38c4a6a1fd9ea34e7f6a58
2014-11-25 17:13:04 +01:00
Julius Volz b7bf11230a Add absent() function.
A common problem in Prometheus alerting is to detect when no timeseries
exist for a given metric name and label combination. Unfortunately,
Prometheus alert expressions need to be of vector type, and
"count(nonexistent_metric)" results in an empty vector, yielding no
output vector elements to base an alert on. The newly introduced
absent() function solves this issue:

  ALERT FooAbsent IF absent(foo{job="myjob"}) [...]

absent() has the following behavior:

- if the vector passed to it has any elements, it returns an empty
  vector.

- if the vector passed to it has no elements, it returns a 1-element
  vector with the value 1.

In the second case, absent() tries to be smart about deriving labels of
the 1-element output vector from the input vector:

  absent(nonexistent{job="myjob"})                   => {job="myjob"}
  absent(nonexistent{job="myjob",instance=~".*"})    => {job="myjob"}
  absent(sum(nonexistent{job="myjob"}))              => {}

That is, if the passed vector is a literal vector selector, it takes all
"=" label matchers as the basis for the output labels, but ignores all
non-equals or regex matchers. Also, if the passed vector results from a
non-selector expression, no labels can be derived.

Change-Id: I948505a1488d50265ab5692a3286bd7c8c70cd78
2014-11-25 17:13:04 +01:00
Julius Volz 3d47f94149 Drop metric names after transformations.
After many transformations, it doesn't make sense to keep the metric
names, since the result of the transformation is no longer that metric.
This drops the metric name after such transformations and makes the web
UI deal well with missing metric names.

This depends on the current branch on the following things:

- prometheus/client_golang needs to be at
  e237cf15c6
  in branch "julius/int-fingerprints" (to be merged with new storage)

- prometheus/promdash needs to be at
  dd7691c9c2

Change-Id: Ib3c8cad8d647d9854e8c653c424b8c235ccc231d
2014-11-25 17:13:04 +01:00
Bjoern Rabenstein 53c0a43754 Update README.md.
Change-Id: Ife51ed266333ff8bc228260187ea4bcf377bde31
2014-11-25 17:13:04 +01:00
Bjoern Rabenstein 14bda4180c Changes after pair code review.
Change-Id: Ib72d40f8e9027818cfbbd32a7a7201eebda07455
2014-11-25 17:12:59 +01:00
Bjoern Rabenstein a2feed343a Convert another occurrence from chan bool to chan struct{}.
Change-Id: I11ba127a934ee3aec0fcd139ad32a7751cff77a0
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 006b5517e2 Simplify makefiles.
This removes the dependancy on C leveldb and snappy.
It also takes care of fewer dependencies as they would
anyway not work on any non-Debian, non-Brew system.

Change-Id: Ia70dce1ba8a816a003587927e0b3a3f8ad2fd28c
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 9ea808cd8b Remove debug log line.
Change-Id: Icdd2351b89f2d37ac2b615f9cf872e054c694ad1
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein bb42cc2e2d Evict based on memory pressure. Evict recently used chunks last.
Change-Id: Ie6168f0cdb3917bdc63b6fe15585dd70c1e42afe
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein e23ee0f7cc Fix race in test.
Change-Id: I53e1a4c5a6b5f846acd76043166b6cb7bf7d5dc7
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein d73e851b14 Tweak timing in the maintenance loop.
Change-Id: I9801c4f9a22c3b3dc1ce1af81fdd9e992a4f4dd7
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 2672aa8ece Instrument series maintenance.
Change-Id: Ie4269d07ad4d23d44230c95a523088b472718e54
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 371445f4da Fix typo in comment.
Change-Id: Ib3ee493bb29e7e9967214627e543baedb897ab67
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 74c143c4c9 Improve scraper shutdown time.
- Stop target pools in parallel.
- Stop individual scrapers in goroutines, too.
- Timing tweaks.

Change-Id: I9dff1ee18616694f14b04408eaf1625d0f989696
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 92156ee89d Drain the newBaseLabels channel upon shutdown.
This should help cut down shutdown times.

Change-Id: I6e70a598a9e49aa6eeeb2034105b1bc6e9014324
2014-11-25 17:10:39 +01:00
Bjoern Rabenstein 3f61d304ce Reorganize maintenance loop.
Change-Id: Iac10f988ba3e93ffb188f49c30f92e0b6adce5a3
2014-11-25 17:10:30 +01:00
Bjoern Rabenstein a5f56639b8 Instrument unwritten samples queue.
Change-Id: Id77387314d340a5118490cf08e7bbc37c7366b25
2014-11-25 17:10:30 +01:00
Bjoern Rabenstein c087ee35f7 Remove archiveMtx.
Change-Id: Ie8019f860bbda68621f74380c90a4e57930d3d7a
2014-11-25 17:10:30 +01:00
Bjoern Rabenstein 7af42eda65 Optimize purging.
Now only purge if there is something to purge.
Also, set savedFirstTime and archived time range appropriately.
(Which is needed for the optimization.)

Change-Id: Idcd33319a84def3ce0318d886f10c6800369e7f9
2014-11-25 17:10:30 +01:00
Bjoern Rabenstein 33b959b898 Persist savedFirstTime in checkpoint.
Change-Id: Ibdfdea16fad0608ec104fbccc749e824a171f227
2014-11-25 17:10:30 +01:00
Bjoern Rabenstein 904acd43da Add crash recovery.
Fix the behavior if preload for non-existent series is requested.

Instead of returning an error (which triggers a panic further up),
simply count those incidents. They can happen regularly, we just want
to know if they happen too frequently because that would mean the
indexing is behind or broken.

Change-Id: I4b2d1b93c4146eeea897d188063cb9574a270f8b
2014-11-25 17:09:43 +01:00
Bjoern Rabenstein 7a9efc9c59 Fix typo in test.
Change-Id: I3c2fd76bc5f50446c58f8ef693d9c6595197feaa
2014-11-25 17:09:43 +01:00
Bjoern Rabenstein 4efc60174b Tweak and verify a few parameters.
Remove TODOs accordingly.

Change-Id: Ic062e13b6ae89a9135d3f14011114fe1cca1cef8
2014-11-25 17:09:43 +01:00
Bjoern Rabenstein 5f8e9617ef Add more tests.
Add an end-to-end fuzz and race test.

Fix a race exposed by the above.

Change-Id: Ifaa39a90cefbde8d4c29bda197cc92592ded21bb
2014-11-25 17:09:17 +01:00
Bjoern Rabenstein d215e013b7 Fix the weird chunkDesc shuffling bug.
The root cause was that after chunkDesc eviction, the offset between
memory representation of chunk layout (via chunkDescs in memory) was
shiftet against chunks as layed out on disk. Keeping the offset up to
date is by no means trivial, so this commit is pretty involved.

Also, found a race that for some reason didn't bite us so far:
Persisting chunks was completel unlocked, so if chunks were purged on
disk at the same time, disaster would strike. However, locking the
persisting of chunk revealed interesting dead locks. Basically, never
queue under the fp lock.

Change-Id: I1ea9e4e71024cabbc1f9601b28e74db0c5c55db8
2014-11-25 17:09:17 +01:00