mirror of
https://github.com/prometheus/prometheus.git
synced 2024-12-25 21:54:10 -08:00
fc6737b7fb
tl;dr: This is not a fundamental solution to the indexing problem
(like tindex is) but it at least avoids utilizing the intersection
problem to the greatest possible amount.
In more detail:
Imagine the following query:
nicely:aggregating:rule{job="foo",env="prod"}
While it uses a nicely aggregating recording rule (which might have a
very low cardinality), Prometheus still intersects the low number of
fingerprints for `{__name__="nicely:aggregating:rule"}` with the many
thousands of fingerprints matching `{job="foo"}` and with the millions
of fingerprints matching `{env="prod"}`. This totally innocuous query
is dead slow if the Prometheus server has a lot of time series with
the `{env="prod"}` label. Ironically, if you make the query more
complicated, it becomes blazingly fast:
nicely:aggregating:rule{job=~"foo",env=~"prod"}
Why so? Because Prometheus only intersects with non-Equal matchers if
there are no Equal matchers. That's good in this case because it
retrieves the few fingerprints for
`{__name__="nicely:aggregating:rule"}` and then starts right ahead to
retrieve the metric for those FPs and checking individually if they
match the other matchers.
This change is generalizing the idea of when to stop intersecting FPs
and go into "retrieve metrics and check them individually against
remaining matchers" mode:
- First, sort all matchers by "expected cardinality". Matchers
matching the empty string are always worst (and never used for
intersections). Equal matchers are in general consider best, but by
using some crude heuristics, we declare some better than others
(instance labels or anything that looks like a recording rule).
- Then go through the matchers until we hit a threshold of remaining
FPs in the intersection. This threshold is higher if we are already
in the non-Equal matcher area as intersection is even more expensive
here.
- Once the threshold has been reached (or we have run out of matchers
that do not match the empty string), start with "retrieve metrics
and check them individually against remaining matchers".
A beefy server at SoundCloud was spending 67% of its CPU time in index
lookups (fingerprintsForLabelPairs), serving mostly a dashboard that
is exclusively built with recording rules. With this change, it spends
only 35% in fingerprintsForLabelPairs. The CPU usage dropped from 26
cores to 18 cores. The median latency for query_range dropped from 14s
to 50ms(!). As expected, higher percentile latency didn't improve that
much because the new approach is _occasionally_ running into the worst
case while the old one was _systematically_ doing so. The 99th
percentile latency is now about as high as the median before (14s)
while it was almost twice as high before (26s).
|
||
|---|---|---|
| .github | ||
| cmd | ||
| config | ||
| console_libraries | ||
| consoles | ||
| documentation | ||
| notifier | ||
| promql | ||
| retrieval | ||
| rules | ||
| scripts | ||
| storage | ||
| template | ||
| util | ||
| vendor | ||
| web | ||
| .dockerignore | ||
| .gitignore | ||
| .promu.yml | ||
| .travis.yml | ||
| AUTHORS.md | ||
| CHANGELOG.md | ||
| circle.yml | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| Makefile | ||
| NOTICE | ||
| README.md | ||
| VERSION | ||
Prometheus 
Visit prometheus.io for the full documentation, examples and guides.
Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.
Prometheus' main distinguishing features as compared to other monitoring systems are:
- a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
- a flexible query language to leverage this dimensionality
- no dependency on distributed storage; single server nodes are autonomous
- timeseries collection happens via a pull model over HTTP
- pushing timeseries is supported via an intermediary gateway
- targets are discovered via service discovery or static configuration
- multiple modes of graphing and dashboarding support
- support for hierarchical and horizontal federation
Architecture overview
Install
There are various ways of installing Prometheus.
Precompiled binaries
Precompiled binaries for released versions are available in the releases section of the GitHub repository. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.
Debian packages are available.
Container images
Container images are available on Quay.io.
Building from source
To build Prometheus from the source code yourself you need to have a working Go environment with version 1.5 or greater installed.
You can directly use the go tool to download and install the prometheus
and promtool binaries into your GOPATH. We use Go 1.5's experimental
vendoring feature, so you will also need to set the GO15VENDOREXPERIMENT=1
environment variable in this case:
$ GO15VENDOREXPERIMENT=1 go get github.com/prometheus/prometheus/cmd/...
$ prometheus -config.file=your_config.yml
You can also clone the repository yourself and build using make:
$ mkdir -p $GOPATH/src/github.com/prometheus
$ cd $GOPATH/src/github.com/prometheus
$ git clone https://github.com/prometheus/prometheus.git
$ cd prometheus
$ make build
$ ./prometheus -config.file=your_config.yml
The Makefile provides several targets:
- build: build the
prometheusandpromtoolbinaries - test: run the tests
- format: format the source code
- vet: check the source code for common errors
- assets: rebuild the static assets
- docker: build a docker container for the current
HEAD
More information
- The source code is periodically indexed: Prometheus Core.
- You will find a Travis CI configuration in
.travis.yml. - All of the core developers are accessible via the Prometheus Developers Mailinglist and the
#prometheuschannel onirc.freenode.net.
Contributing
Refer to CONTRIBUTING.md
License
Apache License 2.0, see LICENSE.