The Prometheus monitoring system and time series database.
Find a file
Brian Brazil d782387f81
Stream symbols during compaction. (#6468)
Rather than buffer up symbols in RAM, do it one by one
during compaction. Then use the reader's symbol handling
for symbol lookups during the rest of the index write.

There is some slowdown in compaction, due to having to look through a file
rather than a hash lookup. This is noise to the overall cost of compacting
series with thousands of samples though.

benchmark                                                                                   old ns/op       new ns/op       delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        539917175       675341565       +25.08%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       2441815993      2477453524      +1.46%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       3978543559      3922909687      -1.40%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       8430219716      8586610007      +1.86%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      1786424591      1909552782      +6.89%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     5328998202      6020839950      +12.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     10085059958     11085278690     +9.92%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     25497010155     27018079806     +5.97%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               2427391406      2817217987      +16.06%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               2592965497      2538805050      -2.09%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               2437388343      2668012858      +9.46%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               2317095324      2787423966      +20.30%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               2600239857      2096973860      -19.35%

benchmark                                                                                   old allocs     new allocs     delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        500851         470794         -6.00%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       821527         791451         -3.66%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       1141562        1111508        -2.63%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       2141576        2111504        -1.40%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      871466         841424         -3.45%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     1941428        1911415        -1.55%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     3071573        3041510        -0.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     6771648        6741509        -0.45%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               731493         824888         +12.77%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               793918         887311         +11.76%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               811842         905204         +11.50%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               832244         925081         +11.16%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               921553         1019162        +10.59%

benchmark                                                                                   old bytes      new bytes      delta
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4        40532648       35698276       -11.93%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4       60340216       53409568       -11.49%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4       81087336       72065552       -11.13%
BenchmarkCompaction/type=normal,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4       142485576      120878544      -15.16%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=101-4      208661368      203831136      -2.31%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=1001-4     347345904      340484696      -1.98%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=2001-4     585185856      576244648      -1.53%
BenchmarkCompaction/type=vertical,blocks=4,series=10000,samplesPerSeriesPerBlock=5001-4     1357641792     1358966528     +0.10%
BenchmarkCompactionFromHead/labelnames=1,labelvalues=100000-4                               126486664      119666744      -5.39%
BenchmarkCompactionFromHead/labelnames=10,labelvalues=10000-4                               122323192      115117224      -5.89%
BenchmarkCompactionFromHead/labelnames=100,labelvalues=1000-4                               126404504      119469864      -5.49%
BenchmarkCompactionFromHead/labelnames=1000,labelvalues=100-4                               119047832      112230408      -5.73%
BenchmarkCompactionFromHead/labelnames=10000,labelvalues=10-4                               136576016      116634800      -14.60%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-12-17 19:49:54 +00:00
.circleci Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
.github disable github actions (#6207) 2019-10-23 23:37:53 +03:00
cmd *: avoid missed Alertmanager targets (#6455) 2019-12-12 17:00:19 +01:00
config Add config option for remote job name (#6043) 2019-12-12 12:47:23 -08:00
console_libraries Added humanizePercentage formatting to templates (#5670) 2019-06-15 08:59:57 +01:00
consoles Update console template example metric names. (#5678) 2019-06-17 12:13:29 +01:00
discovery discovery/kubernetes: fix client metrics 2019-11-29 12:34:36 +01:00
docs Merge pull request #6442 from gotjosh/single-metric-api 2019-12-11 18:21:16 +01:00
documentation Simplify remote write dashboard in mixin. 2019-11-18 19:58:07 -08:00
notifier *: avoid missed Alertmanager targets (#6455) 2019-12-12 17:00:19 +01:00
pkg Add exemplar support to the openmetrics parser (#6292) 2019-11-19 09:33:30 +00:00
prompb Pushdown aggregator group by through read hint (#6401) 2019-12-05 14:06:28 +00:00
promql PromQL: Parse Metrics using generated parser (#6466) 2019-12-16 16:38:18 +00:00
rules *: improve error messages when parsing bad rules (#5965) 2019-08-28 17:36:48 +02:00
scrape Make the scrape.metricMetadataStore interface public 2019-12-05 10:29:58 +00:00
scripts React UI: Serve React UI under /new (#6229) 2019-10-28 10:45:53 +01:00
storage Add config option for remote job name (#6043) 2019-12-12 12:47:23 -08:00
template template: remove redundant parentheses. (#5904) 2019-08-16 09:41:01 +01:00
tsdb Stream symbols during compaction. (#6468) 2019-12-17 19:49:54 +00:00
util util/promlint: simplify sorting function (#6448) 2019-12-11 15:45:26 +01:00
vendor upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00
web Load only some offsets into the symbol table into memory. 2019-12-17 18:56:58 +00:00
.dockerignore Docker images for ARM32v7 and ARM64v8 (#5031) 2019-04-15 14:27:00 +02:00
.gitignore Add tsdb/tsdb to .gitignore 2019-11-15 12:41:37 +01:00
.golangci.yml *: enable all default linters (#5504) 2019-05-03 15:11:28 +02:00
.promu.yml Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
CHANGELOG.md Cut 2.14.0 release (#6298) 2019-11-11 15:20:12 +01:00
code-of-conduct.md Add CNCF code of conduct as the Prometheus code of conduct 2016-10-19 21:39:19 +02:00
CONTRIBUTING.md *: bump Go version to 1.13 (#6020) 2019-09-13 13:55:18 +01:00
Dockerfile Integrate beginning of React UI (#5694) 2019-10-17 14:38:09 +02:00
fuzzit.sh Fix fuzzit job (#6005) 2019-09-11 09:08:20 +02:00
go.mod upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00
go.sum upgrade aws-sdk-go (#6429) 2019-12-09 23:16:18 +00:00
LICENSE Clean up license issues. 2015-01-21 20:07:45 +01:00
MAINTAINERS.md Update and re-work maintainers file. 2019-11-28 19:01:35 +00:00
Makefile Add unit tests for react-app (#6234) 2019-10-28 18:09:48 +01:00
Makefile.common Bump golangci-lint to v1.18.0 (#6004) 2019-09-10 17:40:08 +02:00
NOTICE Integrate beginning of React UI (#5694) 2019-10-17 14:38:09 +02:00
README.md Remove Travis as a CI platform (#6369) 2019-11-25 14:59:26 +01:00
RELEASE.md Volunteering for release shepharding (#6384) 2019-11-27 11:45:23 -07:00
VERSION Cut 2.14.0 release (#6298) 2019-11-11 15:20:12 +01:00

Prometheus

CircleCI Docker Repository on Quay Docker Pulls Go Report Card CII Best Practices fuzzit

Visit prometheus.io for the full documentation, examples and guides.

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Prometheus's main distinguishing features as compared to other monitoring systems are:

  • a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
  • a flexible query language to leverage this dimensionality
  • no dependency on distributed storage; single server nodes are autonomous
  • timeseries collection happens via a pull model over HTTP
  • pushing timeseries is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • support for hierarchical and horizontal federation

Architecture overview

Install

There are various ways of installing Prometheus.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.

Debian packages are available.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch a Prometheus container for trying it out with

$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Prometheus will now be reachable at http://localhost:9090/.

Building from source

To build Prometheus from the source code yourself you need to have a working Go environment with version 1.13 or greater installed. You will also need to have Node.js and Yarn installed in order to build the frontend assets.

You can directly use the go tool to download and install the prometheus and promtool binaries into your GOPATH:

$ go get github.com/prometheus/prometheus/cmd/...
$ prometheus --config.file=your_config.yml

However, when using go get to build Prometheus, Prometheus will expect to be able to read its web assets from local filesystem directories under web/ui/static and web/ui/templates. In order for these assets to be found, you will have to run Prometheus from the root of the cloned repository. Note also that these directories do not include the new experimental React UI unless it has been built explicitly using make assets or make build.

An example of the above configuration file can be found here.

You can also clone the repository yourself and build using make build, which will compile in the web assets so that Prometheus can be run from anywhere:

$ mkdir -p $GOPATH/src/github.com/prometheus
$ cd $GOPATH/src/github.com/prometheus
$ git clone https://github.com/prometheus/prometheus.git
$ cd prometheus
$ make build
$ ./prometheus --config.file=your_config.yml

The Makefile provides several targets:

  • build: build the prometheus and promtool binaries (includes building and compiling in web assets)
  • test: run the tests
  • test-short: run the short tests
  • format: format the source code
  • vet: check the source code for common errors
  • docker: build a docker container for the current HEAD

React UI Development

For more information on building, running, and developing on the new React-based UI, see the React app's README.md.

More information

  • The source code is periodically indexed: Prometheus Core.
  • You will find a CircleCI configuration in .circleci/config.yml.
  • See the Community page for how to reach the Prometheus developers and users on various communication channels.

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.