The Prometheus monitoring system and time series database.
Find a file
Paschalis Tsilias d1122e0743
Introduce TSDB changes for appending metadata to the WAL (#10972)
* Append metadata to the WAL

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove extra whitespace; Reword some docstrings and comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use RLock() for hasNewMetadata check

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use single byte for metric type in RefMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update proposed WAL format for single-byte type metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implementa MetadataAppender interface for the Agent

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of review comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Amend description of metadata in wal.md

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Correct key used to retrieve metadata from cache

When we're setting metadata entries in the scrapeCace, we're using the
p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and
use it as the cache key. When checking for cache entries though, we used
p.Series() as the key, which included the metric name _with_ its labels.
That meant that we were never actually hitting the cache. We're fixing
this by utiling the __name__ internal label for correctly getting the
cache entries after they've been set by setHelp(), setType() or
setUnit().

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Put feature behind a feature flag

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix AppendMetadata docstring

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reorder WAL format document

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse temporary buffer in Metadata encoder

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Only keep latest metadata for each refID during checkpointing

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test that's referencing decoding metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Avoid creating metadata block if no new metadata are present

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for corrupt metadata block and relevant record type

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix CR comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Extract logic about changing metadata in an anonymous function

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implement new proposed WAL format and amend relevant tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use 'const' for metadata field names

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Apply metadata to head memSeries in Commit, not in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add docstring and rename extracted helper in scrape.go

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for tsdb-related cases

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol1

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol2

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix Windows test by closing WAL reader files

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use switch instead of two if statements in metadata decoding

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments around TestMetadata* tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add code for replaying WAL; test correctness of in-memory data after a replay

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove scrape-loop related code from PR

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify tests by sorting slices before comparison

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test to use separate transactions

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Empty out buffer and record slices after encoding latest metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linting issue

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update calculation for DroppedMetadata metric

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse buffer when encoding latest metadata for each series

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments; Check all returned error values using two helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify use of helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Satisfy linter

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-07-19 10:58:52 +02:00
.circleci Check syntax of example configurations (#10753) 2022-05-26 18:17:21 +10:00
.github add missing npm registry url (#10987) 2022-07-05 16:38:16 +02:00
cmd Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
config feat(nomad): add nomad service discovery 2022-06-30 14:17:08 +02:00
console_libraries Make React UI the default, keep old UI under /classic (#8142) 2020-11-03 14:51:48 +01:00
consoles Cleaned up a little bit of HTML 2021-07-28 20:12:06 -04:00
discovery nomad: Use failuresCount counter 2022-06-30 14:19:58 +02:00
docs template: adding formatTime function to TemplateExpander (#10993) 2022-07-14 23:45:32 +02:00
documentation documentation: fix example dependencies 2022-07-13 20:50:10 +02:00
model Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
notifier Improve notifier queue test to reduce flakiness (#10984) 2022-07-05 15:27:26 +02:00
plugins feat(nomad): add nomad service discovery 2022-06-30 14:17:08 +02:00
prompb Amend Protobuf docstrings referring to model/ packages 2022-04-16 00:37:16 +02:00
promql Prettifier: Add spaces with non-callable keywords (#11005) 2022-07-15 00:09:56 +02:00
rules refactor (rules): move from github.com/pkg/errors to 'errors' and 'fmt' (#10855) 2022-06-17 09:54:25 +02:00
scrape Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
scripts Fix UI release script and bump UI release (#10981) 2022-07-05 15:12:59 +02:00
storage Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
template template: adding formatTime function to TemplateExpander (#10993) 2022-07-14 23:45:32 +02:00
tracing refactor (tracing): move from github.com/pkg/errors to 'errors' and 'fmt' (#10819) 2022-06-08 10:44:42 +02:00
tsdb Introduce TSDB changes for appending metadata to the WAL (#10972) 2022-07-19 10:58:52 +02:00
util refactor (util): move from github.com/pkg/errors to 'errors' and 'fmt' (#10877) 2022-06-27 18:16:58 +02:00
web Merge pull request #11022 from prometheus/release-2.37 2022-07-14 18:30:45 +02:00
.dockerignore Add image build for ppc64le architecture 2020-04-06 18:03:58 -03:00
.gitignore Replace vfsgen with embed for static web assets (#10220) 2022-03-09 10:21:31 +01:00
.gitpod.Dockerfile add gitpod dockerfile to upgrade node and npm version 2021-09-27 18:59:41 +02:00
.gitpod.yml fix gitpod by using custome dockerfile and accurate npm ui path 2021-09-27 18:59:41 +02:00
.golangci.yml refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
.promu.yml Drop OpenBSD from promu config (#10867) 2022-06-16 10:36:28 +02:00
.yamllint Relax yamllint 2022-05-06 09:59:59 +02:00
CHANGELOG.md Release 2.37.0 2022-07-14 16:02:48 +02:00
CODE_OF_CONDUCT.md Update link for referenced CNCF code of conduct (#10664) 2022-05-03 18:32:23 +02:00
CONTRIBUTING.md fix markdown lint issues (#10591) 2022-05-03 10:59:09 +02:00
Dockerfile Dockerfile: Optimize and consolidate steps (#9180) 2021-09-30 11:13:44 +02:00
go.mod Merge pull request #11022 from prometheus/release-2.37 2022-07-14 18:30:45 +02:00
go.sum Merge pull request #11022 from prometheus/release-2.37 2022-07-14 18:30:45 +02:00
LICENSE
MAINTAINERS.md Remove Jessica from maintainers list (#10731) 2022-05-24 09:21:46 +02:00
Makefile enable ui module publication (#10876) 2022-06-30 12:10:10 +02:00
Makefile.common remove GO111MODULE and GOVENDOR usage (#10836) 2022-06-13 13:08:59 +02:00
NOTICE Add license notice for code adapted from Go 2021-12-05 09:01:52 +01:00
plugins.yml feat(nomad): add nomad service discovery 2022-06-30 14:17:08 +02:00
README.md Make clear that go install needs to be run in the cloned repo (#10866) 2022-06-13 22:00:27 +02:00
RELEASE.md enable ui module publication (#10876) 2022-06-30 12:10:10 +02:00
SECURITY.md fix markdown lint issues (#10591) 2022-05-03 10:59:09 +02:00
VERSION Release 2.37.0 2022-07-14 16:02:48 +02:00

Prometheus

CircleCI Docker Repository on Quay Docker Pulls Go Report Card CII Best Practices Gitpod ready-to-code Fuzzing Status

Visit prometheus.io for the full documentation, examples and guides.

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

The features that distinguish Prometheus from other metrics and monitoring systems are:

  • A multi-dimensional data model (time series defined by metric name and set of key/value dimensions)
  • PromQL, a powerful and flexible query language to leverage this dimensionality
  • No dependency on distributed storage; single server nodes are autonomous
  • An HTTP pull model for time series collection
  • Pushing time series is supported via an intermediary gateway for batch jobs
  • Targets are discovered via service discovery or static configuration
  • Multiple modes of graphing and dashboarding support
  • Support for hierarchical and horizontal federation

Architecture overview

Architecture overview

Install

There are various ways of installing Prometheus.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch a Prometheus container for trying it out with

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Prometheus will now be reachable at http://localhost:9090/.

Building from source

To build Prometheus from source code, You need:

Start by cloning the repository:

git clone https://github.com/prometheus/prometheus.git
cd prometheus

You can use the go tool to build and install the prometheus and promtool binaries into your GOPATH:

GO111MODULE=on go install github.com/prometheus/prometheus/cmd/...
prometheus --config.file=your_config.yml

However, when using go install to build Prometheus, Prometheus will expect to be able to read its web assets from local filesystem directories under web/ui/static and web/ui/templates. In order for these assets to be found, you will have to run Prometheus from the root of the cloned repository. Note also that these directories do not include the React UI unless it has been built explicitly using make assets or make build.

An example of the above configuration file can be found here.

You can also build using make build, which will compile in the web assets so that Prometheus can be run from anywhere:

make build
./prometheus --config.file=your_config.yml

The Makefile provides several targets:

  • build: build the prometheus and promtool binaries (includes building and compiling in web assets)
  • test: run the tests
  • test-short: run the short tests
  • format: format the source code
  • vet: check the source code for common errors
  • assets: build the React UI

Service discovery plugins

Prometheus is bundled with many service discovery plugins. When building Prometheus from source, you can edit the plugins.yml file to disable some service discoveries. The file is a yaml-formated list of go import path that will be built into the Prometheus binary.

After you have changed the file, you need to run make build again.

If you are using another method to compile Prometheus, make plugins will generate the plugins file accordingly.

If you add out-of-tree plugins, which we do not endorse at the moment, additional steps might be needed to adjust the go.mod and go.sum files. As always, be extra careful when loading third party code.

Building the Docker image

The make docker target is designed for use in our CI system. You can build a docker image locally with the following commands:

make promu
promu crossbuild -p linux/amd64
make npm_licenses
make common-docker-amd64

NB if you are on a Mac, you will need gnu-tar.

Using Prometheus as a Go Library

Remote Write

We are publishing our Remote Write protobuf independently at buf.build.

You can use that as a library:

go get go.buf.build/protocolbuffers/go/prometheus/prometheus

This is experimental.

Prometheus code base

In order to comply with go mod rules, Prometheus release number do not exactly match Go module releases. For the Prometheus v2.y.z releases, we are publishing equivalent v0.y.z tags.

Therefore, a user that would want to use Prometheus v2.35.0 as a library could do:

go get github.com/prometheus/prometheus@v0.35.0

This solution makes it clear that we might break our internal Go APIs between minor user-facing releases, as breaking changes are allowed in major version zero.

React UI Development

For more information on building, running, and developing on the React-based UI, see the React app's README.md.

More information

  • Godoc documentation is available via pkg.go.dev. Due to peculiarities of Go Modules, v2.x.y will be displayed as v0.x.y.
  • You will find a CircleCI configuration in .circleci/config.yml.
  • See the Community page for how to reach the Prometheus developers and users on various communication channels.

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.