Commit graph

491 commits

Author SHA1 Message Date
Martin Disibio 1bcd13d6b5
Exemplar resize (#8974)
* Create experimental circular buffer resize method, benchmarks

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Optimize exemplar resize to only replay as many exemplars as needed

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* More comments, benchmark AddExemplar

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* optimizations

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* comment

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Slight refactor of resize benchmark + make use of resize via runtime
reloadable storage config.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Some more config related changes.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Refactor to remove usage of noopExemplarStorage and avoid race condition
when resizing from Head code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix or add comments to clarify some of the new behaviour.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* fix potential panics related to negative exemplar buffer lengths

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
2021-07-20 10:22:57 +05:30
Levi Harrison 3b5257d869
Changed disabled_features to feature_flags
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-07-13 22:03:51 -04:00
Filip Petkovski 7c125aa5fb
Promtool: Add support for compaction analysis (#8940)
* Extend promtool to support compaction analysis

This commit extends the promtool tsdb analyze command to help
troubleshoot high Prometheus disk usage. The command now plots a
distribution of how full chunks are relative to the maximum capacity of
120 samples per chunk.

Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>

* Update cmd/promtool/tsdb.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-07-02 11:08:52 +01:00
Julius Volz 441e6cd7d6
Merge release-2.28 back into main (#9035)
* Cut v2.28.0-rc.0 (#8954)

* Cut v2.28.0-rc.0

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Changelog fixup

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Address review comments

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Downgrade some features to enhancements

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Adjust release date to today

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Migrate HTTP SD docs from docs repo (#8972)

See discussion in https://github.com/prometheus/docs/pull/1975

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Cut Prometheus v2.28.0 (#8973)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* HTTP SD: Allow charset in content type (#8981)

* Added content type regex

Signed-off-by: Levi Harrison <git@leviharrison.dev>
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* fixed disappeared target groups in http_sd #9019

Signed-off-by: servak <fservak@gmail.com>

* Add a testcase for http-sd

Signed-off-by: servak <fservak@gmail.com>

* HTTP SD: Simplify logic of disappeared targetgroups (#9026)

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Fix logging after the move to go-kit/log (#9021)

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Cut Prometheus v2.28.1 (#9034)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Co-authored-by: Levi Harrison <git@leviharrison.dev>
Co-authored-by: Julien Pivotto <roidelapluie@inuits.eu>
Co-authored-by: servak <fservak@gmail.com>
2021-07-01 18:02:13 +02:00
Levi Harrison 90976e7505
Promtool: Add feature flags to unit tests (#8958)
* Added feature flag support to unit tests

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added/fixed tests

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Addressed review comments

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-30 22:43:39 +01:00
Ankit Goel d437cee73a
Move storage.tsdb.retention.size out of experimental #8728 (#9004)
* Move storage.tsdb.retention.size out of experimental #8728

Signed-off-by: Ankit Goel <ankit.goel@deliveryhero.com>
2021-06-30 01:30:11 +02:00
Levi Harrison ca1896c15b
Promtool: Validate service discovery files (#8950)
* Check SD files in promtool

Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-29 17:32:59 +02:00
Steve Kuznetsov fd6c852567
promtool: backfill: allow configuring block duration (#8919)
* promtool: backfill: allow configuring block duration

When backfilling large amounts of data across long periods of time, it
may in certain circumstances be useful to use a longer block duration to
increase the efficiency and speed of the backfilling process. This patch
adds a flag --block-duration-power to allow a user to choose the power N
where the block duration is 2^(N+1)h.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* promtool: use sub-tests in backfill testing

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* backfill: add messages to tests for clarity

When someone new breaks a test, seeing "expected: false, got: true" is
really not useful. A nice message helps here.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* backfill: test long block durations

A test that uses a long block duration to write bigger blocks is added.
The check to make sure all blocks are the default duration is removed.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2021-06-29 14:53:38 +05:30
Ben Kochie 7cb55d5732
Merge pull request #8802 from mwasilew2/yaml-linting
Adds yamllinting to Makefile.common
2021-06-24 15:59:35 +02:00
Julien Pivotto ba76bceb6b
Merge pull request #8917 from stevekuznetsov/skuznets/silence-backfill
promtool: backfill: allow silencing output
2021-06-14 23:27:18 +02:00
Michal Wasilewski 3f686cad8b
fixes yamllint errors
Signed-off-by: Michal Wasilewski <mwasilewski@gmx.com>
2021-06-12 12:47:47 +02:00
Levi Harrison b5f6f8fb36 Switched to go-kit/log
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-11 12:28:36 -04:00
Steve Kuznetsov ee771a2a66
promtool: backfill: allow silencing output
When using the backfill command to add data to an ephemeral/test
Prometheus instance, it is not important to see which data was added as
it is often generated ahead of time and mostly irrelevant to the
use-case. The current approach prints information about each block that
is written, but does so in a generally inefficient and costly manner.
This patch adds a `--quiet` flag that allows a user to opt out of this
behavior.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2021-06-10 15:31:16 -07:00
Levi Harrison 7bc11dcb06
React UI: Add Starting Screen (#8662)
* Added walreplay API endpoint

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added starting page to react-ui

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Documented the new endpoint

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed typos

Signed-off-by: Levi Harrison <git@leviharrison.dev>

Co-authored-by: Julius Volz <julius.volz@gmail.com>

* Removed logo

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed isResponding to isUnexpected

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed width of progress bar

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed width of progress bar

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added DB stats object

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Updated starting page to work with new fields

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil (pt. 2)

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil (pt. 3)

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil (and also implementing a method this time) (pt. 4)

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil (and also implementing a method this time) (pt. 5)

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed const to let

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Passing nil (pt. 6)

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Remove SetStats method

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Added comma

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed api

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed to triple equals

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed data response types

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Don't return pointer

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Changed version

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed interface issue

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed pointer

Signed-off-by: Levi Harrison <git@leviharrison.dev>

* Fixed copying lock value error

Signed-off-by: Levi Harrison <git@leviharrison.dev>

Co-authored-by: Julius Volz <julius.volz@gmail.com>
2021-06-05 15:29:32 +01:00
Levi Harrison 17ea8d006a
Added external URL access
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-05-30 23:35:26 -04:00
Bartlomiej Plotka 80545bfb2e
Instrumented circular exemplar storage. (#8712)
* Instrumented circular storage.

Fixes: https://github.com/prometheus/prometheus/issues/8708
Fixes: https://github.com/prometheus/prometheus/issues/8707

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed CB.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Julien comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Callum comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-04-16 13:44:53 +01:00
nberkley f9e2dd0697
Add support for smaller block chunk segment allocations (#8478)
* Add support for --storage.tsdb.max-chunk-size to suport small chunks for space limited prometheus instances.

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update tsdb/compact.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update tsdb/db.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update cmd/prometheus/main.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Change naming scheme to

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Add a lower bound to --storage.tsdb.max-block-chunk-segment-size

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update storage.md to explain what a chunk segment is

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Apply suggestions from code review

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Force tests

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Fix code style

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-04-15 14:25:01 +05:30
Julien Pivotto ae73a6296a
Merge pull request #8683 from cuirunxing-hub/main
typos correct
2021-04-02 20:14:55 +02:00
cuirunxing-hub 57bc2e94e2 typos correct
Signed-off-by: cuirunxing-hub <cuirunxing@inspur.com>
2021-04-02 09:03:00 +08:00
Jess G 731545ad34
Add documentation for recording rule backfiller (#8674)
* add docs for rule backfiller

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-04-01 22:38:00 +02:00
Julien Pivotto e635ca834b Add environment variable expansion in external label values
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-30 01:36:28 +02:00
Björn Rabenstein 9549a15c6f
Merge pull request #7675 from JessicaGreben/jg/11-retroactive-rule-eval
Add rule importer to backfill
2021-03-29 19:09:21 +02:00
jessicagreben 896c828bb5 close writer after flush
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-29 06:45:12 -07:00
jessicagreben d89a1d999f add log with start/end times, close blocks before end of func
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-28 12:13:58 -07:00
Ben Kochie f0bccba1c3
Update Go modules for 2.26 (#8636)
* Update Go modules for 2.26

Bump all Go modules to the latest upstream.

Signed-off-by: Ben Kochie <superq@gmail.com>

* Fix promtool for new client_golang

LabelValues now requires a list of string matchers.

Signed-off-by: Ben Kochie <superq@gmail.com>
2021-03-24 09:41:12 +00:00
Julien Pivotto c0c36b1155
Improve promql-negative-offset docs (#8631)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-22 10:16:43 +01:00
jessicagreben 8de4da3716 add changes per comments, fix tests
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-20 12:38:30 -07:00
Callum Styan 289ba11b79
Add circular in-memory exemplars storage (#6635)
* Add circular in-memory exemplars storage

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>

* Fix some comments, clean up exemplar metrics struct and exemplar tests.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix exemplar query api null vs empty array issue.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-03-16 15:17:45 +05:30
jessicagreben e3a8132bb3 fix block alignment, add sample alignment
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-15 12:44:58 -07:00
jessicagreben 7c26642460 add block alignment and write in 2 hr blocks
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-14 10:10:55 -07:00
Julien Pivotto 63ea88af82
Merge pull request #8575 from pfreixes/add-scrapes-parameter
Add num scrapes as tsdb write benchmark command flag
2021-03-11 13:09:50 +01:00
Pau Freixes b1ac4a45e6 Add num scrapes as tsdb write benchmark command flag
By default same value that was hardcoded is used, but with the
new flag added the number of scrapes can be increased to any value.

Signed-off-by: Pau Freixes <pfreixes@gmail.com>
2021-03-10 11:17:07 +01:00
Julien Pivotto ad5ed416ba
Merge pull request #8487 from pschou/dev_neg_offset
allow negative offset
2021-03-08 22:18:45 +01:00
Julien Pivotto 5742a18590 Fix subqueries with default resolution in promql unit tests
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-07 09:20:04 +01:00
jessicagreben 9fc53b7edf fix appender.Add -> appender.Append
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-01 05:49:49 -08:00
Arthur Silva Sens 537c0aff49
Prometheus and Promtool binaries now print help and usage to stdout (#8542)
Signed-off-by: ArthurSens <arthursens2005@gmail.com>
2021-02-25 19:52:34 +01:00
jessicagreben 78e84aed89 resolve merge conflict
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-02-24 09:47:29 -08:00
jessicagreben f2db9dc722 add multi rule integration tests
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-02-24 09:42:31 -08:00
pschou f80b52be69
Merge branch 'main' into dev_neg_offset 2021-02-23 20:52:57 -05:00
schou 22cd48868a adding feature flag, promql-negative-offset
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
Julien Pivotto 8c8de46003
Merge pull request #8036 from dgl/promtool-alert-err
promtool: Don't end alert tests early, in some failure situations
2021-02-20 22:35:00 +01:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489)
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
Julien Pivotto 1fac1c783b
Merge pull request #8504 from rbauduin/require_alertname
promtool: alert_rule_test items require alertname
2021-02-17 22:07:52 +01:00
Julien Pivotto 2d172d0896
Merge pull request #8508 from prometheus/release-2.25
Merge back release 2.25
2021-02-17 16:26:34 +01:00
Raphael Bauduin a7d64cad21 promtool: alert_rule_test items require alertname
Accepting alert_rule_test without alertname is confusing as it will
always pass with empty exp_alerts, and never with non-empty exp_alerts.

Signed-off-by: Raphael Bauduin <raphael.bauduin@tessares.net>
2021-02-17 16:23:12 +01:00
Ganesh Vernekar c4536fa28c
Increase block writer size for backfilling
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-17 15:45:41 +05:30
Julien Pivotto a419b75abd
Merge pull request #8485 from hryniuk/promtool-query-errors-details
Print details of API errors received by promtool
2021-02-16 22:47:08 +01:00
Łukasz Hryniuk ab41de68b4 Print details of API errors
Signed-off-by: Łukasz Hryniuk <code@hryniuk.pl>
2021-02-15 23:42:06 +01:00
David Leadbeater 3e30f72af1 promtool: Add more negative alert tests
Signed-off-by: David Leadbeater <dgl@dgl.cx>
2021-02-15 17:00:49 +00:00
Julien Pivotto e29b47b39e
Merge pull request #8440 from mishamo/master
Add optional name property to testgroup for better test failure output
2021-02-09 21:23:24 +01:00