mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-10 07:34:04 -08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
8646 commits 245 branches 436 tags 650 MiB
Commit graph

299 commits

Author SHA1 Message Date
Bryan Boreham d614ae9ecf
[RFC] Add method to get reference number for TSDB Appender (#8600) 
* Add method to get reference number for TSDB Appender

In situations where we need to copy labels before calling Add(),
GetRef() allows to check first, then call AddFast() in the case that the series
is already known.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Add explicit interface for GetRef() method

Suggested in code review by @bwplotka

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Rename OptionalGetRef to GetRef

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Simplify return value of GetRef()

0 can be relied on to mean 'no reference'

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
 2021-03-19 19:28:55 +00:00
Marco Pracucci 6248e685b2
TSDB: optimize series creation on PreCreation() failure (#8620) 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-03-18 20:53:50 +05:30
Callum Styan 289ba11b79
Add circular in-memory exemplars storage (#6635) 
* Add circular in-memory exemplars storage

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>

* Fix some comments, clean up exemplar metrics struct and exemplar tests.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix exemplar query api null vs empty array issue.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
 2021-03-16 15:17:45 +05:30
Bartlomiej Plotka 8bf7bc68f1
Fixed TestChunkDiskMapper_WriteChunk_Chunk_IterateChunks for go1.16 (#8538) 
Fixes https://github.com/prometheus/prometheus/issues/8403

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2021-02-25 14:38:12 +05:30
Julien Pivotto 432d5ebc6c Rename default branch to main 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-02-22 20:28:02 +01:00
Julien Pivotto 889dd0bbd3 Fix DB tests in the default branch 
The main branch tests are not passing due to the fact that #8489 was not
rebased on top of #8007.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-02-18 23:56:27 +01:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489) 
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
 2021-02-18 17:37:00 +05:30
Goutham Veeramachaneni 404d85f7a8
Add ability to set minValidTime in TSDB (#8415) 
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2021-02-18 10:52:35 +05:30
Chris Marchbanks bedcd88343
Compress records before checking segment size (#8501) 
Right now a new segment might be created unnecessarily if the
uncompressed record would not fit, but after compression (typically
reducing record size in half) it would.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2021-02-17 18:45:49 +05:30
Arthur Silva Sens 6a3d55db0a
Rolling tombstones clean up (#8007) 
* CleanupTombstones refactored, now reloading blocks after every compaction.

The goal is to remove deletable blocks after every compaction and, thus, decrease disk space used when cleaning tombstones.

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Protect DB against parallel reloads

Signed-off-by: ArthurSens <arthursens2005@gmail.com>

* Fix typos

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2021-02-17 11:02:43 +05:30
Mauro Stettler 7715fe3219
Add matchers to LabelValues() call (#8400) 
* Accept matchers in querier LabelValues()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* create matcher to only select metrics which have searched label

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test case for merge querier with matchers

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test LabelValues with matchers on head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add test for LabelValues on block

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* formatting fix

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Add comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add missing lock release

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unused parameter

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Benchmarks for LabelValues() methods on block/head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Better comment

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update comment

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* minor refactor make code cleaner

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* better comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix expected errors in test

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Deleting parameter which can only be empty

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary lock

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lookup label value if label name was looked up

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Return error when there is one

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Call .Get() on decoder before checking errors

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lock head.symMtx when necessary

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary delete()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* re-use code instead of duplicating it

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Consistently return error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* move helper func from util.go to querier.go

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Fix test expectation

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* ensure result de-duplication and sorting works

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* return named error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
 2021-02-09 23:08:35 +05:30
Dustin Hooten b9f0baf6ff
Combine NewHead() args into a HeadOptions struct (#8452) 
* Combine NewHead() args into a HeadOptions struct

Signed-off-by: Dustin Hooten <dustinhooten@gmail.com>

* remove overrides params

Signed-off-by: Dustin Hooten <dustinhooten@gmail.com>

* address pr feedback

Signed-off-by: Dustin Hooten <dustinhooten@gmail.com>
 2021-02-09 19:42:48 +05:30
Marcelo E. Magallon 75d86c6747 Update golangci-lint to 1.36.0 
In the previous version, 1.18.0, the "megacheck" linter paid attention
to the '//lint:ignore' comment, but that is no longer there.

Newer version pay attention to '//nolint:<linter>,<linter>,...'
comments, optionally followed by a "second" comment introduced by '//'.

Update the directives to use this style.

This is related to prometheus/blackbox_exporter#738 and
prometheus/blackbox_exporter#745.

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
 2021-02-04 08:53:33 -06:00
Goutham Veeramachaneni e572e77d90
Increase the buckets for compaction duration (#8342) 
We're seeing compactions that are taking hours in Cortex which this is
missing. I know while it is not common in Prometheus, I am pretty sure
there are setups where compaction takes longer than 512s. On our own
Prometheus the average compaction duration is 566s.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2021-01-25 18:12:18 +00:00
Ganesh Vernekar a7e446cf2d
Link blog post on TSDB queries in tsdb/README.md (#8391) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2021-01-21 17:14:40 +05:30
Marco Pracucci d8c17025df
Fix TSDB head struct dump on querier error (#8379) 
* Fix TSDB head struct dump on querier error

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Added mint/maxt to RangeHead.String()

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2021-01-21 16:07:29 +05:30
Li Zhijian 9602933db2
Cleanup tempfiles in tests (#8179) 
* test: cleanup tempdir for TestBlockWriter

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>

* test: cleanup tempdir for TestLogPartialWrite

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
 2021-01-15 13:49:57 +05:30
Nguyen Le Vu Long fbe960f2c1
fix: remove pre-2.21 tmp blocks on start (#8353) 
* fix: remove pre-2.21 tmp blocks on start

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>

* fix: commenting

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
 2021-01-09 10:02:26 +01:00
Arthur Silva Sens 7e932637e3
Reload tsdb blocks every minute (#8340) 
* Reload tsdb blocks every minute

Signed-off-by: ArthurSens <arthursens2005@gmail.com>

* Proteced tsdb with mutex locks

Signed-off-by: ArthurSens <arthursens2005@gmail.com>
 2021-01-07 13:00:08 +05:30
Nguyen Le Vu Long cd1dafc2fd
tsdb: Expose total number of label pairs in head in TSDB stats page (#8343) 
* tsdb: Expose total number of label pairs in head

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>

* fix: add comment for NumLabelPairs

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>

* fix: remove comment

Signed-off-by: Nguyen Le Vu Long <vulongvn98@gmail.com>
 2021-01-07 12:11:32 +05:30
Guangwen Feng e2cd6c5f57 Fix golint issue caused by typo 
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
 2021-01-06 15:54:35 +08:00
Björn Rabenstein 7889bdf16e
Merge pull request #8313 from pracucci/lower-compactor-overlapping-blocks-log 
Lowered 'Found overlapping blocks during compaction' log level
 2020-12-28 14:59:59 +01:00
arthursens 8493704b9b Change seconds()*1000 to milliseconds() 
Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-12-25 10:45:23 -03:00
Marco Pracucci 2b4b138244
Lowered 'Found overlapping blocks during compaction' log level 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-12-23 17:23:04 +01:00
Ganesh Vernekar faa1554aa1
Don't call runtime.GC() after compaction (#8276) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-12-22 14:44:17 +00:00
Arthur Silva Sens 64a106c5dd
Logging added for when compaction takes more than the block time range (#8151) 
* Logging added for when compaction takes more than the block time range

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Log only if no errors were already logged

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Log duration as human readable string

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Move logging from compactHead() to Compact()

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Compute duration of all head compactions plus wal truncation

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Remove named return added os first commits

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Address nits

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change miliseconds to seconds to make fuzzit tests happy

Signed-off-by: ArthurSens <arthursens2005@gmail.com>
 2020-12-07 21:29:43 +00:00
johncming a6e18916ab
tsdb: Remove duplicate variables. (#8239) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-11-30 08:55:33 +00:00
Atibhi Agrawal b317b6ab9c
Backfill from OpenMetrics format (#8084) 
* get parser working

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* import file created

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Find min and max ts

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make two passes over file and write to tsdb

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* print error messages

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Max and Min initializer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Start with unit tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reset file read

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* align blocks to two hour range

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add cleanup test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove .ds_store

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add license to import_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Circle CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Refactor code
Move backfill from tsdb to promtool directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix gitignore

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove panic
Rename ContenType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* adjust mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return statement

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go modules

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added unit test for backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix file handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close DB

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Error Handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* inline err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix command line flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add spaces before func
fix pointers

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add defer'd calls

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* move openmetrics.go content to backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changed args to flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add tests for wrong OM files

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added additional tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add comment to warn of func reuse

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Make input required in main.go

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer blockwriter close

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove contentType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove defer from backfilltest

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix defer remove in backfill_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changes to fix CI errors

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go.mod

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change package name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* assert->require

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix format

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix createblock

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* check err for anon func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* update comment

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix for the Flush Bug

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix formatting, comments, names

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* cleanup

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor test to take care of multiple samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* I dont know what I fixed

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests, add test description, print blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* commit after 5000 samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reviews part 1

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Series Count

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove extra func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make timestamp into sec

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Reviews 2

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add Todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fixes

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fixes reviews

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* =0

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove backfill.om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add global err var, remove stuff

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change var name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* sampleLimit pass as parameter

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add test when number of samples greater than batch size

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Change name of batchsize

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* revert export

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add comment, remove newline,consistent err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Modify comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* db.Querier

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add sanity check , get maxt and mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* ci error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* comment change

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* NoError

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>
 2020-11-26 10:37:06 +05:30
Ganesh Vernekar dff967286e
Set the min time of Head properly after truncation (#8212) 
* Set the min time of Head properly after truncation

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix lint

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Enhance compaction plan logic for completely deleted small block

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-11-25 18:33:30 +05:30
Chris Marchbanks c8f810083d
Cache label name and last value when adding series (#8192) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2020-11-19 11:18:12 -07:00
Marco Pracucci db19e05d93
Add option to customise head chunks write buffer size (#8201) 
* Add option to customise head chunks write buffer size

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed tests

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-11-19 18:30:47 +05:30
Bartlomiej Plotka 4513537034
Exposed DeletionIterator and CompactMetas functions. (#8161) 
* Exposed DeletionIterator and CompactMetas functions.

Required for CLI for deletions in Thanos: https://github.com/thanos-io/thanos/pull/3421

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed Thanos usage mentions.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-11-09 16:51:25 +00:00
Julien Pivotto 8bc369bf9b
Calculate head chunk size based on actual disk usage (#8139) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-11-03 15:34:59 +05:30
Marco Pracucci 63be30dcee
Fixed WAL corruption on partial writes within a page (#8125) 
* Fixed WAL corruption on partial writes

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed variable

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-10-29 16:07:03 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122) 
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-29 09:43:23 +00:00
Bartlomiej Plotka 3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. (#8066) 
* MultiError: Refactored MultiError for more concise and safe usage.

* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename package to merrors. (: In different PR.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed review comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix after rebase.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-28 15:24:58 +00:00
johncming 28ca0965f0
tsdb/chunkenc: fix typo of return error. (#7670) 
* tsdb/chunkenc: fix typo of return error.

Signed-off-by: johncming <johncming@yahoo.com>

* tsdb: fix typo of function in markdonw.

Signed-off-by: johncming <johncming@yahoo.com>
 2020-10-28 12:03:11 +00:00
Ganesh Vernekar 3245b3267b
Don't use returned DB to close resources on TSDB startup error (#8113) 
* Don't use returned DB to close resources on TSDB startup error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add unit test and fix another panic

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comment

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-28 15:39:03 +05:30
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110) 
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-22 11:00:08 +02:00
Arthur Silva Sens c5a832b394
Close resources after failing to startup TSDB (#8031) 
* Close resources after failing to startup TSDB

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Return close error instead of logging

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change named return's name

Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-10-21 20:38:28 +05:30
Brian Brazil fdf1c29224
Fix metric description of prometheus_tsdb_symbol_table_size_bytes. (#8080) 
This is how much memory we use to load in the on-disk
symbol tables, not the size of the tables themselves.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-21 14:35:40 +01:00
Ganesh Vernekar ce4b3ac282
Simplify TestHeadReadWriter_Truncate (#7437) 
* Simplify TestHeadReadWriter_Truncate

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:41:39 +05:30
Ganesh Vernekar 2624d827fa
Read repair empty last file in chunks_head (#8061) 
* Read repair empty file in chunks_head

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor and introduce repairLastChunkFile

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Attempt windows test fix

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:27:13 +05:30
Bartlomiej Plotka 2fe1e9fa93
Create a checkpoint only at the end of Compact call (#8067) 
* Create a checkpoint only at the end of Compact call

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix Bartek's offline reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Introduce TruncateInMemory and TruncateWAL

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Small enhancements and test fixing attempts

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add TestOneCheckpointPerCompactCall

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Don't truncate WAL on block compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplified the algo.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Better protection around calling truncateWAL, truncate WAL on Head compaction error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 20:57:08 +05:30
Ganesh Vernekar d9cd913219
Add links to TSDB blog post series in tsdb/README.md (#8078) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 17:17:10 +05:30
Julien Pivotto 59733b1a26
TSDB: use blocks instead of db.blocks in condition (#8068) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-19 16:51:54 +05:30
Ganesh Vernekar 5de7d15879
Consistent naming and better names for fields in TSDB format docs (#8077) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 14:27:19 +05:30
Bartlomiej Plotka 9981b3f3ee
labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) 
* labels: Reduce allocated memory by Hash method in edge cases; Added tests.

Old:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 5366161	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 1700371	       767 ns/op	    2048 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	     356	   3743115 ns/op	10523442 B/op	       1 allocs/op
PASS

New:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 4758883	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 3324492	       357 ns/op	      80 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	    1087	   1083949 ns/op	    9734 B/op	       1 allocs/op
PASS

Process finished with exit code 0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Kemal's comment.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Moved to v2 xxhash for improvements.

New:

/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ -test.benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	53447894	       221 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	42341754	       326 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	   10000	   1248546 ns/op	       0 B/op	       0 allocs/op
PASS

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed old xxhash package.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Pined minor version.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-15 11:31:28 +01:00
Ganesh Vernekar 4ed3ec7b46
More info in mmap error message (#8058) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-15 12:58:13 +05:30