mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-12-27 14:39:40 -08:00
Code Issues Actions 5 Packages Projects Releases Wiki Activity
8335 commits 243 branches 444 tags 649 MiB
Commit graph

274 commits

Author SHA1 Message Date
Arthur Silva Sens 64a106c5dd
Logging added for when compaction takes more than the block time range (#8151) 
* Logging added for when compaction takes more than the block time range

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Log only if no errors were already logged

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Log duration as human readable string

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Move logging from compactHead() to Compact()

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Compute duration of all head compactions plus wal truncation

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Remove named return added os first commits

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Address nits

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change miliseconds to seconds to make fuzzit tests happy

Signed-off-by: ArthurSens <arthursens2005@gmail.com>
 2020-12-07 21:29:43 +00:00
johncming a6e18916ab
tsdb: Remove duplicate variables. (#8239) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-11-30 08:55:33 +00:00
Atibhi Agrawal b317b6ab9c
Backfill from OpenMetrics format (#8084) 
* get parser working

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* import file created

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Find min and max ts

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make two passes over file and write to tsdb

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* print error messages

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Max and Min initializer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Start with unit tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reset file read

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* align blocks to two hour range

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add cleanup test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove .ds_store

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add license to import_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Circle CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Refactor code
Move backfill from tsdb to promtool directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix gitignore

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove panic
Rename ContenType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* adjust mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return statement

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go modules

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added unit test for backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix file handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close DB

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Error Handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* inline err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix command line flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add spaces before func
fix pointers

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add defer'd calls

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* move openmetrics.go content to backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changed args to flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add tests for wrong OM files

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added additional tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add comment to warn of func reuse

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Make input required in main.go

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer blockwriter close

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove contentType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove defer from backfilltest

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix defer remove in backfill_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changes to fix CI errors

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go.mod

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change package name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* assert->require

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix format

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix createblock

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* check err for anon func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* update comment

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix for the Flush Bug

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix formatting, comments, names

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* cleanup

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor test to take care of multiple samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* I dont know what I fixed

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests, add test description, print blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* commit after 5000 samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reviews part 1

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Series Count

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove extra func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make timestamp into sec

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Reviews 2

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add Todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fixes

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fixes reviews

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* =0

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove backfill.om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add global err var, remove stuff

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change var name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* sampleLimit pass as parameter

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add test when number of samples greater than batch size

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Change name of batchsize

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* revert export

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add comment, remove newline,consistent err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Modify comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* db.Querier

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add sanity check , get maxt and mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* ci error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* comment change

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* NoError

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>
 2020-11-26 10:37:06 +05:30
Ganesh Vernekar dff967286e
Set the min time of Head properly after truncation (#8212) 
* Set the min time of Head properly after truncation

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix lint

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Enhance compaction plan logic for completely deleted small block

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-11-25 18:33:30 +05:30
Chris Marchbanks c8f810083d
Cache label name and last value when adding series (#8192) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2020-11-19 11:18:12 -07:00
Marco Pracucci db19e05d93
Add option to customise head chunks write buffer size (#8201) 
* Add option to customise head chunks write buffer size

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed tests

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-11-19 18:30:47 +05:30
Bartlomiej Plotka 4513537034
Exposed DeletionIterator and CompactMetas functions. (#8161) 
* Exposed DeletionIterator and CompactMetas functions.

Required for CLI for deletions in Thanos: https://github.com/thanos-io/thanos/pull/3421

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed Thanos usage mentions.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-11-09 16:51:25 +00:00
Julien Pivotto 8bc369bf9b
Calculate head chunk size based on actual disk usage (#8139) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-11-03 15:34:59 +05:30
Marco Pracucci 63be30dcee
Fixed WAL corruption on partial writes within a page (#8125) 
* Fixed WAL corruption on partial writes

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed variable

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-10-29 16:07:03 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122) 
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-29 09:43:23 +00:00
Bartlomiej Plotka 3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. (#8066) 
* MultiError: Refactored MultiError for more concise and safe usage.

* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename package to merrors. (: In different PR.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed review comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix after rebase.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-28 15:24:58 +00:00
johncming 28ca0965f0
tsdb/chunkenc: fix typo of return error. (#7670) 
* tsdb/chunkenc: fix typo of return error.

Signed-off-by: johncming <johncming@yahoo.com>

* tsdb: fix typo of function in markdonw.

Signed-off-by: johncming <johncming@yahoo.com>
 2020-10-28 12:03:11 +00:00
Ganesh Vernekar 3245b3267b
Don't use returned DB to close resources on TSDB startup error (#8113) 
* Don't use returned DB to close resources on TSDB startup error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add unit test and fix another panic

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comment

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-28 15:39:03 +05:30
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110) 
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-22 11:00:08 +02:00
Arthur Silva Sens c5a832b394
Close resources after failing to startup TSDB (#8031) 
* Close resources after failing to startup TSDB

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Return close error instead of logging

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change named return's name

Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-10-21 20:38:28 +05:30
Brian Brazil fdf1c29224
Fix metric description of prometheus_tsdb_symbol_table_size_bytes. (#8080) 
This is how much memory we use to load in the on-disk
symbol tables, not the size of the tables themselves.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-21 14:35:40 +01:00
Ganesh Vernekar ce4b3ac282
Simplify TestHeadReadWriter_Truncate (#7437) 
* Simplify TestHeadReadWriter_Truncate

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:41:39 +05:30
Ganesh Vernekar 2624d827fa
Read repair empty last file in chunks_head (#8061) 
* Read repair empty file in chunks_head

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor and introduce repairLastChunkFile

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Attempt windows test fix

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:27:13 +05:30
Bartlomiej Plotka 2fe1e9fa93
Create a checkpoint only at the end of Compact call (#8067) 
* Create a checkpoint only at the end of Compact call

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix Bartek's offline reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Introduce TruncateInMemory and TruncateWAL

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Small enhancements and test fixing attempts

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add TestOneCheckpointPerCompactCall

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Don't truncate WAL on block compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplified the algo.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Better protection around calling truncateWAL, truncate WAL on Head compaction error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 20:57:08 +05:30
Ganesh Vernekar d9cd913219
Add links to TSDB blog post series in tsdb/README.md (#8078) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 17:17:10 +05:30
Julien Pivotto 59733b1a26
TSDB: use blocks instead of db.blocks in condition (#8068) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-19 16:51:54 +05:30
Ganesh Vernekar 5de7d15879
Consistent naming and better names for fields in TSDB format docs (#8077) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 14:27:19 +05:30
Bartlomiej Plotka 9981b3f3ee
labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) 
* labels: Reduce allocated memory by Hash method in edge cases; Added tests.

Old:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 5366161	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 1700371	       767 ns/op	    2048 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	     356	   3743115 ns/op	10523442 B/op	       1 allocs/op
PASS

New:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 4758883	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 3324492	       357 ns/op	      80 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	    1087	   1083949 ns/op	    9734 B/op	       1 allocs/op
PASS

Process finished with exit code 0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Kemal's comment.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Moved to v2 xxhash for improvements.

New:

/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ -test.benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	53447894	       221 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	42341754	       326 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	   10000	   1248546 ns/op	       0 B/op	       0 allocs/op
PASS

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed old xxhash package.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Pined minor version.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-15 11:31:28 +01:00
Ganesh Vernekar 4ed3ec7b46
More info in mmap error message (#8058) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-15 12:58:13 +05:30
frbimo 260cd84d8f
log on empty head compaction result (#8037) 
Signed-off-by: frbimo <fr.bimo@gmail.com>
 2020-10-14 18:59:59 +05:30
Gayathri Venkatesh 73e2ce1bd6
Do not ignore reload errors in compactHead (#8051) 
* Modified unknownRefs to unknownRefs.Load()

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Modified db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Revert "Modified unknownRefs to unknownRefs.Load()"

This reverts commit 79caf595fa9b9193878dc0dd9dec17d58851ae42.

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Made changes to reload error in db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>
 2020-10-14 15:05:24 +05:30
Goutham Veeramachaneni 1cc02930b6
Implement series API query hint to not hit chunks (#8050) 
Fixes https://github.com/prometheus/prometheus/issues/5547

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2020-10-14 14:36:17 +05:30
Ling Jin 9145200842
tsdb: fix unkown ref in log (#8048) 
Signed-off-by: JinLingChristopher <jinl1037@hotmail.com>
 2020-10-13 20:03:16 +05:30
Arthur Silva Sens 4f45e201cc
Promtool tsdb list now prints block sizes (#7993) 
* promtool tsdb list now prints blocks' size

Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-10-12 23:15:40 +02:00
Jess G 90680b092c
Add a tsdb writer interface (#7950) 
* add a tsdb writer interface

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in tsdbutil block creator

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* pass thru context

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in blockutil and also in tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* fix link err

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add err handling for defer

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add blockwriter test

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* mv initHead into NewBlockWriter, update tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use blocksize arg, make test more concise

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-10-12 17:04:20 +01:00
garanews c38816828f
fix few typo (#8023) 
Signed-off-by: garanews <puntogtg@tiscali.it>
 2020-10-07 16:51:31 +01:00
Brian Brazil 073e93c768
Gracefully handle unknown WAL record types. (#8004) 
As we're looking to expand what's in the WAL,
having old Prometheus servers ignore the new record types
rather than treating them as corruption allows for better
upgrade/downgrade paths.

Adjust some tests accordingly, so they're still testing what they're
meant to test.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-05 10:09:59 +01:00
Ganesh Vernekar 50ae4e298b
Fix magic number in docs (#7998) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-01 15:05:01 +05:30
Ganesh Vernekar f0d87b5d86
Encode and Decode method for tombstones (#7967) 
* Encode and Decode method for tombstones

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-09-30 10:25:09 +05:30
Max Neverov 7e1c27b853
Add tsdb startup duration metric (#7737) 
* Add tsdb wal replay duration metric

Signed-off-by: Max Neverov <neverov.max@gmail.com>
 2020-09-21 18:25:05 +02:00
Jorge Vallecillo 7aa5fb01bf
tsdb/chunks/head_chunks_test.go: Fix typo (#7953) 
tsdb/db_test.go: Fix typo

Signed-off-by: Jorge Vallecillo <jorgevallecilloc@gmail.com>
 2020-09-20 18:42:01 +01:00
Tobias Klauser b5b3bc34da
tsdb/fileutil: Use FcntlFstore from golang.org/x/sys/unix (#7949) 
Direct syscalls using syscall.Syscall(SYS_*, ...) should no longer be
used on darwin, see [1]. Instead, use the FcntlFstore libSystem wrapper
provided by the golang.org/x/sys/unix package to implement
preallocFixed.

[1] https://golang.org/doc/go1.12#darwin

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
 2020-09-19 10:57:13 +01:00
Xiaochao Dong a282d25099
tsdb: remove duplicate values set to reduce memory usage(map overhead) (#7915) 
Signed-off-by: Xiaochao Dong (@damnever) <dxc.wolf@gmail.com>
 2020-09-10 20:35:47 +05:30
zhulongcheng 5ea6bdca1f
tsdb: sync temporary checkpoint directory before rename (#7004) 
Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>
 2020-09-07 20:04:49 +05:30
johncming 75ae384192
tsdb: remove redundant fields. (#7869) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-09-02 17:03:21 +01:00
Ganesh Vernekar 2255b6f62f
Refactor WAL.Segments method to be part of the wal package (#6477) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-09-01 14:46:57 +05:30
johncming 252265fa81
tsdb/chunks: remove un-used code. (#6944) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-31 16:55:47 +02:00
Ganesh Vernekar c806262206
Fix 'chunks.HeadReadWriter: maxt of the files are not set' error (#7856) 
* Fix chunks.HeadReadWriter: maxt of the files are not set

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-08-26 19:59:18 +02:00
Yukun Sun cfd4e05c9e
fix: return a corruption error when iterator function find a chunk that is out of sequence (#7855) 
Signed-off-by: sunyukun <sunyukun@didiglobal.com>

Co-authored-by: sunyukun <sunyukun@didiglobal.com>
 2020-08-26 20:36:27 +05:30
Max Neverov bb5c6b38e2
Fix Possible Race Condition in TSDB (#7815) 
* Replace tsdb chunk mapper size with atomic; protect mmappedChunkFiles with read path mutex on DeleteCorrupted

Signed-off-by: Max Neverov <neverov.max@gmail.com>

* PR fixes

Signed-off-by: Max Neverov <neverov.max@gmail.com>
 2020-08-26 14:22:48 +05:30
johncming 3e7b463908
add testcase for rangeForTimestamp. (#6454) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-25 16:16:43 +01:00
Guangwen Feng 9439b2150d
Fix golint warning caused by misspell (#7842) 
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
 2020-08-25 09:17:41 +01:00
johncming b521612042
tsdb: simplify code. (#7792) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-14 15:15:08 +05:30
gastonqiu 5a7d398d19
Chore: Log segment number when segment read failed (#7687) 
* Chore: Log segment number when segment read failed

To manually fix the WAL files, it is good to know where the corrupt
happened so we should log the segment number when the read failed.

Related Issue #7506

Signed-off-by: gaston.qiu <gaston.qiu@umbocv.com>
 2020-08-13 11:01:21 -07:00