Tom Wilkie
7369561305
Combine Appender.Add and AddFast into a single Append method. ( #8489 )
...
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.
This makes the API easier to consume and implement. In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
Julien Pivotto
9334269f2b
backfill: move checkErr before we close the mmaped file
...
When printing the error, we still need access to the mmapped byte array
of the file. Therefore, we make sure that we run it before closing the
file.
I could have done something more complex like a defer, or not closing
the file, knowing that we would exit the program anyway. However, I
think that in case we extend this in the future, or this is copy/paster
elsewhere, we should continue closing the file. As it is small enough, I
went for the solution to call the function 3 times instead of playing
with a defer.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-02-01 21:18:42 +01:00
beorn7
6bfa33308e
promtool: Print block meta-data slightly more nicely
...
I initially thought I could somehow rescue the current column layout
by recycling the tabwriter, but flushing completely blanks
it. However, by setting a minimum width of 13, we get a slightly
broader DURATION column but otherwise nice formatting, unless numbers
get really big, but that's OK, I guess.
Before:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0KGNP5WWK9T5QMQGBG9F1 2020-11-19 07:39:17 +0000 UTC 2020-11-19 07:44:17 +0000 UTC 5m0.001s 8 2 2 624B
01ETN0KGQSFF0AB2QDZVQG3CWC 2020-11-19 10:25:57 +0000 UTC 2020-11-19 10:30:57 +0000 UTC 5m0.001s 8 2 2 622B
01ETN0KGSW8KYP3YPG4X20P60Z 2020-11-19 13:12:37 +0000 UTC 2020-11-19 13:17:37 +0000 UTC 5m0.001s 8 2 2 625B
```
After:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0R72SXN9A1FG732P7KFFN 2020-11-19 07:39:17 +0000 UTC 2020-11-19 07:44:17 +0000 UTC 5m0.001s 8 2 2 624B
01ETN0R74Y9AG1A1MKN4MZK7WM 2020-11-19 10:25:57 +0000 UTC 2020-11-19 10:30:57 +0000 UTC 5m0.001s 8 2 2 622B
01ETN0R76KXZ5VQECMDNES49J6 2020-11-19 13:12:37 +0000 UTC 2020-11-19 13:17:37 +0000 UTC 5m0.001s 8 2 2 625B
```
After without the `-r` flag:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0RFFJ42274NWR1GH0RTV6 1605771557000 1605771857001 5m0.001s 8 2 2 624
01ETN0RFJ1MZCHHS2SBZS8XC27 1605781557000 1605781857001 5m0.001s 8 2 2 622
01ETN0RFM98N3V4KD2DZXFGHGN 1605791557000 1605791857001 5m0.001s 8 2 2 625
```
Signed-off-by: beorn7 <beorn@grafana.com>
2020-12-28 16:55:12 +01:00
beorn7
651b57b9ab
Merge branch 'backfillhr' of git://github.com/roidelapluie/prometheus into review
2020-12-28 16:18:00 +01:00
yeya24
cedd2dbec9
create output directory before backfilling
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-12-24 23:36:36 -05:00
Julien Pivotto
53480c168d
Backfill: print created blocks only, add human-readable option
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-23 20:42:30 +01:00
Julien Pivotto
7957731339
Inline defer
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-09 09:23:39 +01:00
Julien Pivotto
82b5f1d8b1
Backfill: Use mmap to reuse parser code
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-08 23:48:31 +01:00
Atibhi Agrawal
b317b6ab9c
Backfill from OpenMetrics format ( #8084 )
...
* get parser working
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* import file created
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Find min and max ts
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make two passes over file and write to tsdb
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* print error messages
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Max and Min initializer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Start with unit tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reset file read
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* align blocks to two hour range
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add cleanup test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove .ds_store
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add license to import_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Circle CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Refactor code
Move backfill from tsdb to promtool directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix gitignore
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove panic
Rename ContenType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* adjust mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return statement
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go modules
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added unit test for backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix file handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close DB
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Error Handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* inline err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix command line flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add spaces before func
fix pointers
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add defer'd calls
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* move openmetrics.go content to backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changed args to flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add tests for wrong OM files
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added additional tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add comment to warn of func reuse
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Make input required in main.go
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer blockwriter close
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove contentType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove defer from backfilltest
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix defer remove in backfill_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changes to fix CI errors
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go.mod
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change package name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* assert->require
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix format
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix createblock
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* check err for anon func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* update comment
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix for the Flush Bug
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix formatting, comments, names
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* cleanup
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor test to take care of multiple samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* I dont know what I fixed
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests, add test description, print blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* commit after 5000 samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reviews part 1
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Series Count
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove extra func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make timestamp into sec
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Reviews 2
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add Todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fixes
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fixes reviews
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* =0
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove backfill.om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add global err var, remove stuff
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change var name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* sampleLimit pass as parameter
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add test when number of samples greater than batch size
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Change name of batchsize
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* revert export
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add comment, remove newline,consistent err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Modify comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* db.Querier
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add sanity check , get maxt and mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* ci error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* comment change
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* NoError
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-11-26 10:37:06 +05:30
Bartlomiej Plotka
3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. ( #8066 )
...
* MultiError: Refactored MultiError for more concise and safe usage.
* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.
NOTE: Potentially I would rename package to merrors. (: In different PR.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed review comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Fix after rebase.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-28 15:24:58 +00:00
Julien Pivotto
4e5b1722b3
Move away from testutil, refactor imports ( #8087 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Arthur Silva Sens
4f45e201cc
Promtool tsdb list now prints block sizes ( #7993 )
...
* promtool tsdb list now prints blocks' size
Signed-off-by: arthursens <arthursens2005@gmail.com>
2020-10-12 23:15:40 +02:00
Annanay
9bba8a6eae
Merge branch 'master' into appender-context
...
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-30 16:43:18 +05:30
Annanay
7f98a744e5
Add context to Appender interface
...
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-24 19:40:51 +05:30
chinhnc
e05c19da5d
Display block duration in promtool list blocks command ( #7653 )
...
* Update tsdb.go
Added DURATION column to `tsdb list` command
Signed-off-by: soup <chicknsoupuds@gmail.com>
* Use time.Duration instead of hardcoded hour
Signed-off-by: soup <chicknsoupuds@gmail.com>
2020-07-24 19:01:20 +05:30
Ben Ye
50c261502e
add tsdb cmds into promtool ( #6088 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
update tsdb cli in makefile and promu
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove building tsdb bin
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove useless func
Signed-off-by: yeya24 <yb532204897@gmail.com>
refactor analyzeBlock
Signed-off-by: yeya24 <yb532204897@gmail.com>
Fix Makefile
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2020-07-23 19:35:50 +01:00
