Julien Pivotto
bc9f9ee3aa
Backfilling: fast-path for non-consecutive blocks ( #8324 )
...
* Backfilling: optimize for non-consecutive blocks
When you have missing data for > 2 hours, you spend a lot of time
re-reading the complete file. It is not optimal.
This introduces a fastpath for this scenario.
Next, we do parse the metric even when we know we will not use it, based
on its timestamp. This only computes the metric when we know its
timestamp is right.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-30 02:06:41 +01:00
Julien Pivotto
003d6451fc
Promtool: add web config validation
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-29 16:55:29 +01:00
beorn7
6bfa33308e
promtool: Print block meta-data slightly more nicely
...
I initially thought I could somehow rescue the current column layout
by recycling the tabwriter, but flushing completely blanks
it. However, by setting a minimum width of 13, we get a slightly
broader DURATION column but otherwise nice formatting, unless numbers
get really big, but that's OK, I guess.
Before:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0KGNP5WWK9T5QMQGBG9F1 2020-11-19 07:39:17 +0000 UTC 2020-11-19 07:44:17 +0000 UTC 5m0.001s 8 2 2 624B
01ETN0KGQSFF0AB2QDZVQG3CWC 2020-11-19 10:25:57 +0000 UTC 2020-11-19 10:30:57 +0000 UTC 5m0.001s 8 2 2 622B
01ETN0KGSW8KYP3YPG4X20P60Z 2020-11-19 13:12:37 +0000 UTC 2020-11-19 13:17:37 +0000 UTC 5m0.001s 8 2 2 625B
```
After:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0R72SXN9A1FG732P7KFFN 2020-11-19 07:39:17 +0000 UTC 2020-11-19 07:44:17 +0000 UTC 5m0.001s 8 2 2 624B
01ETN0R74Y9AG1A1MKN4MZK7WM 2020-11-19 10:25:57 +0000 UTC 2020-11-19 10:30:57 +0000 UTC 5m0.001s 8 2 2 622B
01ETN0R76KXZ5VQECMDNES49J6 2020-11-19 13:12:37 +0000 UTC 2020-11-19 13:17:37 +0000 UTC 5m0.001s 8 2 2 625B
```
After without the `-r` flag:
```
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01ETN0RFFJ42274NWR1GH0RTV6 1605771557000 1605771857001 5m0.001s 8 2 2 624
01ETN0RFJ1MZCHHS2SBZS8XC27 1605781557000 1605781857001 5m0.001s 8 2 2 622
01ETN0RFM98N3V4KD2DZXFGHGN 1605791557000 1605791857001 5m0.001s 8 2 2 625
```
Signed-off-by: beorn7 <beorn@grafana.com>
2020-12-28 16:55:12 +01:00
beorn7
651b57b9ab
Merge branch 'backfillhr' of git://github.com/roidelapluie/prometheus into review
2020-12-28 16:18:00 +01:00
yeya24
cedd2dbec9
create output directory before backfilling
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-12-24 23:36:36 -05:00
Julien Pivotto
53480c168d
Backfill: print created blocks only, add human-readable option
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-23 20:42:30 +01:00
AdaephonBen
dca6954b0a
promtool: Add URL scheme when not provided ( #7956 )
...
Signed-off-by: AdaephonBen <ma18btech11011@iith.ac.in>
2020-12-23 19:52:04 +01:00
Julien Pivotto
7957731339
Inline defer
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-09 09:23:39 +01:00
Julien Pivotto
82b5f1d8b1
Backfill: Use mmap to reuse parser code
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-08 23:48:31 +01:00
Atibhi Agrawal
b317b6ab9c
Backfill from OpenMetrics format ( #8084 )
...
* get parser working
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* import file created
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Find min and max ts
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make two passes over file and write to tsdb
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* print error messages
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Max and Min initializer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Start with unit tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reset file read
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* align blocks to two hour range
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add cleanup test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove .ds_store
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add license to import_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Circle CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Refactor code
Move backfill from tsdb to promtool directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix gitignore
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove panic
Rename ContenType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* adjust mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return statement
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go modules
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added unit test for backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix file handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close DB
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Error Handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* inline err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix command line flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add spaces before func
fix pointers
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add defer'd calls
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* move openmetrics.go content to backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changed args to flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add tests for wrong OM files
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added additional tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add comment to warn of func reuse
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Make input required in main.go
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer blockwriter close
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove contentType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove defer from backfilltest
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix defer remove in backfill_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changes to fix CI errors
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go.mod
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change package name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* assert->require
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix format
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix createblock
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* check err for anon func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* update comment
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix for the Flush Bug
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix formatting, comments, names
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* cleanup
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor test to take care of multiple samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* I dont know what I fixed
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests, add test description, print blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* commit after 5000 samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reviews part 1
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Series Count
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove extra func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make timestamp into sec
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Reviews 2
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add Todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fixes
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fixes reviews
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* =0
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove backfill.om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add global err var, remove stuff
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change var name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* sampleLimit pass as parameter
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add test when number of samples greater than batch size
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Change name of batchsize
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* revert export
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add comment, remove newline,consistent err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Modify comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* db.Querier
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add sanity check , get maxt and mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* ci error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* comment change
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* NoError
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-11-26 10:37:06 +05:30
Julien Pivotto
6c56a1faaa
Testify: move to require ( #8122 )
...
* Testify: move to require
Moving testify to require to fail tests early in case of errors.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* More moves
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Bartlomiej Plotka
3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. ( #8066 )
...
* MultiError: Refactored MultiError for more concise and safe usage.
* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.
NOTE: Potentially I would rename package to merrors. (: In different PR.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed review comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Addressed comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Fix after rebase.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-10-28 15:24:58 +00:00
David Leadbeater
e7e60623ff
promtool: Calculate mint and maxt per test ( #8096 )
...
* promtool: Calculate mint and maxt per test
Previously a single test that used a later eval time would make all
other tests in the file share the [mint, maxt] and potentially evaluate
far more samples than needed.
Fixes : #8019
Signed-off-by: David Leadbeater <dgl@dgl.cx>
2020-10-24 12:03:55 +01:00
Julien Pivotto
4e5b1722b3
Move away from testutil, refactor imports ( #8087 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Arthur Silva Sens
4f45e201cc
Promtool tsdb list now prints block sizes ( #7993 )
...
* promtool tsdb list now prints blocks' size
Signed-off-by: arthursens <arthursens2005@gmail.com>
2020-10-12 23:15:40 +02:00
David Leadbeater
77c784ac93
Ensure alert rules are marked as restored in unit tests ( #7661 )
...
This makes sure the ALERTS timeseries is created when unit testing
alerting rules.
Signed-off-by: David Leadbeater <dgl@dgl.cx>
2020-09-21 18:15:34 +02:00
Julien Pivotto
442b3364d7
Promtool: add evaluation time to instant query ( #7829 )
...
* Promtool: add evaluation time to instant query
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Apply suggestion
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-25 11:32:25 +01:00
Andy Bursavich
4e6a94a27d
Invert service discovery dependencies ( #7701 )
...
This also fixes a bug in query_log_file, which now is relative to the config file like all other paths.
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2020-08-20 13:48:26 +01:00
Harold Dost
21a753c4e2
Make file permissions set to allow for wider umask options. ( #7782 )
...
0644 -> 0666 on all non vendored code.
Fixes #7717
Signed-off-by: Harold Dost <harolddost@gmail.com>
2020-08-12 23:23:17 +02:00
Annanay
9bba8a6eae
Merge branch 'master' into appender-context
...
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-30 16:43:18 +05:30
Annanay
7f98a744e5
Add context to Appender interface
...
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-24 19:40:51 +05:30
chinhnc
e05c19da5d
Display block duration in promtool list blocks command ( #7653 )
...
* Update tsdb.go
Added DURATION column to `tsdb list` command
Signed-off-by: soup <chicknsoupuds@gmail.com>
* Use time.Duration instead of hardcoded hour
Signed-off-by: soup <chicknsoupuds@gmail.com>
2020-07-24 19:01:20 +05:30
Ben Ye
50c261502e
add tsdb cmds into promtool ( #6088 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
update tsdb cli in makefile and promu
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove building tsdb bin
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove useless func
Signed-off-by: yeya24 <yb532204897@gmail.com>
refactor analyzeBlock
Signed-off-by: yeya24 <yb532204897@gmail.com>
Fix Makefile
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2020-07-23 19:35:50 +01:00
Ben Ye
e6ea798c32
promtool range query should exit when fail to parse time ( #7505 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-07-16 23:53:04 +01:00
yeya24
797e48c1a3
support time range in promtool query labels
...
Updated prometheus/client_golang and json-iterator/go
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-07-03 11:29:39 -04:00
Jordan Neufeld
268b4c29e1
Support extended durations in promtool unit tests ( Fixes #6285 ) ( #6297 )
...
* Fixed evaluation_time duration parsing in promtool unit tests (Fixes #6285 )
Signed-off-by: Jordan Neufeld <jordan@neufeldtech.com>
2020-06-15 16:03:07 +01:00
Hongcai Ren
c7e82274c6
replace github.com/prometheus/prometheus/testutil/promlint by github.com/prometheus/client_golang/prometheus/testutil/promlint from our codebase ( #7209 )
...
Signed-off-by: RainbowMango <renhongcai@huawei.com>
2020-05-07 11:34:39 +01:00
Brian Brazil
7646cbca32
Use .UTC everywhere we use time.Unix ( #7066 )
...
time.Unix attaches the local timezone, which can then
leak out (e.g. in the alert json). While this is harmless,
we should be consistent.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-03-29 17:35:39 +01:00
Tobias Guggenmos
454ba12676
Fix build errors in promtool
...
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Harkishen Singh
84e6459c4d
Adds support for line-column numbers for invalid rules, promtool ( #6533 )
...
Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>
2020-01-15 18:07:54 +00:00
akerele abraham
9d39fdad0c
unittest: check for rule files existence ( #6075 )
...
Signed-off-by: akerele abraham <abrahamakerele38@gmail.com>
2019-11-18 13:54:52 -08:00
Chris Marchbanks
1d1f64b4bc
Fix Promtool showing false duplicate rule warnings ( #6270 )
...
Alert rules do not use the Record field, so any alerts with the same
labels and different names would be counted as being duplicates.
Promtool will now consider either field when finding duplicates.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-11-05 11:22:31 -07:00
Simon Pasquier
ddff1480a7
cmd/promtool: improve output for PromQL tests ( #6052 )
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-09-25 09:26:29 +02:00
Harkishen Singh
e097c70e6d
add checks for metrics and display duplicate fields ( #6026 )
...
Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2019-09-20 11:29:47 +01:00
Simon Pasquier
06066a3619
*: improve error messages when parsing bad rules ( #5965 )
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-08-28 17:36:48 +02:00
Sayan Chowdhury
cb66e325d8
Show the warnings during label query ( #5924 )
...
This patch loops through the warnings while querying the label and spits the
output to stderr
Fixes #5885
Signed-off-by: Sayan Chowdhury <sayan.chowdhury2012@gmail.com>
2019-08-24 19:42:21 +02:00
Julius Volz
b5c833ca21
Update go.mod dependencies before release ( #5883 )
...
* Update go.mod dependencies before release
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Add issue for showing query warnings in promtool
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Revert json-iterator back to 1.1.6
It produced errors when marshaling Point values with special float
values.
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Fix expected step values in promtool tests after client_golang update
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Update generated protobuf code after proto dep updates
Signed-off-by: Julius Volz <julius.volz@gmail.com>
2019-08-14 11:00:39 +02:00
Simon Pasquier
75886e0464
cmd/promtool: fix panic with empty exp_labels
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-07-17 17:02:31 +02:00
Tom Wilkie
851131b074
Allow injection of arbitrary headers in promtool, for auth etc. ( #4389 )
...
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-06-30 11:50:23 +01:00
Björn Rabenstein
dc22f74153
Merge pull request #5608 from simonpasquier/external-labels-for-alert-tests
...
cmd/promtool: add $externalLabels for alert unit tests
2019-06-20 16:48:12 +02:00
Björn Rabenstein
372b3438e5
Update prometheus/client_golang to v1.0.0 ( #5682 )
...
Signed-off-by: beorn7 <beorn@grafana.com>
2019-06-17 19:14:36 +01:00
Keenan Romain
55f3a9fe4a
Allows globs for rules when unit testing ( #5595 )
...
* Includes glob support when unit testing rule_files.
Signed-off-by: Keenan Romain <Keenan.Romain@mailchimp.com>
2019-06-12 11:31:07 +01:00
Simon Pasquier
74ff35ccdd
cmd/promtool: add $externalLabels for alert unit tests
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-29 16:40:01 +02:00
beorn7
aff4738f33
Adjust TestQueryRange to new Prometheus API client
...
Signed-off-by: beorn7 <bjoern@rabenste.in>
2019-05-17 18:09:47 +02:00
Simon Pasquier
45506841e6
*: enable all default linters ( #5504 )
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-03 15:11:28 +02:00
Simon Pasquier
9c69eec82a
cmd/promtool: use log.NewNopLogger() ( #5531 )
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-03 10:00:07 +01:00
Björn Rabenstein
0be9388f8d
Merge pull request #5463 from prometheus/beorn7/templating
...
Follow-up on #5009
2019-04-24 16:42:23 +02:00
Simon Pasquier
abc1994bec
cmd/promtool: return errors from rule evaluations ( #5483 )
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-04-23 09:59:03 +02:00
Bjoern Rabenstein
38d518c0fe
Rework #5009 after comments
...
Signed-off-by: Bjoern Rabenstein <bjoern@rabenste.in>
2019-04-17 01:40:10 +02:00
Bjoern Rabenstein
a92ef68dd8
Fix staticcheck errors
...
Not sure why they only show up now.
Signed-off-by: Bjoern Rabenstein <bjoern@rabenste.in>
2019-04-17 01:40:10 +02:00