Commit graph

1069 commits

Author SHA1 Message Date
Krasi Georgiev 31f7990d1d
Re-encode chunks that are still being appended to when snapshoti… (#641)
* re encode all head chunks outside the time range.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-07-03 13:47:31 +03:00
Bartek Płotka dc657df591
Merge pull request #644 from codesome/error-wrap
Check error before wrapping
2019-07-02 11:53:53 +01:00
Ganesh Vernekar 0dcaf10564
Check error before wrapping
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-07-02 16:05:27 +05:30
PrometheusBot ee010e9fab Synchronize Makefile.common from prometheus/prometheus (#640)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-06-26 12:06:29 +03:00
Krasi Georgiev 8d86e921cf
refactor block size calculation (#637)
* refactor block size calculation
The block size is kept in memory and not in the meta file anymore.
It now includes the size of the meta file itself for a more
correct block size.
It fixes a bug where the size didn't change when calling `block.Delete()`.
Adds a dedicated test to ensure correct block sizes.
This allows opening a db in a read only mode as it doesn't write to the meta file anymore.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-24 18:42:29 +03:00
Bartek Płotka b5f9f9f0b4
Merge pull request #636 from krasi-georgiev/rename-bug
fileutil.Replace - remove destination only when a directory.
2019-06-24 13:17:41 +01:00
PrometheusBot d501ae4f6e makefile: update Makefile.common with newer version (#638)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-06-24 11:52:42 +03:00
Krasi Georgiev 251c3a4d64 comment
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-24 11:51:49 +03:00
Krasi Georgiev fe3d374586 Merge branch 'master' into rename-bug
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-24 11:50:41 +03:00
Chris Marchbanks 92d15169fc Print label names with highest cumulative label value length during tsdb analyze (#626)
Signed-off-by: Ryan Scott <rscott@splunk.com>
Co-authored-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-06-21 11:07:50 +03:00
Chris Marchbanks b40cc43958 Provide option to compress WAL records (#609)
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-06-19 16:46:24 +03:00
Krasi Georgiev ef9b3461c5 fixed the err handling and better comment
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-19 14:27:19 +03:00
Krasi Georgiev 5d27fc48a8 fileutil.Replace - remove destination only when a directory.
In cases where a rename fails the fileutil.Replace would delete the
source files/folder.

There is no easy way to make directory renaming atomic, but for files
os.Rename is atomic and replaced the destination file so there is no
need to remove the destination file explicitly.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-19 12:10:51 +03:00
Björn Rabenstein c2c921af75 Update prometheus/client_golang to v1.0.0 (#635)
Signed-off-by: beorn7 <beorn@grafana.com>
2019-06-17 20:54:10 +03:00
Krasi Georgiev 69740485c1
move the wal repair logic in db.Open (#633)
* move the wal repair logic in db.Open

This is to allow opening a wal in a read oly mode without triggering a
repair.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-14 17:39:22 +02:00
Björn Rabenstein 4892da59eb
Merge pull request #628 from prometheus/beorn7/client_golang
Make objectives of Summaries explicit
2019-06-14 15:19:59 +02:00
naivewong b05ca5c36e fix two benchmarks (#631)
* fix two benchmarks

Signed-off-by: naivewong <867245430@qq.com>
2019-06-14 14:30:49 +02:00
beorn7 90a7612df3 Make objectives of Summaries explicit
With the next release of client_golang, Summaries will not have
objectives by default.

As it turns out, for prometheus_tsdb_head_gc_duration_seconds and
prometheus_tsdb_wal_truncate_duration_seconds, the objective-less
default makes more sense then the current default.

To make sure we do the right thing before and after the upcoming
release of client_golang, I have set the objectives explicitly
wherever that was not the case so far:

- prometheus_tsdb_head_gc_duration_seconds and
  prometheus_tsdb_wal_truncate_duration_seconds now have no objectives
  explicitly.
- prometheus_tsdb_wal_fsync_duration_seconds now explicitly uses the
  previous default objectives.

Signed-off-by: beorn7 <beorn@grafana.com>
2019-06-14 14:17:24 +02:00
PrometheusBot cb4a085709 Synchronize Makefile.common from prometheus/prometheus (#632)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-06-14 10:53:17 +02:00
Krasi Georgiev 3549c097b4
add some comments for the different WAL repair tests (#630)
The WAL repair tests look similar so added some extra comments to
clarify their purpose.

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-06-12 16:10:37 +02:00
naivewong 6ab483071a Improvement on postings intersection (#616)
* improvement on postings intersection

Signed-off-by: naivewong <867245430@qq.com>
2019-06-11 10:14:25 +02:00
Krasi Georgiev e809cb477d
Add e2e benchmarks for head and block Queries (#467)
* Add e2e had and block query benchmarks

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-06-07 15:41:44 +02:00
Brian Brazil be4edbe174
Start a new WAL segement on head truncation. (#605)
This reduces disk space usage to not be a minimum of 3 128MB files
in small setups. This will possibly also help debug wal data issues,
by making things a bit more deterministic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-06-07 11:35:02 +01:00
Krasi Georgiev 48c4dc63ea
github template for PRs (#624)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-06-07 10:43:04 +02:00
PrometheusBot 4b9970053f makefile: update Makefile.common with newer version (#625)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-06-07 10:03:51 +02:00
Brian Brazil 149c5dc73a
Handle multiple refs for the same series when WAL reading. (#623)
This can happen if a given series is created/truncated/recreated.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-06-06 14:28:54 +01:00
Frederic Branczyk 731ed22ac8
Merge pull request #622 from prometheus/live-reader-metric-suffix
wal: Add `_total` suffix as metric is a counter
2019-06-05 10:02:23 +02:00
Frederic Branczyk e708ba1d14
wal: Add _total suffix as metric is a counter
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
2019-06-05 09:24:25 +02:00
mrasu 3db36f021b Fix make bench (#610)
Remove  flag -log.level
Signed-off-by: mrasu <m.rasu.hitsuji@gmail.com>
2019-06-03 22:37:24 +02:00
Frederic Branczyk b800a89ae3
Merge pull request #619 from prometheus/remove-promauto
wal: Replace promauto with injected registry
2019-06-03 20:07:15 +02:00
Frederic Branczyk abd15845e2
wal: Replace promauto with injected registry
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
2019-06-03 12:29:52 +02:00
Krasi Georgiev 882162d5b9
fix the "failed compaction" metric. (#613)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-05-30 13:57:28 +02:00
naivewong 13c80a5979 Optimize queries using regex matchers for set lookups (#602)
* Original version of the set optimization

Signed-off-by: naivewong <867245430@qq.com>

* simple set matcher

Signed-off-by: naivewong <867245430@qq.com>

* simple set matcher

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* add benchmark

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update benchmark

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update benchmark

Signed-off-by: naivewong <867245430@qq.com>

* update benchmark

Signed-off-by: naivewong <867245430@qq.com>

* update benchmark

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>

* use genSeries from #467

Signed-off-by: naivewong <867245430@qq.com>

* update

Signed-off-by: naivewong <867245430@qq.com>
2019-05-27 16:54:46 +05:30
Callum Styan 562e93e8e6 Always create a new clean segment when starting the WAL. (#608)
* Always create a new clean segment when starting the WAL.
* Ensure we flush the last page after repairing and before recreating the
new segment in Repair.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-05-24 19:33:27 +01:00
Krasi Georgiev d48a5e2d5c
Release 0.8.0 (#607)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-05-17 13:39:03 +03:00
Callum Styan bce663e1d9 Export the current segment index as a metic. (#601)
* Export the current segment index as a metic.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-05-17 11:47:42 +03:00
Krasi Georgiev 96a87845cc
fix wal panic when page flush fails. (#582)
* fix wal panic when page flush fails.

New records should be added to the page only when the last flush
succeeded. Otherwise the page would be full and panics when trying to
add a new record.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-05-16 16:40:43 +03:00
Brian Brazil 30d0ea59d7 Don't crash on an unknown tombstone ref. (#604)
Fixes https://github.com/prometheus/prometheus/issues/5562

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-05-16 16:36:44 +03:00
PrometheusBot 63d9c2ab18 makefile: update Makefile.common with newer version (#603)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-05-15 10:15:39 +03:00
Brian Brazil 89a90fe96c
Simplify mergedPostings.Seek (#595)
The current implementation leads to very slow behaviour when there's
many lists, this no worse than n log k, where k is the number of posting
lists.

Adjust benchmark to catch this.

Remove flattening of without lists, not needed anymore.

Benchmark versus 0.4.0 (used in Prometheus 2.7):
```
benchmark                                                            old ns/op      new ns/op      delta
BenchmarkHeadPostingForMatchers/n="1"-8                              189907976      188863880      -0.55%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8                      113950106      110791414      -2.77%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8                      104965646      102388760      -2.45%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8                     138743592      104424250      -24.74%
BenchmarkHeadPostingForMatchers/i=~".*"-8                            5279594954     5206096267     -1.39%
BenchmarkHeadPostingForMatchers/i=~".+"-8                            8004610589     6184527719     -22.74%
BenchmarkHeadPostingForMatchers/i=~""-8                              2476042646     1003920432     -59.45%
BenchmarkHeadPostingForMatchers/i!=""-8                              7178244655     6059725323     -15.58%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8              199342649      166642946      -16.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8       215774683      167515095      -22.37%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8                        2214714769     392943663      -82.26%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8                2148727410     322289262      -85.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8              2170658009     338458171      -84.41%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8             235720135      70597905       -70.05%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8       2190570590     343034307      -84.34%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8     2373784439     387297908      -83.68%

benchmark                                                            old allocs     new allocs     delta
BenchmarkHeadPostingForMatchers/n="1"-8                              33             33             +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8                      33             33             +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8                      33             33             +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8                     41             39             -4.88%
BenchmarkHeadPostingForMatchers/i=~".*"-8                            56             56             +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-8                            251577         100115         -60.21%
BenchmarkHeadPostingForMatchers/i=~""-8                              251123         100077         -60.15%
BenchmarkHeadPostingForMatchers/i!=""-8                              251525         100112         -60.20%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8              42             39             -7.14%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8       52             42             -19.23%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8                        251069         100101         -60.13%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8                251473         100101         -60.19%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8              250914         100102         -60.11%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8             30038          11181          -62.78%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8       250813         100105         -60.09%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8     281503         111260         -60.48%

benchmark                                                            old bytes     new bytes     delta
BenchmarkHeadPostingForMatchers/n="1"-8                              10887600      10887600      +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-8                      5456416       5456416       +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-8                      5456416       5456416       +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-8                     5456640       5456544       -0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-8                            258254504     258254472     -0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-8                            520126192     281554792     -45.87%
BenchmarkHeadPostingForMatchers/i=~""-8                              263446640     24908456      -90.55%
BenchmarkHeadPostingForMatchers/i!=""-8                              520121144     281553664     -45.87%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-8              7062448       7062272       -0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-8       7063473       7062384       -0.02%
BenchmarkHeadPostingForMatchers/n="1",i!=""-8                        274325656     35793776      -86.95%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-8                268926824     30362624      -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-8              268882992     30363000      -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-8             33193401      4269304       -87.14%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-8       268875024     30363096      -88.71%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-8     300589656     33099784      -88.99%
```

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-05-13 10:51:07 +01:00
Brian Brazil 6ac81cc7a9
Correctly handle empty labels. (#594)
Currently a time series with empty labels is not treated the same
as one with missing labels. Currently this can only come from
ALERTS&ALERT_FOR_STATE so it's unlikely anyone has actually hit it.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-05-07 11:00:16 +01:00
Goutham Veeramachaneni 2ae028114c
Add FromData function (again)! (#600)
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2019-05-06 19:17:26 +05:30
PrometheusBot 4da65c46c9 makefile: update Makefile.common with newer version (#597)
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-05-05 19:17:35 +03:00
zhulongcheng 3ccab17f5d docs: add label offset table and postings offset table (#593)
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-02 23:23:58 +03:00
Krasi Georgiev 5512826f13
make Close methods for the querier safe to call more than once. (#581)
* make Close methods for the querier safe to call more than once.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-04-30 10:17:07 +03:00
zhulongcheng 19d402d154 cleanup tmp files (#570)
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-04-26 11:27:36 +03:00
Krasi Georgiev 288f67efbf
fix some unchecked errors and remove unused vars. (#592)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-04-25 13:07:04 +03:00
PrometheusBot a10c001689 Synchronize Makefile.common from prometheus/prometheus (#591)
* makefile: update Makefile.common with newer version

Signed-off-by: prombot <prometheus-team@googlegroups.com>

* Add .golangci.yml

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-04-25 12:29:09 +03:00
Brian Brazil c20450564c
Release 0.7.1 (#587)
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-04-24 13:20:45 +01:00
Brian Brazil f9bc0817a1
Avoid allocation in mergedPostings.Seek. (#586)
See
https://github.com/prometheus/prometheus/issues/5424#issuecomment-485199651

benchmark                                                            old
ns/op      new ns/op       delta
BenchmarkHeadPostingForMatchers/n="1"-4
5266           5267            +0.02%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4
6469           7801            +20.59%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4
4984           4963            -0.42%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4
7137           7527            +5.46%
BenchmarkHeadPostingForMatchers/i=~".*"-4
8055830839     8117651746      +0.77%
BenchmarkHeadPostingForMatchers/i=~".+"-4
9369298293     9742710251      +3.99%
BenchmarkHeadPostingForMatchers/i=~""-4
2363120708     2507789029      +6.12%
BenchmarkHeadPostingForMatchers/i!=""-4
9387069195     11740628557     +25.07%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4
109668312      107644136       -1.85%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4
106022679      105759760       -0.25%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
330201810      266421024       -19.32%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
355627801      292774913       -17.67%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
108132305      106697690       -1.33%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
358836972      362579998       +1.04%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
555879392      575858378       +3.59%

benchmark                                                            old
allocs     new allocs     delta
BenchmarkHeadPostingForMatchers/n="1"-4                              8
8              +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4                      11
11             +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4                      11
11             +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4                     17
17             +0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-4                            58
58             +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-4
100115         100115         +0.00%
BenchmarkHeadPostingForMatchers/i=~""-4
100125         100125         +0.00%
BenchmarkHeadPostingForMatchers/i!=""-4
100112         100112         +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4              19
19             +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4       22
22             +0.00%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
100109         100079         -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
100110         100079         -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
11176          11167          -0.08%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
100112         100082         -0.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
111307         111277         -0.03%

benchmark                                                            old
bytes     new bytes     delta
BenchmarkHeadPostingForMatchers/n="1"-4
1360          1360          +0.00%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4
1392          1392          +0.00%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4
1392          1392          +0.00%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4
1520          1520          +0.00%
BenchmarkHeadPostingForMatchers/i=~".*"-4
258254040     258254040     +0.00%
BenchmarkHeadPostingForMatchers/i=~".+"-4
281554280     281554280     +0.00%
BenchmarkHeadPostingForMatchers/i=~""-4
241554888     241554888     +0.00%
BenchmarkHeadPostingForMatchers/i!=""-4
281553664     281553664     +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4
1607277       1607277       +0.00%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4
1607389       1607389       +0.00%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4
73076560      24907600      -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4
73076765      24907723      -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4
5416925       3794909       -29.94%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4
73076779      24907819      -65.92%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4
99874860      51705900      -48.23%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-04-24 13:12:58 +01:00