Commit graph

846 commits

Author SHA1 Message Date
Brian Brazil d8c8e4e6e4 Keep local cache of ids.
With the various goroutines running, the locking
in getByID is notable. This cuts cpu usage by ~25%
and walltime by ~20%.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 15:49:42 +00:00
Brian Brazil f0e79ec264 Actually reuse samples in loadWAL across records.
This cuts walltime by 2.5X and CPU by 2X

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 15:49:42 +00:00
Krasi Georgiev ae91febcbb
Update the files format README.md (#437) 2018-10-29 19:51:47 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419)
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-25 21:06:19 +01:00
Ben a8351dc9d0 Using filepath.Join() instead of strings with slashes (#428)
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
2018-10-25 10:32:57 +01:00
Chris Marchbanks f4afc7dff2 Add benchmark for querying a persisted block (#425)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-10-24 00:35:52 +03:00
Krasi Georgiev 1dd9a6bd29
comments about the 120samples const and link to Gorilla papers. (#423)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-23 13:43:06 +03:00
Krasi Georgiev 66b6b87cd4 fix windows tests (#421)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-22 15:19:52 -04:00
Thomas Jackson b4132df5f7 Reduce allocations for queries on HEAD (#417)
Some benchmarks for HEAD and allocate the correct slice size in LabelValues , we already know what it'll be

This is ~15% time improvement, and ~25% allocation improvement:


```
benchmark                             old ns/op     new ns/op     delta
BenchmarkHeadPostingForMatchers-4     74452         63514         -14.69%

benchmark                             old allocs     new allocs     delta
BenchmarkHeadPostingForMatchers-4     20             13             -35.00%

benchmark                             old bytes     new bytes     delta
BenchmarkHeadPostingForMatchers-4     5425          3137          -42.18%
```

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2018-10-22 13:52:01 +03:00
Simon Pasquier 18af5763d8
Merge pull request #409 from simonpasquier/remove-prometheus-dep
fileutil: remove dependency on prometheus/prometheus
2018-10-16 10:15:06 +02:00
Simon Pasquier 4dd740e4cc fileutil: remove dependency on prometheus/prometheus
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-10-15 17:21:20 +02:00
Krasi Georgiev cf2af2b371
add maintainers file (#404)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-15 14:32:00 +03:00
Bartek Płotka 047b1b1357 compact: Verify for chunks outside of compacted time range. Added unit test for populateBlocs. (#349)
* compact: Verify for chunks outside of compacted time range. 
 Unit test for populateBlocs.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-12 12:45:19 +03:00
Krasi Georgiev d7492b9350
more descriptive var names and some more logging. (#405)
* more descriptive checkpoint var names and some more logging.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-11 18:23:52 +03:00
Camille Janicki 0ce41118ed Add msg parameter to Equals function in testutil (#398)
* Add msg parameter to Equals function in testutil

Co-authored-by: Chris Marchbanks <csmarchbanks@gmail.com>
Signed-off-by: Camille Janicki <camille.janicki@gmail.com>
2018-10-03 11:08:31 +03:00
Kautilya Tripathi 3fd6d2f920 more meaningful names for serializedStringTuples and stringTuples (#377)
more meaningful names for serializedStringTuples and stringTuples structs

Signed-off-by: knrt10 <tripathi.kautilya@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-02 21:03:12 +03:00
Simon Pasquier 043e3bb5f9
Merge pull request #396 from codesome/new-metrics
Add new metrics.
2018-10-01 16:00:01 +02:00
Ganesh Vernekar 2638b587f6
Merge remote-tracking branch 'upstream/master' into new-metrics
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-30 20:20:00 +05:30
Ganesh Vernekar 61d0868966 Fix TestCorrectNumTombstones (#399)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-28 13:26:29 +03:00
Ganesh Vernekar 61b000ee0e
Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-28 15:00:51 +05:30
Catalin Patulea b4c7c80227 Reword chunk references, LSB usually means 'bits'. (#364)
https://en.wikipedia.org/wiki/Bit_numbering#Least_significant_bit

Signed-off-by: Catalin Patulea <catalinp@google.com>
2018-09-27 21:38:02 +03:00
Ganesh Vernekar 6e712963e2 Fix updating of NumTombstones in block.Delete(..) (#385)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-27 14:43:22 +03:00
Ganesh Vernekar 632dfb349e
Add new metrics.
1. 'prometheus_tsdb_wal_truncate_fail' for failed WAL truncation.
2. 'prometheus_tsdb_checkpoint_delete_fail' for failed old checkpoint delete.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-25 18:50:57 +05:30
Callum Styan a971f52ac8 clean up after running repair tests (#372)
implement a `CopyDirs` helper to walk all dirs and subdirs and copy all files within to a destination folder.

Refactor `TestRepairBadIndexVersion` to use the `CopyDirs` so it works on a copy db and remove the copy after the test.
This allows running the test more than once and doesn't confuse git.

Signed-off-by: Callum Stytan <callumstyan@gmail.com>
2018-09-21 20:35:33 +03:00
Martin Chodur dbd765a0e3 Refactor Makefile to use Makefile.common (#324)
make use of the Makefile.common

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
2018-09-21 11:08:38 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
Krasi Georgiev d38516b1c2
remove unused changes variable (#391)
This was added in
55a9b5428a

and later not used after some refactoring in following PRs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 09:24:01 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-21 09:23:01 +03:00
Ganesh Vernekar e779603633 Reuse testutil (#393)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-21 11:47:59 +05:30
Goutham Veeramachaneni 9c8ca47399
Fix filehandling for windows (#392)
* Fix filehandling for windows

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix more windows filehandling issues

Windows: Close files before deleting Checkpoints.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close writers in case of errors so they can be deleted

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close block so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close file to delete it

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close dir so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: close files so that they can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-21 11:01:22 +05:30
Julius Volz 5ae6c60d39 Handle a bunch of unchecked errors (#365)
As discovered by "gosec".

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-09-20 11:33:52 +03:00
Fabian Reinartz dfcb7d0d50
Merge pull request #389 from gouthamve/wal-corruptio
Make sure WAL Repair can handle wrapped errors
2018-09-19 08:47:24 +02:00
Goutham Veeramachaneni c7d0d10da4
Make sure WAL Repair can handle wrapped errors
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-19 12:12:25 +05:30
Björn Rabenstein d1a6edae6f
Merge pull request #386 from prometheus/beorn7/client_golang
Revert "Remove `prometheus_` prefix from metrics"
2018-09-18 19:27:31 +02:00
beorn7 3bc6c670fa Revert "Remove prometheus_ prefix from metrics"
This reverts commit 98fe30438c.

After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-18 19:19:19 +02:00
Chris Marchbanks a8966cb53d Fix race condition between gc and committing (#378)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-17 19:58:42 +03:00
Callum Styan 722f0ab920 break MigrateWAL into two functions, detection and migration (#371)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-09-17 19:30:56 +03:00
Frederic Branczyk 1b651ea7d4
Merge pull request #383 from prometheus/beorn7/client_golang
Remove `prometheus_` prefix from metrics
2018-09-17 15:16:04 +02:00
beorn7 98fe30438c Remove prometheus_ prefix from metrics
This can now be added by users of the library as needed with the new
https://godoc.org/github.com/prometheus/client_golang/prometheus#WrapRegistererWithPrefix

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-17 14:54:28 +02:00
Krasi Georgiev ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e (#382)
no logic changes just uncommented and fixed the issues.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-17 14:28:55 +03:00
Bob Shannon cb7f320d42 Expose prometheus_tsdb_lowest_timestamp metric (#363)
* Expose prometheus_tsdb_start_time_seconds metric

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* Search for block with smallest minTime

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comments

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comment: Make metric name more accurate

Signed-off-by: Bob Shannon <bshannon@palantir.com>
2018-09-14 17:37:45 +05:30
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345)
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-09-13 19:34:26 +03:00
Ganesh Vernekar 2945db18ca Changes in series names (and types) exposed (#376)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-12 14:39:02 +05:30
Goutham Veeramachaneni d9129adb52
Merge pull request #375 from codesome/symbol-table-size
Added metric for symbol table size
2018-09-11 17:43:30 +05:30
Ganesh Vernekar 77d2c7cf3d
Added metric for symbol table size
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-08 23:58:36 +05:30
Simon Pasquier eeb3a7435a
Merge pull request #319 from simonpasquier/improve-repair-err
repair: improve error message
2018-08-07 16:40:27 +02:00
Fabian Reinartz 06f01d45ad
Merge pull request #366 from prometheus/32bit
Fixes for 32bit archs
2018-08-07 06:25:08 -05:00
Fabian Reinartz 22cae653d8 Fixes for 32bit archs
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-07 06:52:16 -04:00
Fabian Reinartz 2a0e96eb19
Merge pull request #332 from prometheus/newwal
wal: add write ahead log package
2018-08-07 04:14:15 -05:00
Fabian Reinartz 74b3501161 Fix Rename call
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-03 11:25:27 -04:00