Commit graph

1004 commits

Author SHA1 Message Date
Alin Sinpalean 171fc4ab5d Limit the returned db.Querier to the requested time range (#351)
Limit the returned `db.Querier` to the requested time range. Preallocate the `baseChunkSeries.lset` and `baseChunkSeries.chks` slices to the previous series' slice sizes to avoid unnecessary grow slice reallocations.
2018-11-09 15:54:56 +02:00
Krasi Georgiev e4843938ba
add missing zero to tombstone magic number (#448)
add missing zero to tombstone magic number constant.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-09 13:37:02 +02:00
Krasi Georgiev a9470dd8d5
few more comments to explain the WAL workflow (#430)
More comments for the WAL package.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-08 10:27:16 +02:00
Ganesh Vernekar 3a08a71d86 LabelNames() method to get all unique label names (#369)
* LabelNames() method to get all unique label names

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-07 17:52:41 +02:00
Ganesh Vernekar a95323c021 Add license headers to missing files (#447)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-06 20:19:42 +02:00
Brian Brazil c7e7fd355e Only send WAL read workers the samples they need.
Calculating the modulus in each worker was a hotspot,
and meant that you had more work to do the more cores you had.
This cuts CPU usage (on my 8 core, 4 real core machine) by
33%, and walltime by 3%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 22:52:26 +00:00
Brian Brazil a64b0d51c4 Precalculate memSeries.head
This is read far more than it changes.
This cuts ~14% off walltme and ~27% off CPU for WAL reading.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 15:49:42 +00:00
Brian Brazil d8c8e4e6e4 Keep local cache of ids.
With the various goroutines running, the locking
in getByID is notable. This cuts cpu usage by ~25%
and walltime by ~20%.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 15:49:42 +00:00
Brian Brazil f0e79ec264 Actually reuse samples in loadWAL across records.
This cuts walltime by 2.5X and CPU by 2X

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-10-31 15:49:42 +00:00
Ye Ben 23a5f09085 fix a typo dont -> dont't (#438)
Signed-off-by: yeya24 <ben.ye@daocloud.io>
2018-10-31 13:49:57 +02:00
Krasi Georgiev ae91febcbb
Update the files format README.md (#437) 2018-10-29 19:51:47 +02:00
Krasi Georgiev d804a27062
refactor util funcs to allow re-usage. (#419)
* refactor util funcs to allow reusage.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-25 21:06:19 +01:00
Ben a8351dc9d0 Using filepath.Join() instead of strings with slashes (#428)
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
2018-10-25 10:32:57 +01:00
Chris Marchbanks f4afc7dff2 Add benchmark for querying a persisted block (#425)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-10-24 00:35:52 +03:00
Krasi Georgiev 1dd9a6bd29
comments about the 120samples const and link to Gorilla papers. (#423)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-23 13:43:06 +03:00
Krasi Georgiev 66b6b87cd4 fix windows tests (#421)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-22 15:19:52 -04:00
Thomas Jackson b4132df5f7 Reduce allocations for queries on HEAD (#417)
Some benchmarks for HEAD and allocate the correct slice size in LabelValues , we already know what it'll be

This is ~15% time improvement, and ~25% allocation improvement:


```
benchmark                             old ns/op     new ns/op     delta
BenchmarkHeadPostingForMatchers-4     74452         63514         -14.69%

benchmark                             old allocs     new allocs     delta
BenchmarkHeadPostingForMatchers-4     20             13             -35.00%

benchmark                             old bytes     new bytes     delta
BenchmarkHeadPostingForMatchers-4     5425          3137          -42.18%
```

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2018-10-22 13:52:01 +03:00
Simon Pasquier 18af5763d8
Merge pull request #409 from simonpasquier/remove-prometheus-dep
fileutil: remove dependency on prometheus/prometheus
2018-10-16 10:15:06 +02:00
Simon Pasquier 4dd740e4cc fileutil: remove dependency on prometheus/prometheus
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-10-15 17:21:20 +02:00
Krasi Georgiev cf2af2b371
add maintainers file (#404)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-15 14:32:00 +03:00
Bartek Płotka 047b1b1357 compact: Verify for chunks outside of compacted time range. Added unit test for populateBlocs. (#349)
* compact: Verify for chunks outside of compacted time range. 
 Unit test for populateBlocs.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-12 12:45:19 +03:00
Krasi Georgiev d7492b9350
more descriptive var names and some more logging. (#405)
* more descriptive checkpoint var names and some more logging.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-11 18:23:52 +03:00
Camille Janicki 0ce41118ed Add msg parameter to Equals function in testutil (#398)
* Add msg parameter to Equals function in testutil

Co-authored-by: Chris Marchbanks <csmarchbanks@gmail.com>
Signed-off-by: Camille Janicki <camille.janicki@gmail.com>
2018-10-03 11:08:31 +03:00
Kautilya Tripathi 3fd6d2f920 more meaningful names for serializedStringTuples and stringTuples (#377)
more meaningful names for serializedStringTuples and stringTuples structs

Signed-off-by: knrt10 <tripathi.kautilya@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-02 21:03:12 +03:00
Simon Pasquier 043e3bb5f9
Merge pull request #396 from codesome/new-metrics
Add new metrics.
2018-10-01 16:00:01 +02:00
Ganesh Vernekar 2638b587f6
Merge remote-tracking branch 'upstream/master' into new-metrics
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-30 20:20:00 +05:30
Ganesh Vernekar 61d0868966 Fix TestCorrectNumTombstones (#399)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-28 13:26:29 +03:00
Ganesh Vernekar 61b000ee0e
Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-28 15:00:51 +05:30
Catalin Patulea b4c7c80227 Reword chunk references, LSB usually means 'bits'. (#364)
https://en.wikipedia.org/wiki/Bit_numbering#Least_significant_bit

Signed-off-by: Catalin Patulea <catalinp@google.com>
2018-09-27 21:38:02 +03:00
Ganesh Vernekar 6e712963e2 Fix updating of NumTombstones in block.Delete(..) (#385)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-27 14:43:22 +03:00
Ganesh Vernekar 632dfb349e
Add new metrics.
1. 'prometheus_tsdb_wal_truncate_fail' for failed WAL truncation.
2. 'prometheus_tsdb_checkpoint_delete_fail' for failed old checkpoint delete.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-25 18:50:57 +05:30
Callum Styan a971f52ac8 clean up after running repair tests (#372)
implement a `CopyDirs` helper to walk all dirs and subdirs and copy all files within to a destination folder.

Refactor `TestRepairBadIndexVersion` to use the `CopyDirs` so it works on a copy db and remove the copy after the test.
This allows running the test more than once and doesn't confuse git.

Signed-off-by: Callum Stytan <callumstyan@gmail.com>
2018-09-21 20:35:33 +03:00
Martin Chodur dbd765a0e3 Refactor Makefile to use Makefile.common (#324)
make use of the Makefile.common

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
2018-09-21 11:08:38 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
Krasi Georgiev d38516b1c2
remove unused changes variable (#391)
This was added in
55a9b5428a

and later not used after some refactoring in following PRs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 09:24:01 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-21 09:23:01 +03:00
Ganesh Vernekar e779603633 Reuse testutil (#393)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-21 11:47:59 +05:30
Goutham Veeramachaneni 9c8ca47399
Fix filehandling for windows (#392)
* Fix filehandling for windows

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix more windows filehandling issues

Windows: Close files before deleting Checkpoints.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close writers in case of errors so they can be deleted

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close block so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close file to delete it

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close dir so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: close files so that they can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-21 11:01:22 +05:30
Julius Volz 5ae6c60d39 Handle a bunch of unchecked errors (#365)
As discovered by "gosec".

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-09-20 11:33:52 +03:00
Fabian Reinartz dfcb7d0d50
Merge pull request #389 from gouthamve/wal-corruptio
Make sure WAL Repair can handle wrapped errors
2018-09-19 08:47:24 +02:00
Goutham Veeramachaneni c7d0d10da4
Make sure WAL Repair can handle wrapped errors
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-19 12:12:25 +05:30
Björn Rabenstein d1a6edae6f
Merge pull request #386 from prometheus/beorn7/client_golang
Revert "Remove `prometheus_` prefix from metrics"
2018-09-18 19:27:31 +02:00
beorn7 3bc6c670fa Revert "Remove prometheus_ prefix from metrics"
This reverts commit 98fe30438c.

After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-18 19:19:19 +02:00
Chris Marchbanks a8966cb53d Fix race condition between gc and committing (#378)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-17 19:58:42 +03:00
Callum Styan 722f0ab920 break MigrateWAL into two functions, detection and migration (#371)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-09-17 19:30:56 +03:00
Frederic Branczyk 1b651ea7d4
Merge pull request #383 from prometheus/beorn7/client_golang
Remove `prometheus_` prefix from metrics
2018-09-17 15:16:04 +02:00
beorn7 98fe30438c Remove prometheus_ prefix from metrics
This can now be added by users of the library as needed with the new
https://godoc.org/github.com/prometheus/client_golang/prometheus#WrapRegistererWithPrefix

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-17 14:54:28 +02:00
Krasi Georgiev ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e (#382)
no logic changes just uncommented and fixed the issues.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-17 14:28:55 +03:00
Bob Shannon cb7f320d42 Expose prometheus_tsdb_lowest_timestamp metric (#363)
* Expose prometheus_tsdb_start_time_seconds metric

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* Search for block with smallest minTime

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comments

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comment: Make metric name more accurate

Signed-off-by: Bob Shannon <bshannon@palantir.com>
2018-09-14 17:37:45 +05:30
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345)
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-09-13 19:34:26 +03:00