Commit graph

824 commits

Author SHA1 Message Date
Ganesh Vernekar 632dfb349e
Add new metrics.
1. 'prometheus_tsdb_wal_truncate_fail' for failed WAL truncation.
2. 'prometheus_tsdb_checkpoint_delete_fail' for failed old checkpoint delete.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-25 18:50:57 +05:30
Callum Styan a971f52ac8 clean up after running repair tests (#372)
implement a `CopyDirs` helper to walk all dirs and subdirs and copy all files within to a destination folder.

Refactor `TestRepairBadIndexVersion` to use the `CopyDirs` so it works on a copy db and remove the copy after the test.
This allows running the test more than once and doesn't confuse git.

Signed-off-by: Callum Stytan <callumstyan@gmail.com>
2018-09-21 20:35:33 +03:00
Martin Chodur dbd765a0e3 Refactor Makefile to use Makefile.common (#324)
make use of the Makefile.common

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
2018-09-21 11:08:38 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
Krasi Georgiev d38516b1c2
remove unused changes variable (#391)
This was added in
55a9b5428a

and later not used after some refactoring in following PRs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 09:24:01 +03:00
Chris Marchbanks 2db59a71a6 Fix assert order being backwards in a few places (#388)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-21 09:23:01 +03:00
Ganesh Vernekar e779603633 Reuse testutil (#393)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-21 11:47:59 +05:30
Goutham Veeramachaneni 9c8ca47399
Fix filehandling for windows (#392)
* Fix filehandling for windows

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix more windows filehandling issues

Windows: Close files before deleting Checkpoints.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close writers in case of errors so they can be deleted

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close block so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close file to delete it

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close dir so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: close files so that they can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-21 11:01:22 +05:30
Julius Volz 5ae6c60d39 Handle a bunch of unchecked errors (#365)
As discovered by "gosec".

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-09-20 11:33:52 +03:00
Fabian Reinartz dfcb7d0d50
Merge pull request #389 from gouthamve/wal-corruptio
Make sure WAL Repair can handle wrapped errors
2018-09-19 08:47:24 +02:00
Goutham Veeramachaneni c7d0d10da4
Make sure WAL Repair can handle wrapped errors
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-19 12:12:25 +05:30
Björn Rabenstein d1a6edae6f
Merge pull request #386 from prometheus/beorn7/client_golang
Revert "Remove `prometheus_` prefix from metrics"
2018-09-18 19:27:31 +02:00
beorn7 3bc6c670fa Revert "Remove prometheus_ prefix from metrics"
This reverts commit 98fe30438c.

After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-18 19:19:19 +02:00
Chris Marchbanks a8966cb53d Fix race condition between gc and committing (#378)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-17 19:58:42 +03:00
Callum Styan 722f0ab920 break MigrateWAL into two functions, detection and migration (#371)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-09-17 19:30:56 +03:00
Frederic Branczyk 1b651ea7d4
Merge pull request #383 from prometheus/beorn7/client_golang
Remove `prometheus_` prefix from metrics
2018-09-17 15:16:04 +02:00
beorn7 98fe30438c Remove prometheus_ prefix from metrics
This can now be added by users of the library as needed with the new
https://godoc.org/github.com/prometheus/client_golang/prometheus#WrapRegistererWithPrefix

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-17 14:54:28 +02:00
Krasi Georgiev ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e (#382)
no logic changes just uncommented and fixed the issues.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-17 14:28:55 +03:00
Bob Shannon cb7f320d42 Expose prometheus_tsdb_lowest_timestamp metric (#363)
* Expose prometheus_tsdb_start_time_seconds metric

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* Search for block with smallest minTime

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comments

Signed-off-by: Bob Shannon <bshannon@palantir.com>

* PR comment: Make metric name more accurate

Signed-off-by: Bob Shannon <bshannon@palantir.com>
2018-09-14 17:37:45 +05:30
Alexey Palazhchenko 9be883670e fix the test data file path in benchmarks. (#345)
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-09-13 19:34:26 +03:00
Ganesh Vernekar 2945db18ca Changes in series names (and types) exposed (#376)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-12 14:39:02 +05:30
Goutham Veeramachaneni d9129adb52
Merge pull request #375 from codesome/symbol-table-size
Added metric for symbol table size
2018-09-11 17:43:30 +05:30
Ganesh Vernekar 77d2c7cf3d
Added metric for symbol table size
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-08 23:58:36 +05:30
Simon Pasquier eeb3a7435a
Merge pull request #319 from simonpasquier/improve-repair-err
repair: improve error message
2018-08-07 16:40:27 +02:00
Fabian Reinartz 06f01d45ad
Merge pull request #366 from prometheus/32bit
Fixes for 32bit archs
2018-08-07 06:25:08 -05:00
Fabian Reinartz 22cae653d8 Fixes for 32bit archs
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-07 06:52:16 -04:00
Fabian Reinartz 2a0e96eb19
Merge pull request #332 from prometheus/newwal
wal: add write ahead log package
2018-08-07 04:14:15 -05:00
Fabian Reinartz 74b3501161 Fix Rename call
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-03 11:25:27 -04:00
Fabian Reinartz 76990518d3
Merge pull request #340 from prometheus/wal_migrate
Migrate write ahead log
2018-08-02 16:58:17 -05:00
Fabian Reinartz ee7ee059ef Fix doc comments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-02 17:57:34 -04:00
Fabian Reinartz a9a8fabd2c
Merge pull request #339 from prometheus/inittime
Properly initialize head time
2018-08-02 16:54:38 -05:00
Fabian Reinartz f8ec0074e7 Add Replace function
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-02 17:51:49 -04:00
Fabian Reinartz b81e0fbf2a Address comments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-20 02:26:12 -04:00
Fabian Reinartz 45071c657c Properly initialize head time
This fixes various issues when initializing the head time range
under different starting conditions.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:41:02 -04:00
Fabian Reinartz 22fd3ef24e Deal with zero-length segments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Fabian Reinartz 92e1b20957 Fix close handling
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Fabian Reinartz 1a5573b4ce Migrate write ahead log
On startup, rewrite the old write ahead log into the new format once.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Fabian Reinartz 3e76f0163e Address comments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz 0ad2b8a349 docs: add new WAL format
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz 3f538817f8 move WAL lock
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz d951140ab8 wal: avoid heap allocation in WAL reader
The buffers we allocated were escaping to the heap, resulting in large
memory usage spikes during startup and checkpointing in Prometheus.
This attaches the buffer to the reader object to prevent this.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz 7841d417b3 Ensure blocks are time-ordered in memory
We assume in multiple places that the block list held by DB
has blocks sequential by time.
A regression caused us to hold them ordered by ULID, i.e. by creation
time instead.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz def912ce0e Integrate new WAL and checkpoints
Remove the old WAL and drop in the new one

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
Fabian Reinartz 008399a6e0 Add checkpointing of WAL segments
Create checkpoints from a sequence of WAL segments while filtering
out obsolete data. The checkpoint format is again a sequence of WAL
segments, which allows us to reuse the serialization format and
implementation.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:24:40 -04:00
Fabian Reinartz 449a2d0db7 wal: add segment type and repair procedure
Allow to repair the WAL based on the error returned by a reader
during a full scan over all records.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:24:40 -04:00
Fabian Reinartz 8e1f97fad4 wal: add write ahead log package
This adds a new WAL that's agnostic to the actual record contents.
It's much simpler and should be more resilient than the existing one.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:24:40 -04:00
Fabian Reinartz 99a2c4314f
Merge pull request #330 from codwu/tsdb-delete
add rwmutex to prevent concurrent map read when delete series
2018-07-11 13:21:26 +02:00
codwu bc6ef0b94e rename mts to intvlGroups
Signed-off-by: codwu <wuhan9087@163.com>
2018-07-10 21:24:13 +08:00
codwu e4444ca48c update addInterval function and test.
Signed-off-by: codwu <wuhan9087@163.com>
2018-07-06 20:30:27 +08:00
codwu 667e539a7a Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete 2018-07-06 20:21:32 +08:00