Commit graph

97 commits

Author SHA1 Message Date
zhulongcheng 19d402d154 cleanup tmp files (#570)
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-04-26 11:27:36 +03:00
Alec e7436e13f0 Merge encoding_helpers.go to tsdbutil (#526)
remove duplicate encoding helper funcs and move to own package so they can be reused.

Signed-off-by: naivewong <867245430@qq.com>
2019-02-22 19:11:11 +02:00
Krasi Georgiev 48c439d26d
fix statick check errors (#475)
fix the tests for `check_license` and `staticcheck`

the static check also found some actual bugs.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-02 19:48:42 +03:00
Krasi Georgiev 2e0571caba
remove unused WALFlushInterval option and NopWAL struct (#468)
The WALFlushInterval is not used anywhere in the code base.
The WAL is not an interface anymore to save some lookup time so can't use NopWAL in the tests. Instead can just pass nil as the code checks for that and it is essentially a noop.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-28 20:42:46 +03:00
JoeWrightss c8c03ff85b fix some typos (#466)
Signed-off-by: JoeWrightss <zhoulin.xie@daocloud.io>
2018-12-07 12:24:02 +03:00
Krasi Georgiev d05611c027
removed some unused code and moved mockSeriesSet in querier_test (#394)
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
Goutham Veeramachaneni 9c8ca47399
Fix filehandling for windows (#392)
* Fix filehandling for windows

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix more windows filehandling issues

Windows: Close files before deleting Checkpoints.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close writers in case of errors so they can be deleted

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close block so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close file to delete it

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: Close dir so that it can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Windows: close files so that they can be deleted.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2018-09-21 11:01:22 +05:30
beorn7 3bc6c670fa Revert "Remove prometheus_ prefix from metrics"
This reverts commit 98fe30438c.

After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-18 19:19:19 +02:00
Callum Styan 722f0ab920 break MigrateWAL into two functions, detection and migration (#371)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-09-17 19:30:56 +03:00
beorn7 98fe30438c Remove prometheus_ prefix from metrics
This can now be added by users of the library as needed with the new
https://godoc.org/github.com/prometheus/client_golang/prometheus#WrapRegistererWithPrefix

Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-17 14:54:28 +02:00
Fabian Reinartz 74b3501161 Fix Rename call
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-03 11:25:27 -04:00
Fabian Reinartz ee7ee059ef Fix doc comments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-08-02 17:57:34 -04:00
Fabian Reinartz 22fd3ef24e Deal with zero-length segments
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Fabian Reinartz 92e1b20957 Fix close handling
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Fabian Reinartz 1a5573b4ce Migrate write ahead log
On startup, rewrite the old write ahead log into the new format once.

Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:34:18 -04:00
Alexey Palazhchenko 9283c68e7d Fix ineffective assignment.
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-05-07 16:02:49 +03:00
Mario Trangoni b7173eb0e5 fix some comments typos (#315) 2018-04-08 10:28:30 +01:00
Mario Trangoni c0e888e82b fix megacheck issues: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd. 2018-03-21 22:39:15 +01:00
Tobias Schmidt 6027af95ca Fix missing prefix of tsdb_wal_* metrics 2017-11-08 14:58:03 +01:00
Fabian Reinartz b1df857819
Merge pull request #190 from prometheus/walsync
wal: synchronize background operations
2017-11-01 18:11:22 +01:00
Fabian Reinartz a993f0ccc0
Fix typo 2017-11-01 18:11:09 +01:00
ranbochen a27cf34a36 fix bugs on platform windows to pass all test case. (#192)
* fix bugs on platform windows to pass all test case.

* fix bugs on platform windows to pass all test case

* clean up codes
2017-10-31 15:37:41 +01:00
Fabian Reinartz 8aedb7671e wal: synchronize background operations
This adds an actor channel and thereby serializes all background
operations through the run() method.
Fixes an existing race.
2017-10-30 09:50:21 +01:00
Fabian Reinartz 80055bb95b Truncate segments on broken header 2017-10-20 13:16:44 +02:00
Fabian Reinartz c3e502b194 Merge pull request #168 from prometheus/fasterwal
wal: decode and process in separate threads.
2017-10-10 18:11:44 +02:00
Fabian Reinartz 7efb830d70 wal: parallelize sample processing 2017-10-09 15:22:38 +02:00
Fabian Reinartz d3682d701c wal: decode and process in separate threads. 2017-10-06 14:46:52 +02:00
Simon Pasquier 3e17cd1621 Instrument WAL corruptions 2017-10-06 13:50:20 +02:00
Fabian Reinartz 27f1b8aac3 Merge pull request #162 from BasPH/fsync-duration
Instrument WAL fsync
2017-10-05 08:18:36 +02:00
Bas Harenslak 5e1c258a98 Instrument WAL fsync 2017-10-04 22:17:02 +02:00
Goutham Veeramachaneni da565f975e Merge pull request #161 from prometheus/fileutil
Remove dependency on etcd/pkg/fileutil
2017-10-04 17:08:54 +05:30
Fabian Reinartz bbe72dccb9 Remove dependency on etcd/pkg/fileutil 2017-10-04 10:23:41 +02:00
Goutham Veeramachaneni c35d3a65bd
Add levels to all log lines.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-28 12:49:34 +05:30
Fabian Reinartz 1e88ba06b4 Use boolean function instead of postings to drop WAL series
There is not guarantee or requirement for WAL writers to only add
series entries in increasing order of IDs. A postings list cannot look
back and thus unordered WAL entries would skip over IDs to not truncate
from the WAL.
We replace it with a simple boolean check function that does not require
order.
2017-09-21 13:31:01 +02:00
Fabian Reinartz 162a48e4f2 Create series with ID recorded in WAL when reading it back 2017-09-19 11:31:16 +02:00
Fabian Reinartz 7ada9cd805 Simplify series create logic in head 2017-09-18 12:38:39 +02:00
Fabian Reinartz 6892fc6dcb Finish old WAL segment async, default to no fsync
We were still fsyncing while holding the write lock when we cut a new
segment. Given we cannot do anything but logging errors, we might just
as well complete segments asynchronously.

There's not realistic use case where one would fsync after every WAL
entry, thus make the default of a flush interval of 0 to never fsync
which is a much more likely use case.
2017-09-08 18:41:12 +02:00
Fabian Reinartz 1d5f85817d Fix various races 2017-09-08 08:48:19 +02:00
Fabian Reinartz c2916736be Re-use slices on deocding the WAL. Fix tests. 2017-09-08 08:17:01 +02:00
Fabian Reinartz 0db4c227b7 Fix min/max time handling and concurrent crc32 usage 2017-09-07 13:04:02 +02:00
Fabian Reinartz 970bffec8d Fix WAL errors and add tests for it 2017-09-07 10:58:34 +02:00
Fabian Reinartz 81222849bc Filter WAL data in Head, misc fixes 2017-09-06 16:20:37 +02:00
Fabian Reinartz 33e9bdf403 WAL refactoring and truncation fixes and test 2017-09-06 14:59:25 +02:00
Fabian Reinartz c36d574290 Replace single head lock with granular locks
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00
Goutham Veeramachaneni 1698c516ad [WIP]: WAL implementation
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-04 14:52:40 +02:00
Fabian Reinartz 893b6ec506 Add tests for GC and chunk truncation 2017-09-01 14:38:49 +02:00
Fabian Reinartz 5cf2662074 Refactor WAL into Head and misc improvements 2017-09-01 11:50:58 +02:00
Fabian Reinartz 3901b6e70b Remove multiple heads
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Alin Sinpalean 8f7934d025 Take the fdatasync() syscall out from under mutex lock to (mostly) avoid scrape and eval from blocking on disk I/O. 2017-09-01 09:45:54 +02:00