Commit graph

261 commits

Author SHA1 Message Date
Fabian Reinartz 81222849bc Filter WAL data in Head, misc fixes 2017-09-06 16:20:37 +02:00
Fabian Reinartz 33e9bdf403 WAL refactoring and truncation fixes and test 2017-09-06 14:59:25 +02:00
Fabian Reinartz c36d574290 Replace single head lock with granular locks
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Fabian Reinartz 1ddedf2b30 Change series ID from uint32 to uint64 2017-09-04 16:08:38 +02:00
Goutham Veeramachaneni 1698c516ad [WIP]: WAL implementation
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-04 14:52:40 +02:00
Fabian Reinartz 893b6ec506 Add tests for GC and chunk truncation 2017-09-01 14:38:49 +02:00
Fabian Reinartz 4f037da462 Remove defer statement in hot path 2017-09-01 12:09:29 +02:00
Fabian Reinartz 5cf2662074 Refactor WAL into Head and misc improvements 2017-09-01 11:50:58 +02:00
Fabian Reinartz 8209e3ec23 Add various metrics 2017-09-01 11:50:58 +02:00
Fabian Reinartz 3901b6e70b Remove multiple heads
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Goutham Veeramachaneni 7438ed7035 Expose Intervals type for use by TombstoneReader.
TombstoneReader is exposed but Intervals is not.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-25 16:06:36 +05:30
Fabian Reinartz 905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz 66ff7b12e9 Pool Chunk objects during compaction 2017-08-08 17:35:34 +02:00
Fabian Reinartz 2644c8665c Don't allocate ChunkMetas, reuse postings slices 2017-08-06 20:41:24 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Goutham Veeramachaneni f1ae239c20 Persist the right MaxTime when snapshotting
This is because we cut a new block from where the snapshotted block ends
if we restore from backups and highTimestamp would be where we should be
 starting from.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 13:48:13 +02:00
Fabian Reinartz 1e74c155eb Return empty string to signal non-caching 2017-06-26 14:58:00 +02:00
Fabian Reinartz 3410559c1b Compact head block early
Let older head blocks be compacted once the newest once has samples at
50% of its total range. This allows the memory of the compacted blocks
to be released and garbage collected before a new head block gets
created. Thereby the number of head blocks is 1 or 2 instead of 2 or 3
and memory spikes are reduced.
2017-06-26 08:52:59 +02:00
Fabian Reinartz 9963a4c7c3 Merge pull request #95 from Gouthamve/wal-ahead
Fix race condition for 2 appenders having same ts
2017-06-12 11:17:49 +02:00
Goutham Veeramachaneni 73cc5bae51 Colocate defer statements near relevant functions
Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>
2017-06-12 14:37:58 +05:30
Goutham Veeramachaneni b51a05044e
Fix race condition for 2 appenders having same ts
Race:
Suppose we have 100 existing series inside a HeadBlock.
Now we open two appenders in two routines A1, A2 and append 30 new series and
60 new series respectively with some common series.

Both try to commit at the same time and the following happens in the given order:

A2 executes createSeries()
A1 executes createSeries() (with its common series referencing the ids from A2)
A1 persists its newlabels, samples
A2 persists its newlabels, samples

Now when reading it back, we read A1's samples which reference A2's id and
thereby fail.

Ref: prometheus/promtheus#2795

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-08 16:26:25 +05:30
Fabian Reinartz 05e411a8eb Improve heuristic to spread chunks across block 2017-06-08 11:30:32 +02:00
Goutham Veeramachaneni a110a64abd
Add full Snapshot support
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-06 18:15:54 +05:30
Goutham Veeramachaneni a1c8425357
Initial implementation of HeadBlock Snapshots
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-05 13:48:31 +05:30
Goutham Veeramachaneni 29c73f05f2
Make sure that mint and maxt are not modified.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-27 21:59:49 +05:30
Goutham Veeramachaneni 44e9ae38b5
Incorporate PR feedback.
* Expose Stone as it is used in an exported method.
* Move from tombstoneReader to []Stone for the same reason as above.
* Make WAL reading a little cleaner.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-26 21:26:31 +05:30
Goutham Veeramachaneni 6febabeb28
Final delete fixes.
* Make sure no reads happen on the block when delete is in progress.
* Fix bugs in compaction.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-26 16:31:45 +05:30
Goutham Veeramachaneni c211ec4f49
Fix concurrent map access.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-24 16:58:04 +05:30
Goutham Veeramachaneni f29fb62fba
Make TombstoneReader a Getter.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-24 11:24:24 +05:30
Goutham Veeramachaneni 9bf7aa9af1
Misc. fixes incorporating feedback.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-23 18:13:30 +05:30
Goutham Veeramachaneni 31cf939448
Add NumTombstones to BlockMeta.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-23 17:37:04 +05:30
Goutham Veeramachaneni 3eb4119ab1
Make HeadBlock use WAL.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-23 16:15:16 +05:30
Goutham Veeramachaneni 244b73fce1
Rename for clarity and consistency.
Misc. changes for code cleanliness.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 16:42:36 +05:30
Goutham Veeramachaneni 8434019ad9
Merge branch 'master' into deletes-1
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 12:58:38 +05:30
Goutham Veeramachaneni 662d8173fe
Make Appends after Delete visible.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-22 11:28:24 +05:30
Goutham Veeramachaneni d6bd64357b
Fix Delete on HeadBlock
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-19 22:54:29 +05:30
Goutham Veeramachaneni 45d3db4e9e
Use a *mapTombstoneReader instead of map
We need to recalculate the sorted ref list everytime we make a
Tombstones() call. This avoids that.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-19 11:56:37 +05:30
Fabian Reinartz 39df7e2bba Switch blocks to ULID directories, drop sequenc numbers 2017-05-18 16:09:30 +02:00
Fabian Reinartz 285bc07030 Switch append refs to string 2017-05-18 10:56:57 +02:00
Goutham Veeramachaneni 22c1b5b492
Make SeriesSets use tombstones.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-17 14:49:42 +05:30
Goutham Veeramachaneni 34a86af3c6
Move tombstones to their own thing.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-17 08:36:56 +05:30
Goutham Veeramachaneni cea3c88f17
Add Tombstones() method to Block.
Also add Seek() to TombstoneReader

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-16 19:48:28 +05:30
Goutham Veeramachaneni 4f1d857590
Implement Delete on HeadBlock
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-15 23:28:14 +05:30
Goutham Veeramachaneni 5579efbd5b
Initial implentation of Deletes on persistedBlock
Very much a WIP

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-14 14:36:26 +05:30
Fabian Reinartz 8b51b7e2be Make WAL for HeadBlock composeable. 2017-05-13 18:14:18 +02:00
Fabian Reinartz 4862b261d0 Abstract WAL into interface 2017-05-13 17:09:26 +02:00
Fabian Reinartz 535532ca02 Export refdSample
The type was part of a exported method signatures and should therefore
be exported as well.
2017-05-12 17:06:26 +02:00
Fabian Reinartz 5534e6c53c Make HeadBlock impl public, make interface private 2017-05-12 16:34:41 +02:00
Goutham Veeramachaneni 2fa647f50b Fix missing postings in Merge and Intersect (#77)
* Test for a previous implematation of Intersect

Before we were moving the postings list everytime we create a new
chained `intersectPostings`. That was causing some postings to be
skipped. This test fails on the older version.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Advance on Seek only when valid.

Issue:
Before in mergedPostings and others we advance everytime we `Seek`,
which causes issues with `Intersect`.

Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and
b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek.

Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which
would advance a and b beyond 30.

Fix:
Now we advance only when the seeking value is greater than the current
value, as the definition specifies.

Also, posting 0 will not be a valid posting and will be used to signal
finished or un-initialized PostingsList.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Add test for Merge+Intersect edgecase.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Add comments to trivial tests.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-12 09:44:41 +02:00
Fabian Reinartz 291137781b Merge branch 'panic-fix2' of https://github.com/Gouthamve/tsdb into Gouthamve-panic-fix2 2017-05-09 16:22:19 +02:00
Fabian Reinartz 09cd2021de Merge pull request #75 from Gouthamve/head-gen
E2E test for headBlock
2017-05-05 18:56:53 +02:00
Goutham Veeramachaneni 8096d11e4e
Add bounds check to headBlockAppender
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-05 19:52:11 +05:30
Goutham Veeramachaneni adaf4d2099
Handle duplicate & out of order values in same txn
Add docs about not erroring out on exact dupes.
Moved tests to require.*

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-05-03 02:36:40 +05:30
Brian Brazil 72443bddfc Remove dead code. 2017-04-20 13:45:07 +01:00
Brian Brazil bceb5c1b16 When checking for amended points, do it in terms of bits.
NaN != NaN, so the previous code would incorrectly report
it as changed.

There's also plans to take advantage of the NaN payload,
so look at the entire value.
2017-04-12 16:25:32 +01:00
Fabian Reinartz 778103b450 Add liecence file and headers 2017-04-10 20:59:45 +02:00
Fabian Reinartz c73a397da2 Adjust maximum samples per chunk. 2017-04-07 10:58:37 +02:00
Goutham Veeramachaneni a51b2666d7 Fix Panic When Accessing Uncut memorySeries
When calling AddFast, we check the details of the head chunk of the
referred memorySeries. But it could happen that there are no chunks in
the series at all.

Currently, we are deferring chunk creation to when we actually append
samples, but we can be sure that there will be samples if the series is
created. We will be consuming no extra memory by cutting a chunk when we
create the series.

Ref: #28 comment 2

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-04-06 00:22:31 +05:30
Fabian Reinartz 10c7c9acbe Adjust import names to new repository organisation 2017-04-04 11:27:26 +02:00
Fabian Reinartz 87d48bf9de Merge branch 'master' of github.com:fabxc/tsdb 2017-03-27 19:07:27 +02:00
Fabian Reinartz a52980e0a8 Add workaround for deadlocks
This adds a workaround to avoid deadlocks for inconsistent write lock
order across headBlocks.
Things keep working if transactions only append data for the same
timestamp, which is generally the case for Prometheus.

Full behavior should be restored in a subsequent change.
2017-03-27 19:05:34 +02:00
Goutham Veeramachaneni 61f866bb94
Add Sample Back
The compilation and tests are broken as head.go requires sample which
has been moved to another package while moving BufferedSeriesIterator.

Duplication seemed better compared to exposing sample from tsdbutil.
2017-03-26 23:22:58 +05:30
Fabian Reinartz 3be4ef94ce Move BufferedSeriesIterator in own package
This functionality is useful for a lot of clients but not relevant to
the TSDB's core features.
2017-03-24 13:23:32 +01:00
Fabian Reinartz e478d0e3bc Actually close olds blocks in reloadBlocks
This fixes a bug leaking memory because blocks were not actually closed
as the closing call references the initial, empty slice
2017-03-23 18:27:20 +01:00
Fabian Reinartz 789e8224ff Fix wrong comparison in head block resorting 2017-03-21 12:12:33 +01:00
Fabian Reinartz 55ee4b5b3b Merge branch 'master' of github.com:fabxc/tsdb 2017-03-21 10:11:39 +01:00
Fabian Reinartz c18e055d7c Fix races and add comments on remaining ones 2017-03-21 10:11:23 +01:00
Fabian Reinartz e837034360 Merge pull request #14 from Gouthamve/log-update
Update kit/log To New API
2017-03-21 09:56:32 +01:00
Fabian Reinartz 9c93f8f2aa Fix various races
This fixes different race condition encoutnered when running Prometheus.
It reduces the overall performance in the synthetic benchmark a fair bit
but has no indiciations of impacting a real-world setup notably.
2017-03-20 14:45:27 +01:00
Fabian Reinartz 3635569257 Trigger reload correctly on interrupted compaction 2017-03-20 10:41:43 +01:00
Fabian Reinartz 2c999836fb Add Queryable interface to Block
This adds the Queryable interface to the Block interface. Head and
persisted blocks now implement their own Querier() method and thus
isolate customization (e.g. remapPostings) more cleanly.
2017-03-20 10:21:21 +01:00
Fabian Reinartz 11be2cc585 Add composed Block interfaces, remove head generation
This adds more lower-leve interfaces which are used to compose
to different Block interfaces.
The DB only uses interfaces instead of explicit persistedBlock and
headBlock. The headBlock generation property is dropped as the use-case
can be implemented using block sequence numbers.
2017-03-20 09:02:36 +01:00
Goutham Veeramachaneni df7db4ac07
Update kit/log To New API
NewContext has been removed couple of weeks back.
Ref: https://github.com/go-kit/kit/releases/tag/v0.4.0
2017-03-19 21:03:09 +05:30
Fabian Reinartz 65b846ae5b Remove unreturned locks, detect writes on closed heads 2017-03-17 12:12:50 +01:00
Fabian Reinartz e0b33a7a28 Remove leftover signaling channels from headBlock 2017-03-17 10:16:55 +01:00
Fabian Reinartz 34efe4e2c8 Drop position mapper from head block
The position mapper was intended to pre-computed "expensive" ordering
of label sets. It was expensive to update and caused a lot of trouble.
Skipping this optimization entirely actually revelead it was pointless
and even harmful from the e2e perspective.
2017-03-15 14:44:29 +01:00
Fabian Reinartz ad5812d83a Handle WAL corruption by truncating
This adds handling for various corruption scenarios of the WAL.
If corruption is encountered, we truncate the WAL after the last valid
entry transparently and continue appending after the offset.
2017-03-15 11:16:16 +01:00
Fabian Reinartz a8e8903350 Use ChunkMeta references for clarity
This has been a common source of hard to debug issues. Its a premature
and unbenchmarked optimization and semantically, we want ChunkMetas to
be references in all changed cases.
2017-03-14 15:40:16 +01:00
Fabian Reinartz e825a0b40c Make mapper updates asynchronous 2017-03-14 10:57:00 +01:00
Fabian Reinartz cb4dde7659 Fix WAL log recovery bug
This fixes a bug where the last WAL file was closed after consuming it
instead of being left open for further writes.
Reloading of blocks on startup considers loading head blocks now.
2017-03-08 16:53:07 +01:00
Fabian Reinartz 87805fb83f Remove Partitioned* code 2017-03-06 17:34:49 +01:00
Fabian Reinartz 50791a412e Remove leaky Appender abstraction for hashedAppend 2017-03-06 14:27:33 +01:00
Fabian Reinartz 55a9b5428a Add separate head mutex
Introduce a seperate mutex for the head blocks to avoid a race where
a post-compaction reload may run between switching the DB's base mutex
to create a new head block in an appender.
2017-03-04 17:41:01 +01:00
Fabian Reinartz 92120448c2 Properly cleanup compacted dirs, fixes, docs 2017-03-02 14:32:09 +01:00
Fabian Reinartz 2c3e778d90 Compactor interface 2017-03-02 09:17:30 +01:00
Fabian Reinartz 306831f151 Add per-block state ULID 2017-02-27 10:46:15 +01:00
Fabian Reinartz 78780cd2ba Segment chunk file
This adds write path support for segmented chunk data files.
Files of 512MB are pre-allocated and written to. If the file size
is exceeded, the next file is started. On completion, files
are truncated to their final size.
2017-02-23 10:50:22 +01:00
Fabian Reinartz a3d042b54e Support multiple chunk files in read path 2017-02-18 17:33:20 +01:00
Fabian Reinartz 9c7a88223e Add full encode/decode WAL cycle test 2017-02-14 21:55:50 -08:00
Fabian Reinartz f1435f2e2c Track appended samples properly in metric 2017-02-08 16:13:16 -08:00
Fabian Reinartz 012cf4ef25 Count writer references on head blocks 2017-02-04 11:53:52 +01:00
Fabian Reinartz 5a1c8eaa0e Fix missing appends after reference lookups 2017-02-02 11:10:17 +01:00
Fabian Reinartz ac5229e1b4 Correctly write empty blocks and extend appenders by new blocks 2017-02-02 07:58:54 +01:00
Fabian Reinartz 30efe4a58c Support writing to multiple head blocks
This is an initial (and hacky) first pass on allowing
appending to multiple blocks simultaniously to avoid
dropping samples right after cutting a new head block.
It's also required for cases like the PGW, where a scrape may
contain varying timestamps.
2017-02-01 15:57:28 +01:00
Fabian Reinartz c20cc44b06 Add docs, write sequence number to meta.json 2017-01-29 08:11:47 +01:00
Fabian Reinartz 472c618c39 Drop out-of-bound samples 2017-01-19 15:03:57 +01:00
Fabian Reinartz d4779b374c Properly track and write meta file 2017-01-19 14:01:38 +01:00
Fabian Reinartz 9ddbd64d00 Move stats into meta.json file, cleanup, docs 2017-01-19 11:22:47 +01:00
Fabian Reinartz 5ceca3c810 Write to WAL before appending to memory storage 2017-01-17 16:33:58 +01:00
Fabian Reinartz 343dd9d94c Fix wrong byte size in WAL base ref 2017-01-17 08:40:31 +01:00
Fabian Reinartz 5fb01d41aa Use new Prometheus text format parser 2017-01-16 21:29:53 +01:00
Fabian Reinartz dd0b69fe1b Export ErrNotFound 2017-01-16 14:18:32 +01:00
Fabian Reinartz 1c80c33e72 Fix bug of unsorted postings lists being created
The former approach created unordered postings list by either
map iteration of new series being unsorted (fixable) or concurrent
writers creating new series interleaved.

We switch back to generating ephemeral references for a single batch.
Newly created series have to be re-set upon the next insert.
2017-01-13 16:22:20 +01:00
Fabian Reinartz c7f5590a71 Ensure order of postings when adding new series 2017-01-13 15:25:11 +01:00
Fabian Reinartz d970f0256a Add Rollback() and docs to Appender interface 2017-01-12 20:17:49 +01:00
Fabian Reinartz 22db9c3413 Remove old appendBatch methods 2017-01-12 20:04:49 +01:00
Fabian Reinartz fde69dab49 Use buffer pool for head appenders 2017-01-12 20:03:44 +01:00
Fabian Reinartz a317f252b9 Expose series references to clients
This exposes a reference number of a series represented by a label set
to clients.
Subsequent samples can be directly added via the reference rather than
repeatedly passing in the full labels. This drasitcally speeds up the
append process.

The appender chain uses different sections of the reference number for
assignment to child appenders and invalidating reference numbers as
necessary.

Clients can either pass out reference numbers themselves or have their
own optimized lookup, i.e. by directly associating unparsed metric
descriptors strings with reference numbers.
2017-01-12 20:00:54 +01:00
Fabian Reinartz 5e028710d5 Add fast past to validation after lock switch 2017-01-12 15:51:08 +01:00
Fabian Reinartz 1b39887baa Revalidate series existance after lock switch 2017-01-11 14:05:58 +01:00
Fabian Reinartz ca5791efbc Simplify creation of new series 2017-01-11 13:58:26 +01:00
Fabian Reinartz 0ca755b4ae Replace single head chunk per series with memSeries
This adds a memory series holding several chunk to replace
the single head chunk per series so far.
This is necessary for uniform maximum chunk sizes in cases
where some series have higher frequency samples than others.
2017-01-11 13:02:38 +01:00
Fabian Reinartz c32a94d409 Unexport HeadBlock, export Block interface 2017-01-10 15:41:57 +01:00
Fabian Reinartz d86e8a63c7 Report correct number of appended samples 2017-01-10 11:17:37 +01:00
Fabian Reinartz 29883a18fc Add own Appender() method for DB 2017-01-09 22:54:08 +01:00
Fabian Reinartz 4c4e0c614e Simplify position mapper updating 2017-01-09 19:24:05 +01:00
Fabian Reinartz 0dffd52238 Use page writer in compaction 2017-01-09 18:47:43 +01:00
Fabian Reinartz 89d8467f5c Add missing lock 2017-01-09 18:07:45 +01:00
Fabian Reinartz 8c31c6e934 Make concurrent head chunk reads safe, fix misc races
This adds a 4 sample buffer to every head chunk. The XOR
compression scheme may edit bytes in place. The minimum size
of a sample is 2 bits. So keeping the last 4 samples in an in-memory
buffer makes it safe to query the preceeding ones while samples
are added
2017-01-09 16:51:39 +01:00
Fabian Reinartz 1943f8d1bb Fix head block stats races 2017-01-07 18:02:17 +01:00
Fabian Reinartz 6aa922c5a6 Fix races 2017-01-07 16:20:32 +01:00
Fabian Reinartz 300f4e2abf Use separate lock for series creation
This uses the head block's own lock to only lock if new series were
encountered.
In the general append case we just need to hold a
2017-01-06 18:10:50 +01:00
Fabian Reinartz 63e12807da Don't update head postings mapper on every append 2017-01-06 16:43:18 +01:00
Fabian Reinartz 71efd2e08d Periodically fsync WAL, make head cut async 2017-01-06 15:18:06 +01:00
Fabian Reinartz 96c2bd249f Handle compaction trigger and reinitializing in DB 2017-01-06 13:03:23 +01:00
Fabian Reinartz 937cdb579c Switch to sequential block names
This changes block directory names from the int64 timestamp
to sequential numbering.
2017-01-06 10:45:03 +01:00
Fabian Reinartz 9790aa98ac Add postings wrapper that emits head postings in label set order
This adds a position mapper that takes series from a head block
in the order they were appended and creates a mapping representing
them in order of their label sets.

Write-repair of the postings list would cause very expensive writing.
Hence, we keep them as they are and only apply the postition mapping
at the very end, after a postings list has been sufficienctly reduced
through intersections etc.
2017-01-05 16:05:42 +01:00
Fabian Reinartz 5aa7f7cce8 Compact head block into persisted block 2017-01-04 21:11:15 +01:00
Fabian Reinartz 3f72d5d027 Fix last timestamp initialization
This initializes the chunkDesc's last timestamp to the minimum
value so initial samples with a timestamp of 0 (e.g. in tests)
are not accidentally dropped.
2017-01-04 14:06:40 +01:00
Fabian Reinartz ac49f8c15e Consolidate persistence and compaction 2017-01-03 16:49:37 +01:00
Fabian Reinartz e7f04d14d5 Lock mmapped files 2017-01-03 10:09:20 +01:00
Fabian Reinartz 91b65b55e7 Run persistence in separate goroutine 2017-01-02 22:24:35 +01:00
Fabian Reinartz a648ef5252 Convert persister into function 2017-01-02 16:58:47 +01:00
Fabian Reinartz dbd2b21d2e Make persistence atomic 2017-01-02 14:41:13 +01:00
Fabian Reinartz beb842a856 Change block interface
This changes the block interface to directly expose index
and series readers rather than the direct querier
2017-01-02 11:12:28 +01:00
Fabian Reinartz 5d7ec06e04 Use ChunkMeta in SeriesWriter 2017-01-01 20:01:17 +01:00
Fabian Reinartz c00d17e691 Modify IndexReader API to accomodate compaction
This changes the IndexReader API to expose plain labels
and chunk meta information instead of a Series interface.
Dropping of irrelevant chunks is moved into the querier.

A LabelIndices method is added to query for existing label
value indices.
2016-12-31 16:24:04 +01:00
Fabian Reinartz 1e1a37b15b Remove double-reference in chunk hashmap 2016-12-31 10:19:02 +01:00
Fabian Reinartz 675f0886f0 Append to chunks cannot error 2016-12-31 10:10:27 +01:00
Fabian Reinartz 7280533c42 Add basic shard metrics 2016-12-31 09:48:49 +01:00
Fabian Reinartz 201d7687b6 Fix uniqueness of new series 2016-12-26 16:55:32 +01:00
Fabian Reinartz 787199a88e Fix erroneous value assignments 2016-12-22 20:57:00 +01:00
Fabian Reinartz 8aba95048a Rename to OpenHeadBlock 2016-12-22 20:00:24 +01:00
Fabian Reinartz 1dde3b6d31 Add WAL decoder+loading and benchmarks 2016-12-22 15:18:33 +01:00
Fabian Reinartz 0b8c77361e Add initial WAL writing 2016-12-22 12:05:24 +01:00
Fabian Reinartz 2a825f6c28 Consolidate mem index into HeadBlock 2016-12-22 01:12:28 +01:00
Fabian Reinartz ede733ab6c Extract labels package 2016-12-21 09:39:01 +01:00
Fabian Reinartz 1b23d62e3f Properly close files before reopening 2016-12-19 22:37:03 +01:00
Fabian Reinartz 282d9ae6e2 Implement label value queries in all layers. 2016-12-19 12:26:25 +01:00
Fabian Reinartz bad93d8d57 Extract head serialization into Head method 2016-12-18 14:43:27 +01:00