Fabian Reinartz
eeb03e97e2
Add headers and format flag for WAL segments
2017-02-14 00:24:53 -08:00
Fabian Reinartz
79944a5912
Break out WAL into segment files
2017-02-13 23:53:19 -08:00
Fabian Reinartz
987a90d149
Don't delete files if retention duration is 0
2017-02-11 11:49:17 -08:00
Fabian Reinartz
9c76624df2
Add initial retention cutoff
2017-02-09 17:54:26 -08:00
Fabian Reinartz
f1435f2e2c
Track appended samples properly in metric
2017-02-08 16:13:16 -08:00
Fabian Reinartz
012cf4ef25
Count writer references on head blocks
2017-02-04 11:53:52 +01:00
Fabian Reinartz
5a1c8eaa0e
Fix missing appends after reference lookups
2017-02-02 11:10:17 +01:00
Fabian Reinartz
f556036037
Close blocks before overwriting with compacted one
...
This prevents the meta.json file of a compacted block
to be overwritten with data of the old block.
2017-02-02 09:32:06 +01:00
Fabian Reinartz
ac5229e1b4
Correctly write empty blocks and extend appenders by new blocks
2017-02-02 07:58:54 +01:00
Fabian Reinartz
42fa342229
Improve multi-head handling
...
This improves handling of multiple head blocks. Configuration is
simplified to specify the number of concurrently appendable blocks.
2017-02-01 21:35:21 +01:00
Fabian Reinartz
30efe4a58c
Support writing to multiple head blocks
...
This is an initial (and hacky) first pass on allowing
appending to multiple blocks simultaniously to avoid
dropping samples right after cutting a new head block.
It's also required for cases like the PGW, where a scrape may
contain varying timestamps.
2017-02-01 15:57:28 +01:00
Fabian Reinartz
a61a31a5d7
compaction: add fast-path for compacting mem blocks
2017-01-30 09:49:36 +01:00
Fabian Reinartz
c20cc44b06
Add docs, write sequence number to meta.json
2017-01-29 08:11:47 +01:00
Fabian Reinartz
30d8866c13
Limit compaction range, make cut and compact concurrent
2017-01-20 07:58:19 +01:00
Fabian Reinartz
67d185ceb9
Compact based on compaction generation
2017-01-19 19:45:52 +01:00
Fabian Reinartz
472c618c39
Drop out-of-bound samples
2017-01-19 15:03:57 +01:00
Fabian Reinartz
d4779b374c
Properly track and write meta file
2017-01-19 14:01:38 +01:00
Fabian Reinartz
9ddbd64d00
Move stats into meta.json file, cleanup, docs
2017-01-19 11:22:47 +01:00
Fabian Reinartz
2f02f86b62
Fix WAL tests
2017-01-19 08:48:11 +01:00
Fabian Reinartz
e006bc6dc6
Improve error messages, create regular dir for block
2017-01-19 08:40:15 +01:00
Fabian Reinartz
d2322f6095
Improve compaction processing
2017-01-18 06:18:32 +01:00
Fabian Reinartz
5ceca3c810
Write to WAL before appending to memory storage
2017-01-17 16:33:58 +01:00
Fabian Reinartz
343dd9d94c
Fix wrong byte size in WAL base ref
2017-01-17 08:40:31 +01:00
Fabian Reinartz
5fb01d41aa
Use new Prometheus text format parser
2017-01-16 21:29:53 +01:00
Fabian Reinartz
dd0b69fe1b
Export ErrNotFound
2017-01-16 14:18:32 +01:00
Fabian Reinartz
9cf49f68e9
wal: use larger buffer
2017-01-16 14:18:25 +01:00
Fabian Reinartz
1c80c33e72
Fix bug of unsorted postings lists being created
...
The former approach created unordered postings list by either
map iteration of new series being unsorted (fixable) or concurrent
writers creating new series interleaved.
We switch back to generating ephemeral references for a single batch.
Newly created series have to be re-set upon the next insert.
2017-01-13 16:22:20 +01:00
Fabian Reinartz
c7f5590a71
Ensure order of postings when adding new series
2017-01-13 15:25:11 +01:00
Fabian Reinartz
d970f0256a
Add Rollback() and docs to Appender interface
2017-01-12 20:17:49 +01:00
Fabian Reinartz
22db9c3413
Remove old appendBatch methods
2017-01-12 20:04:49 +01:00
Fabian Reinartz
fde69dab49
Use buffer pool for head appenders
2017-01-12 20:03:44 +01:00
Fabian Reinartz
a317f252b9
Expose series references to clients
...
This exposes a reference number of a series represented by a label set
to clients.
Subsequent samples can be directly added via the reference rather than
repeatedly passing in the full labels. This drasitcally speeds up the
append process.
The appender chain uses different sections of the reference number for
assignment to child appenders and invalidating reference numbers as
necessary.
Clients can either pass out reference numbers themselves or have their
own optimized lookup, i.e. by directly associating unparsed metric
descriptors strings with reference numbers.
2017-01-12 20:00:54 +01:00
Fabian Reinartz
5e028710d5
Add fast past to validation after lock switch
2017-01-12 15:51:08 +01:00
Fabian Reinartz
1b39887baa
Revalidate series existance after lock switch
2017-01-11 14:05:58 +01:00
Fabian Reinartz
ca5791efbc
Simplify creation of new series
2017-01-11 13:58:26 +01:00
Fabian Reinartz
0ca755b4ae
Replace single head chunk per series with memSeries
...
This adds a memory series holding several chunk to replace
the single head chunk per series so far.
This is necessary for uniform maximum chunk sizes in cases
where some series have higher frequency samples than others.
2017-01-11 13:02:38 +01:00
Fabian Reinartz
80affd98a8
Add barrier to benchmark writer
...
This adds a barrier to avoid issues with unfair goroutine scheduling
that causes some fake scrapers to run away from the other ones.
2017-01-11 13:01:30 +01:00
Fabian Reinartz
c32a94d409
Unexport HeadBlock, export Block interface
2017-01-10 15:41:57 +01:00
Fabian Reinartz
d86e8a63c7
Report correct number of appended samples
2017-01-10 11:17:37 +01:00
Fabian Reinartz
29883a18fc
Add own Appender() method for DB
2017-01-09 22:54:08 +01:00
Fabian Reinartz
4c4e0c614e
Simplify position mapper updating
2017-01-09 19:24:05 +01:00
Fabian Reinartz
142c89b8b0
Fix/update metrics
2017-01-09 19:14:21 +01:00
Fabian Reinartz
0dffd52238
Use page writer in compaction
2017-01-09 18:47:43 +01:00
Fabian Reinartz
89d8467f5c
Add missing lock
2017-01-09 18:07:45 +01:00
Fabian Reinartz
8c31c6e934
Make concurrent head chunk reads safe, fix misc races
...
This adds a 4 sample buffer to every head chunk. The XOR
compression scheme may edit bytes in place. The minimum size
of a sample is 2 bits. So keeping the last 4 samples in an in-memory
buffer makes it safe to query the preceeding ones while samples
are added
2017-01-09 16:51:39 +01:00
Fabian Reinartz
1943f8d1bb
Fix head block stats races
2017-01-07 18:02:17 +01:00
Fabian Reinartz
6aa922c5a6
Fix races
2017-01-07 16:20:32 +01:00
Fabian Reinartz
54f5027406
Put WAL lock down into encoder
2017-01-06 18:36:42 +01:00
Fabian Reinartz
300f4e2abf
Use separate lock for series creation
...
This uses the head block's own lock to only lock if new series were
encountered.
In the general append case we just need to hold a
2017-01-06 18:10:50 +01:00
Fabian Reinartz
63e12807da
Don't update head postings mapper on every append
2017-01-06 16:43:18 +01:00