Mario Trangoni
c0e888e82b
fix megacheck issues: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd.
2018-03-21 22:39:15 +01:00
Mario Trangoni
09142e4dd1
fix unconvert issues: unnecessary conversion
2018-03-21 22:39:14 +01:00
Fabian Reinartz
1e55b7987f
Improve comments, handle allPostingsKey properly
2017-12-22 09:43:34 +01:00
Fabian Reinartz
67f0ca8f0e
Move index and chunk encoders to own packages
2017-12-21 11:27:54 +01:00
Callum Styan
ff0b0ac4b6
use test utils in chunk_test
2017-12-06 19:05:58 -08:00
Daniel Swarbrick
0585ab3fcd
Use Go 1.9 math/bits implementation for counting leading / trailing zeros
2017-11-08 23:18:33 +01:00
Fabian Reinartz
ea817e169b
Return nop iterator for invalid chunk references
2017-10-20 09:43:52 +02:00
Fabian Reinartz
905af27cf9
Refactor compactor
2017-08-09 11:10:29 +02:00
Fabian Reinartz
66ff7b12e9
Pool Chunk objects during compaction
2017-08-08 17:35:34 +02:00
Fabian Reinartz
96d7f540d4
Persist series without allocating the full set
...
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.
As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Goutham Veeramachaneni
401bdfd4a6
Make Chunks return the number of samples inside
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:45:12 +02:00
Julius Volz
0e2f8917ed
Also add original license info to bstream.go
...
Plus, extend the messages a bit to mention the origin repo.
2017-05-03 01:32:09 +02:00
Fabian Reinartz
87676f4a26
Add dgryski/go-tsz LICENSE to header
2017-05-02 11:26:01 +02:00
Julius Volz
8d1fb4fa01
Minor comment fixes and additions.
2017-04-28 15:41:42 +02:00
Fabian Reinartz
778103b450
Add liecence file and headers
2017-04-10 20:59:45 +02:00
Goutham Veeramachaneni
761e4768f3
Lint and Vet Fixes
2017-03-19 21:35:01 +05:30
Fabian Reinartz
91b65b55e7
Run persistence in separate goroutine
2017-01-02 22:24:35 +01:00
Fabian Reinartz
62b8ded0a5
Rename iterator value getters to At()
2017-01-02 13:33:06 +01:00
Fabian Reinartz
5d75a3dc7b
Add basic compaction
...
This adds a basic compactor that will merge two persisted blocks into
one. It simply fully rewrites the index and concatenates the chunk
lists.
It just writes into the current working dir and doesn't properly handle
which blocks to compact for now.
2017-01-02 10:34:55 +01:00
Fabian Reinartz
675f0886f0
Append to chunks cannot error
2016-12-31 10:10:27 +01:00
Fabian Reinartz
ce7f4106c2
Reda correct label number, fix buffered iterator panic
2016-12-20 14:21:50 +01:00
Fabian Reinartz
70a0224f19
Change chunk sample number to BigEndian
2016-12-12 08:11:53 +01:00
Fabian Reinartz
40a451694f
Refactor persistence into interfaces
2016-12-09 20:45:46 +01:00
Fabian Reinartz
b845f8d3a1
Reduce test data allocations
2016-12-08 11:59:54 +01:00
Fabian Reinartz
c5945177fb
chunks: helper for bit range
2016-12-07 15:37:37 +01:00
Fabian Reinartz
9b459458d0
Docs and interface definitions
2016-12-05 21:26:19 +01:00
Fabian Reinartz
0b6d621471
chunks: bring back lead/trail reuse, truncate incomplete sample
2016-11-30 23:01:01 +01:00
Fabian Reinartz
8c48dc2ca5
chunks: varint encoding in first full 64bit numbers
...
This saves about 7 bytes per chunk
2016-11-30 22:14:23 +01:00
Fabian Reinartz
5e76fd3126
chunks: drop shift, remove branch
...
This removes the need for the shift value and just calculates
everything on the fly through count.
Removes a branch condition.
2016-11-30 21:50:39 +01:00
Fabian Reinartz
c9ee572709
chunks: don't mutate in bstream iterator
...
This replaces mutation of underlying bytes in the iterated slice
with a shift counter, which is used when reading the head byte.
This is avoids having to copy the entire slice for every new iterator.
2016-11-30 19:39:22 +01:00
Fabian Reinartz
f392c01c12
chunks: add randomized test
2016-11-29 22:43:24 +01:00
Fabian Reinartz
fa181a34c1
chunks: cleanup anything but xor encoding
...
xor encoding is fast enough for our purposes and provides
very good compression.
We remove all other ones that partially don't support floats
for the sake of simplicity.
2016-11-29 22:02:58 +01:00
Fabian Reinartz
e67cf768dc
chunks: remove intermeidate copy from xor chunk
2016-11-20 16:24:46 +01:00
Fabian Reinartz
7874d28f32
chunks: implement xor encoding
2016-11-20 14:33:00 +01:00
Fabian Reinartz
ae7b18ae34
Add chunk package
2016-11-15 10:33:34 +01:00