Commit graph

35 commits

Author SHA1 Message Date
Mario Trangoni c0e888e82b fix megacheck issues: os.SEEK_SET is deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd. 2018-03-21 22:39:15 +01:00
Mario Trangoni 09142e4dd1 fix unconvert issues: unnecessary conversion 2018-03-21 22:39:14 +01:00
Fabian Reinartz 1e55b7987f Improve comments, handle allPostingsKey properly 2017-12-22 09:43:34 +01:00
Fabian Reinartz 67f0ca8f0e Move index and chunk encoders to own packages 2017-12-21 11:27:54 +01:00
Callum Styan ff0b0ac4b6 use test utils in chunk_test 2017-12-06 19:05:58 -08:00
Daniel Swarbrick 0585ab3fcd Use Go 1.9 math/bits implementation for counting leading / trailing zeros 2017-11-08 23:18:33 +01:00
Fabian Reinartz ea817e169b Return nop iterator for invalid chunk references 2017-10-20 09:43:52 +02:00
Fabian Reinartz 905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz 66ff7b12e9 Pool Chunk objects during compaction 2017-08-08 17:35:34 +02:00
Fabian Reinartz 96d7f540d4 Persist series without allocating the full set
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.

As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
Goutham Veeramachaneni 401bdfd4a6 Make Chunks return the number of samples inside
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:45:12 +02:00
Julius Volz 0e2f8917ed Also add original license info to bstream.go
Plus, extend the messages a bit to mention the origin repo.
2017-05-03 01:32:09 +02:00
Fabian Reinartz 87676f4a26 Add dgryski/go-tsz LICENSE to header 2017-05-02 11:26:01 +02:00
Julius Volz 8d1fb4fa01 Minor comment fixes and additions. 2017-04-28 15:41:42 +02:00
Fabian Reinartz 778103b450 Add liecence file and headers 2017-04-10 20:59:45 +02:00
Goutham Veeramachaneni 761e4768f3
Lint and Vet Fixes 2017-03-19 21:35:01 +05:30
Fabian Reinartz 91b65b55e7 Run persistence in separate goroutine 2017-01-02 22:24:35 +01:00
Fabian Reinartz 62b8ded0a5 Rename iterator value getters to At() 2017-01-02 13:33:06 +01:00
Fabian Reinartz 5d75a3dc7b Add basic compaction
This adds a basic compactor that will merge two persisted blocks into
one. It simply fully rewrites the index and concatenates the chunk
lists.

It just writes into the current working dir and doesn't properly handle
which blocks to compact for now.
2017-01-02 10:34:55 +01:00
Fabian Reinartz 675f0886f0 Append to chunks cannot error 2016-12-31 10:10:27 +01:00
Fabian Reinartz ce7f4106c2 Reda correct label number, fix buffered iterator panic 2016-12-20 14:21:50 +01:00
Fabian Reinartz 70a0224f19 Change chunk sample number to BigEndian 2016-12-12 08:11:53 +01:00
Fabian Reinartz 40a451694f Refactor persistence into interfaces 2016-12-09 20:45:46 +01:00
Fabian Reinartz b845f8d3a1 Reduce test data allocations 2016-12-08 11:59:54 +01:00
Fabian Reinartz c5945177fb chunks: helper for bit range 2016-12-07 15:37:37 +01:00
Fabian Reinartz 9b459458d0 Docs and interface definitions 2016-12-05 21:26:19 +01:00
Fabian Reinartz 0b6d621471 chunks: bring back lead/trail reuse, truncate incomplete sample 2016-11-30 23:01:01 +01:00
Fabian Reinartz 8c48dc2ca5 chunks: varint encoding in first full 64bit numbers
This saves about 7 bytes per chunk
2016-11-30 22:14:23 +01:00
Fabian Reinartz 5e76fd3126 chunks: drop shift, remove branch
This removes the need for the shift value and just calculates
everything on the fly through count.
Removes a branch condition.
2016-11-30 21:50:39 +01:00
Fabian Reinartz c9ee572709 chunks: don't mutate in bstream iterator
This replaces mutation of underlying bytes in the iterated slice
with a shift counter, which is used when reading the head byte.
This is avoids having to copy the entire slice for every new iterator.
2016-11-30 19:39:22 +01:00
Fabian Reinartz f392c01c12 chunks: add randomized test 2016-11-29 22:43:24 +01:00
Fabian Reinartz fa181a34c1 chunks: cleanup anything but xor encoding
xor encoding is fast enough for our purposes and provides
very good compression.
We remove all other ones that partially don't support floats
for the sake of simplicity.
2016-11-29 22:02:58 +01:00
Fabian Reinartz e67cf768dc chunks: remove intermeidate copy from xor chunk 2016-11-20 16:24:46 +01:00
Fabian Reinartz 7874d28f32 chunks: implement xor encoding 2016-11-20 14:33:00 +01:00
Fabian Reinartz ae7b18ae34 Add chunk package 2016-11-15 10:33:34 +01:00