mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-18 11:34:05 -08:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
prometheus/tsdb/docs/format/chunks.md
Dieter Plaetinck 4d27816ea5
Sparsehistogram: improve dod encoding, testing, encode chunk metadata (#9015) 
* factor out different varbit schemes and include Beorn's "optimum" for buckets

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* use more compact dod encoding scheme for SHS chunk columns

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* remove FB VB and xor dod encoding because we won't use it

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* HistoChunk metadata encoding

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* add SparseHistogram.Copy()

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* histogram test: test appending a few histograms

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* add license headers

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
2021-06-30 16:15:43 +05:30

3.1 KiB
Raw Blame History

Chunks Disk Format

The following describes the format of a chunks file, which is created in the chunks/ directory of a block. The maximum size per segment file is 512MiB.

Chunks in the files are referenced from the index by uint64 composed of in-file offset (lower 4 bytes) and segment sequence number (upper 4 bytes).

┌──────────────────────────────┐
│  magic(0x85BD40DD) <4 byte>  │
├──────────────────────────────┤
│    version(1) <1 byte>       │
├──────────────────────────────┤
│    padding(0) <3 byte>       │
├──────────────────────────────┤
│ ┌──────────────────────────┐ │
│ │         Chunk 1          │ │
│ ├──────────────────────────┤ │
│ │          ...             │ │
│ ├──────────────────────────┤ │
│ │         Chunk N          │ │
│ └──────────────────────────┘ │
└──────────────────────────────┘

Chunk

┌───────────────┬───────────────────┬──────────────┬────────────────┐
│ len <uvarint> │ encoding <1 byte> │ data <bytes> │ CRC32 <4 byte> │
└───────────────┴───────────────────┴──────────────┴────────────────┘

Histogram chunk

┌──────────────┬─────────────────┬──────────────────────────┬──────────────────────────┬──────────────┐
│ len <uint16> │ schema <varint> │ pos-spans <span-section> │ neg-spans <span-section> │ data <bytes> │
└──────────────┴─────────────────┴──────────────────────────┴──────────────────────────┴──────────────┘

span-section:

┌──────────────┬──────────────────┬──────────────────┬────────────┐
│ len <varint> │ length1 <varint> │ offset1 <varint> │ length2... │
└──────────────┴──────────────────┴──────────────────┴────────────┘