The Prometheus monitoring system and time series database.
Find a file
Brian Brazil 259847a6b1
Be smarter in how we look at matchers. (#572)
* Add unittests for PostingsForMatcher.

* Selector methods are all stateless, don't need a reference.

* Be smarter in how we look at matchers.

Look at all matchers to see if a label can be empty.

Optimise Not handling, so i!="2" is a simple lookup
rather than an inverse postings list.

All all the Withouts together, rather than
having to subtract each from all postings.

Change the pre-expand the postings logic to always do it before doing a
Without only. Don't do that if it's already a list.

The initial goal here was that the oft-seen pattern
i=~"something.+",i!="foo",i!="bar" becomes more efficient.

benchmark                                                            old ns/op     new ns/op     delta
BenchmarkHeadPostingForMatchers/n="1"-4                              5888          6160          +4.62%
BenchmarkHeadPostingForMatchers/n="1",j="foo"-4                      7190          6640          -7.65%
BenchmarkHeadPostingForMatchers/j="foo",n="1"-4                      6038          5923          -1.90%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"-4                     6030884       4850525       -19.57%
BenchmarkHeadPostingForMatchers/i=~".*"-4                            887377940     230329137     -74.04%
BenchmarkHeadPostingForMatchers/i=~".+"-4                            490316101     319931758     -34.75%
BenchmarkHeadPostingForMatchers/i=~""-4                              594961991     130279313     -78.10%
BenchmarkHeadPostingForMatchers/i!=""-4                              537542388     318751015     -40.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"-4              10460243      8565195       -18.12%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"-4       44964267      8561546       -80.96%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"-4                42244885      29137737      -31.03%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"-4              35285834      32774584      -7.12%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"-4             8951047       8379024       -6.39%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"-4       63813335      30672688      -51.93%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"-4     45381112      44924397      -1.01%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2019-04-09 11:59:45 +01:00
chunkenc Remove obsolete FromData function (#540) 2019-03-06 11:12:28 +02:00
chunks remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
cmd/tsdb Fixed default benchmark test path (#571) 2019-03-27 16:33:59 +02:00
docs/format docs: add multi chunks layout in chunks readme file (#567) 2019-03-25 12:33:38 +02:00
encoding Fix exported function comments (#566) 2019-03-25 10:17:28 +02:00
errors Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
fileutil remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
goversion remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
index Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
labels Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
test index: simplify checksum validation 2017-11-09 15:58:36 +00:00
testdata Add repair test 2018-02-09 13:43:42 +01:00
testutil Test to corrupt segments mid-WAL, repair and check we can read the correct number of records. (#528) 2019-02-18 19:05:07 +00:00
tsdbutil Add license headers to missing files (#447) 2018-11-06 20:19:42 +02:00
wal remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
.gitignore Add default benchmark output dir to .gitignore 2018-01-14 11:07:52 +00:00
.travis.yml remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
block.go use fsync to force persisting the meta file. (#573) 2019-04-01 16:23:45 +03:00
block_test.go Verify chunk format version (#544) 2019-03-24 22:33:08 +02:00
CHANGELOG.md Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
checkpoint.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
checkpoint_test.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
compact.go selectOverlappingDirs selects wrong dirs where there are 2 disjoint sets of overlapping dirs 2019-04-08 15:27:06 +03:00
compact_test.go selectOverlappingDirs selects wrong dirs where there are 2 disjoint sets of overlapping dirs 2019-04-08 15:27:06 +03:00
db.go remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
db_test.go selectOverlappingDirs selects wrong dirs where there are 2 disjoint sets of overlapping dirs 2019-04-08 15:27:06 +03:00
go.mod Use Go modules (#454) 2018-11-28 11:39:56 +01:00
go.sum Use Go modules (#454) 2018-11-28 11:39:56 +01:00
head.go Add Head.compactable method (#542) 2019-04-01 11:19:06 +03:00
head_bench_test.go Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
head_test.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
LICENSE Add license file 2017-04-10 21:02:42 +02:00
MAINTAINERS.md add maintainers file (#404) 2018-10-15 14:32:00 +03:00
Makefile Update Makefile.common (#548) 2019-03-13 10:40:17 +02:00
Makefile.common Update Makefile.common (#548) 2019-03-13 10:40:17 +02:00
mocks_test.go Vertical query merging and compaction (#370) 2019-02-14 14:29:41 +01:00
querier.go Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
querier_test.go Be smarter in how we look at matchers. (#572) 2019-04-09 11:59:45 +01:00
README.md README: move the build badge to the badge list (#555) 2019-03-18 14:44:58 +02:00
record.go Merge encoding_helpers.go to tsdbutil (#526) 2019-02-22 19:11:11 +02:00
record_test.go Add test for be64() in encoding/encoding.go (#521) (#533) 2019-02-23 11:51:47 +02:00
repair.go remove Fsync workaround for macos. (#574) 2019-04-03 11:16:54 +03:00
repair_test.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
tombstones.go force persisting the tombstone file (#578) 2019-04-08 15:06:40 +03:00
tombstones_test.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00
wal.go Merge encoding_helpers.go to tsdbutil (#526) 2019-02-22 19:11:11 +02:00
wal_test.go Test createBlock and check all os.RemoveAll in the tests for errors. (#549) 2019-03-19 15:31:57 +02:00

TSDB

Build Status GoDoc Go Report Card

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.