While implementing a different feature, I found that Labels.Get() was
performing a linear search. I wondered whether it would perform any
better with a binary search approach, and wrote a benchmark: the answer
is that it's probably doesn't worth it, so I just decided to leave the
benchmark and the results for the next reader.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use global string map for MatchType.String()
We were unnecessarily creating a new map for each call of String().
This is a 10x improvement in MatchType.String() performance in time,
from 53ns to 4ns on my i7 laptop, and I guess that this method is being
called quite often so why throw out the resources.
I was surprised that benchmark says that there are no allocations made
in the old version.
I also tries using `//go:generate stringer` and the result is even
better, at about 2.8ns, but having to keep the generated code updated
isn't worth the change (at least it's bigger than a small change I was
intended to do)
Benchmark comparison:
name \ time/op old global_map stringer
MatchType_String 53.6ns ± 1% 4.1ns ± 1% 2.8ns ± 1%
name \ alloc/op old global_map stringer
MatchType_String 0.00B 0.00B 0.00B
name \ allocs/op old global_map stringer
MatchType_String 0.00 0.00 0.00
Old benchmark:
goos: darwin
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMatchType_String 21766578 54.36 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21742339 53.28 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21985470 53.37 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21676282 53.50 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 22075573 53.33 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/prometheus/prometheus/pkg/labels 6.252s
New with global map:
goos: darwin
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMatchType_String 283412692 4.129 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 294859941 4.091 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 295750158 4.113 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 282827982 4.072 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 292942393 4.047 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/prometheus/prometheus/pkg/labels 8.238s
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use array instead of map
Since MatchType is an iota type, we can safely use an array here.
This solution is even better:
name \ time/op old global_map stringer array
MatchType_String 53.6ns ± 1% 4.1ns ± 1% 2.8ns ± 1% 1.0ns ± 1%
name \ alloc/op old global_map stringer array
MatchType_String 0.00B 0.00B 0.00B 0.00B
name \ allocs/op old global_map stringer array
MatchType_String 0.00 0.00 0.00 0.00
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark all MatchType values
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use constants for limits
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Write exemplars to the WAL and send them over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Update example for exemplars, print data in a more obvious format.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add metrics for remote write of exemplars.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Fix incorrect slices passed to send in remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* We need to unregister the new metrics.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Order of exemplar append vs write exemplar to WAL needs to change.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Condense sample/exemplar delivery tests to parameterized sub-tests
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename test methods for clarity now that they also handle exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename counter variable. Fix instances where metrics were not updated correctly
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Add exemplars to LoadWAL benchmark
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* last exemplars timestamp metric needs to convert value to seconds with
ms precision
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Process exemplar records in a separate go routine when loading the WAL.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Regenerate types proto with comments, update protoc version again.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Put remote write of exemplars behind a feature flag.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address some of Ganesh's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Move exemplar remote write feature flag to a config file field.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address Bartek's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address more reivew comments from Ganesh.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add exemplar total label length check.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address a few last review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
Currently hashmod hashes all the labels, even if there is a newline. To
ensure it still works in the future, let's add a test.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
This commit adds `@ <timestamp>` modifier as per this design doc: https://docs.google.com/document/d/1uSbD3T2beM-iX4-Hp7V074bzBRiRNlqUdcWP6JTDQSs/edit.
An example query:
```
rate(process_cpu_seconds_total[1m])
and
topk(7, rate(process_cpu_seconds_total[1h] @ 1234))
```
which ranks based on last 1h rate and w.r.t. unix timestamp 1234 but actually plots the 1m rate.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Testify: move to require
Moving testify to require to fail tests early in case of errors.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* More moves
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Refactor test assertions
This pull request gets rid of assert.True where possible to use
fine-grained assertions.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
This was already fixed by #8013, but add a test case anyway
in case the regexp engine changes in future.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
* cleanup tempfiles for web_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* cleanup tempfiles for api_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* cleanup tempfiles for file_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* Optimized label regex matcher with literal prefix and/or suffix
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Added license
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Added more tests cases with newlines
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Restored deleted test
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Use go1.14 new hash/maphash to hash both RHS and LHS instead of XOR'ing
which has been resulting in hash collisions.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Refactor engine labelset signature generation, just use labels.Labels
instead of hashes.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments; function comments + store result of
lhs.String+rhs.String as key.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Replace all signatureFunc usage with signatureFuncString.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Make optimizations to labels String function and generation of rhs+lhs
as string in resultMetric.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use separate string functions that don't use strconv just for engine
maps.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use a byte invalid separator instead of quoting and have a buffer
attached to EvalNodeHelper instead of using a global pool in the labels
package.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address more review comments, labels has a function that now builds a
byte slice without turning it into a string.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use two different non-ascii hex codes as byte separators between labels
and between sets of labels when building bytes of a Labels struct.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* We only need the 2nd byte invalid sep. at the beginning of a
labels.Bytes
Signed-off-by: Callum Styan <callumstyan@gmail.com>
time.Unix attaches the local timezone, which can then
leak out (e.g. in the alert json). While this is harmless,
we should be consistent.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
* tsdb: don't allow ingesting empty labelsets
When we ingest an empty labelset in the head, further blocks can not be
compacted, with the error:
```
level=error ts=2020-02-27T21:26:58.379Z caller=db.go:659 component=tsdb
msg="compaction failed" err="persist head block: write compaction:
add series: out-of-order series added with label set \"{}\" / prev:
\"{}\""
```
We should therefore reject those invalid empty labelsets upfront.
This can be reproduced with the following:
```
cat << END > prometheus.yml
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 1s
basic_auth:
username: test
password: test
metric_relabel_configs:
- regex: ".*"
action: labeldrop
static_configs:
- targets:
- 127.0.1.1:9090
END
./prometheus --storage.tsdb.min-block-duration=1m
```
And wait a few minutes.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
We can avoid setting a prev token in the OM parser. The previous
coundition that checked for prev was unreacheable.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
