Commit graph

100 commits

Author SHA1 Message Date
Bryan Boreham 5421c778ba labels: in tests use labels.FromStrings
Replacing code which assumes the internal structure of `Labels`.

Add a convenience function `EmptyLabels()` which is more efficient than
calling `New()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-09 13:34:49 +02:00
Julien Pivotto 96d5a32659
Update go to 1.19, set min version to 1.18 (#11279)
* Update go to 1.19, set min version to 1.18

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

* Update golangci-lint

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-09-07 11:30:48 +02:00
Cosrider bef6556ca5
delete redundant alias (#11180)
Signed-off-by: Cosrider <cosrider7@gmail.com>

Signed-off-by: Cosrider <cosrider7@gmail.com>
2022-08-31 15:50:38 +02:00
Bryan Boreham 8b863c42dd
Optimise relabeling by re-using memory (#11147)
* model/relabel: Add benchmark

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* model/relabel: re-use Builder across relabels

Saves memory allocations.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* labels.Builder: allow re-use of result slice

This reduces memory allocations where the caller has a suitable slice available.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* model/relabel: re-use source values slice

To reduce memory allocations.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Unwind one change causing test failures

Restore original behaviour in PopulateLabels, where we must not overwrite the input set.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* relabel: simplify values optimisation

Use a stack-based array for up to 16 source labels, which will be the
vast majority of cases.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* lint

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-19 15:27:52 +05:30
Bryan Boreham a7f19b5775 labels: add a test for JSON and YAML marshalling
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham 10699c37a3 labels: test BytesWithoutLabels does not remove __name__ by default
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham d46ef0aa8e labels: tweak BenchmarkLabels_Get()
So the benchmark works without requiring `Labels` internals to be a slice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham 24ebff9c4a labels: don't test that Hash() works on unordered labels
Labels are required to be ordered.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham fb2d883f1e labels: test Compare without knowing the exact result
Only the sign of the result is important

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Paschalis Tsilias d1122e0743
Introduce TSDB changes for appending metadata to the WAL (#10972)
* Append metadata to the WAL

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove extra whitespace; Reword some docstrings and comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use RLock() for hasNewMetadata check

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use single byte for metric type in RefMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update proposed WAL format for single-byte type metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implementa MetadataAppender interface for the Agent

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of review comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Amend description of metadata in wal.md

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Correct key used to retrieve metadata from cache

When we're setting metadata entries in the scrapeCace, we're using the
p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and
use it as the cache key. When checking for cache entries though, we used
p.Series() as the key, which included the metric name _with_ its labels.
That meant that we were never actually hitting the cache. We're fixing
this by utiling the __name__ internal label for correctly getting the
cache entries after they've been set by setHelp(), setType() or
setUnit().

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Put feature behind a feature flag

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix AppendMetadata docstring

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reorder WAL format document

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse temporary buffer in Metadata encoder

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Only keep latest metadata for each refID during checkpointing

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test that's referencing decoding metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Avoid creating metadata block if no new metadata are present

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for corrupt metadata block and relevant record type

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix CR comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Extract logic about changing metadata in an anonymous function

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implement new proposed WAL format and amend relevant tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use 'const' for metadata field names

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Apply metadata to head memSeries in Commit, not in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add docstring and rename extracted helper in scrape.go

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for tsdb-related cases

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol1

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol2

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix Windows test by closing WAL reader files

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use switch instead of two if statements in metadata decoding

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments around TestMetadata* tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add code for replaying WAL; test correctness of in-memory data after a replay

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove scrape-loop related code from PR

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify tests by sorting slices before comparison

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test to use separate transactions

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Empty out buffer and record slices after encoding latest metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linting issue

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update calculation for DroppedMetadata metric

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse buffer when encoding latest metadata for each series

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments; Check all returned error values using two helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify use of helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Satisfy linter

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-07-19 10:58:52 +02:00
Julien Pivotto c637705403
Merge pull request #10893 from prymitive/unwrap_errors
Implement Unwrap() on errors returned from rulefmt
2022-06-30 11:30:21 +02:00
Matthieu MOREL c2b4de3611
refactor (package model): move from github.com/pkg/errors to 'errors' and 'fmt' packages (#10747)
Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>
Co-Authored-By: Julien Pivotto <roidelapluie@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
2022-06-27 21:29:19 +02:00
Łukasz Mierzwa 648b12d8c5 Implement Unwrap() on errors returned from rulefmt
I'd like to unwrap errors returned from rulefmt but both Error and WrappedError types are missing Unwrap() method.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-06-20 17:01:52 +01:00
Paschalis Tsilias 4f3791024a
Drop extra string held in relabel.Regexp struct (#10846)
* Drop extra string held in relabel.Regexp struct

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use slice operations instead of TrimPrefix/TrimSuffix; Override String() method

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-06-13 15:00:48 +02:00
Bryan Boreham 2e2c014d52
Labels: optimise creation of signature with/without labels (#10667)
* Labels: create signature with/without labels

Instead of creating a new Labels slice then converting to signature,
go directly to the signature and save time.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Labels: refactor Builder tests

Have one test with a range of cases, and have them check the final
output rather than checking the internal structure of the Builder.

Also add a couple of cases where the value is "", which should be
interpreted as 'delete'.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Labels: add 'Keep' function to Builder

This lets us replace `Labels.WithLabels` with the more general `Builder`.

In `engine.resultMetric()` we can call `Keep()` instead of checking
and calling `Del()`.

Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything,
so that `Keep()` has the same performance as `WithLabels()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-06-07 10:08:27 +05:30
Łukasz Mierzwa b11062bfcc
Don't count added labels when initializing slice (#10756)
This was added in #10749 but based on feedback it might be over-allocating so worth removing.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-06-04 08:45:02 +05:30
Łukasz Mierzwa 08262454a3
Preallocate Labels in labels.Builder (#10749)
This tries to avoid re-allocations of labels slice since we know possible max size

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-05-25 16:22:47 +02:00
Łukasz Mierzwa 89de30a0b7
Avoid reallocating map in MatchLabels (#10715)
We know the max size of our map so we can create it with that information and avoid extra allocations

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-05-20 10:18:31 +02:00
Julien Pivotto 71dbb4d091
Add lowercase and uppercase relabel action (#10641)
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-05-03 10:09:53 +02:00
Matthieu MOREL e2ede285a2
refactor: move from io/ioutil to io and os packages (#10528)
* refactor: move from io/ioutil to io and os packages
* use fs.DirEntry instead of os.FileInfo after os.ReadDir

Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>
2022-04-27 11:24:36 +02:00
Peter Štibraný cd18da3605
Fix OpenMetrics parser to sort uppercase labels correctly. (#10510)
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
2022-03-31 20:18:53 +02:00
Nick Pillitteri 53ac9d6d66
Compare equality of label.Label structs directly (#10427)
* labels.Equal benchmark for equal, not equal, and differing lengths

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Compare equality of label.Label structs directly

Compare the structs using `==` instead of the name and value
of each label. This is functionally equivalent and about ~10%
faster in my testing.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Use longer more realistic names and values in benchmark

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
2022-03-15 00:30:04 +01:00
Levi Harrison a9e2ffb717
Specify type in metadata parser errors (#10269) 2022-03-08 08:04:11 -05:00
Łukasz Mierzwa da23c4649a
Enable misspell check in golangci-lint (#10393)
Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-03-03 18:11:19 +01:00
Matthias Loibl afdd1357e0
Merge pull request #10267 from roidelapluie/contenttype
Improve content-type error handling
2022-02-14 11:57:09 +01:00
Julien Pivotto 9a2e93228e
Switch to grafana/regexp everywhere (#10268)
Let's have a consistent library for regexp.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2022-02-13 00:58:27 +01:00
Bryan Boreham 579331446a
Allow downstream projects to use faster regexp (#10251)
* Add benchmark for FastRegexMatcher

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Use modified regexp package with optimisations

See https://github.com/grafana/regexp/tree/speedup#readme

Includes the following changes proposed upstream:
* [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711)
* [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909)
* [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756)
* [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789)
* [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Use regexp code identical to upstream Go

Change `grafana/regexp` import to use `main` branch.

This means Prometheus is not using the proposed optimisations, but
downstream users of Prometheus code are able to `replace` the library
with the `speedup` branch which does.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-02-08 11:03:20 +01:00
Julien Pivotto f695df843f Improve content-type error handling
- Call err everywhere
- Change log message to underscore-separated field

Followup on #10186

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2022-02-08 11:02:51 +01:00
Matheus Pimenta 8d8ce641a4
error for invalid media type should not be completely swallowed (#10186)
* error for invalid media type should not be completely swallowed

Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
2022-02-08 10:57:56 +01:00
ozairasim 52159840eb
Expose Error function for WrappedError (#9662) 2021-11-12 17:38:59 -05:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Bjoern Rabenstein 006b5517e2 Simplify makefiles.
This removes the dependancy on C leveldb and snappy.
It also takes care of fewer dependencies as they would
anyway not work on any non-Debian, non-Brew system.

Change-Id: Ia70dce1ba8a816a003587927e0b3a3f8ad2fd28c
2014-11-25 17:10:39 +01:00
Julius Volz ae30453214 Add label names -> label values index.
Change-Id: Ie39b4044558afc4d1aa937de7dcf8df61f821fb4
2014-03-28 15:16:37 +01:00
Julius Volz 1eee448bc1 Store samples in custom binary encoding.
This has been shown to provide immense decoding speed benefits.

See also:

https://groups.google.com/forum/#!topic/prometheus-developers/FeGl_qzGrYs

Change-Id: I7d45b4650e44ddecaa91dad9d7fdb3cd0b9f15fe
2014-03-09 22:31:38 +01:00
Julius Volz a310afd28f Update "build" -> ".build" in remaining Makefiles. 2013-06-26 16:09:37 +02:00
Matt T. Proud 30b1cf80b5 WIP - Snapshot of Moving to Client Model. 2013-06-25 15:52:42 +02:00
Julius Volz 0226d1ac7a Implement alerts dashboard and expression console links. 2013-06-13 22:35:40 +02:00
Matt T. Proud f895acb9ef Trailing build system cleanups.
The .gitignore files had problems, and the build steps could be
simplified.
2013-06-13 15:38:03 +02:00
Julius Volz fc97e688c6 Improve printing of rules and expressions. 2013-06-11 11:39:31 +02:00
Julius Volz 74cb676537 Implement Stringer interface for rules and all their children. 2013-06-07 15:54:32 +02:00
Julius Volz dcfd09c801 Prepend "exporter_" to labels that already exist in exported metrics.
If the metrics exported by a process already contain any of a target's
base labels (such as "job" or "instance", but also any manually assigned
target-group label), don't overwrite that label, but instead add a new
label consisting of the original label name prepended with "exporter_".
This is to accomodate intermediate exporter jobs, which might indicate
e.g. the jobs and instances for which they are exporting data.
2013-06-02 22:48:46 +02:00
Julius Volz 5b105c77fc Repointerize fingerprints. 2013-05-21 14:28:14 +02:00
Julius Volz 259a0ea467 Optimize fingerprint comparison time. 2013-05-21 14:00:52 +02:00
Matt T. Proud 8f4c7ece92 Destroy naked returns in half of corpus.
The use of naked return values is frowned upon.  This is the first
of two bulk updates to remove them.
2013-05-16 10:53:25 +03:00
Matt T. Proud 244a4a9cdb Update to go1.1.
This commit updates the documentation, Makefiles, formatting, and
code semantics to support the 1.1. runtime, which includes ...

1. ``make advice``,

2. ``make format``, and

3. ``go fix`` on various targets.
2013-05-14 12:39:08 +02:00
Matt T. Proud 161c8fbf9b Include deletion processor for long-tail values.
This commit extracts the model.Values truncation behavior into the actual
tiered storage, which uses it and behaves in a peculiar way—notably the
retention of previous elements if the chunk were to ever go empty.  This is
done to enable interpolation between sparse sample values in the evaluation
cycle.  Nothing necessarily new here—just an extraction.

Now, the model.Values TruncateBefore functionality would do what a user
would expect without any surprises, which is required for the
DeletionProcessor, which may decide to split a large chunk in two if it
determines that the chunk contains the cut-off time.
2013-05-10 12:19:12 +02:00
Matt T. Proud 38d839d810 Include generated Protocol Buffers artifacts.
This commit reduces the general compile time dependencies to omit
the Protocol Buffer compiler and the Go Protocol Buffer generator
tool.  The build steps to furnish them still remain, but they can
optionally be called if data.proto or config.proto are under work.
2013-05-08 21:16:39 +02:00
Matt T. Proud 3b9b1c6ab4 Define dependencies for web. stack concretely.
This commit destroys the use of AppState, which makes passing
concrete state along to various serving components onerous.
2013-05-06 11:13:12 +02:00
Matt T. Proud 6fac20c8af Harden the tests against OOMs.
This commit employs explicit memory freeing for the in-memory storage
arenas.  Secondarily, we take advantage of smaller channel buffer sizes
in the test.
2013-04-29 11:46:01 +02:00
Bernerd Schaefer b04cd28862 Merge pull request #192 from prometheus/feature/negotiate-telemetry-schema-through-mime-type
Use Content-Type data for telemetry versioning
2013-04-29 01:30:37 -07:00