Commit graph

39 commits

Author SHA1 Message Date
Jeanette Tan e9a1e26ab7 Perform integer/float histogram type checking on conversions, and use a consistent method for determining integer vs float histogram
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-04-22 02:27:15 +08:00
beorn7 c3c7d44d84 lint: Adjust to the lint warnings raised by current versions of golint-ci
We haven't updated golint-ci in our CI yet, but this commit prepares
for that.

There are a lot of new warnings, and it is mostly because the "revive"
linter got updated. I agree with most of the new warnings, mostly
around not naming unused function parameters (although it is justified
in some cases for documentation purposes – while things like mocks are
a good example where not naming the parameter is clearer).

I'm pretty upset about the "empty block" warning to include `for`
loops. It's such a common pattern to do something in the head of the
`for` loop and then have an empty block. There is still an open issue
about this: https://github.com/mgechev/revive/issues/810 I have
disabled "revive" altogether in files where empty blocks are used
excessively, and I have made the effort to add individual
`// nolint:revive` where empty blocks are used just once or twice.
It's borderline noisy, though, but let's go with it for now.

I should mention that none of the "empty block" warnings for `for`
loop bodies were legitimate.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:10:10 +02:00
Justin Lei 83f43982c9 Add support for native histograms to concreteSeriesIterator
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:54:15 -07:00
Charles Korn 0a1de58f7e
Mark Histogram.(Positive|Negative)Spans as non-nullable.
As far as I understand it, we'd never expect to receive a nil span,
and remote.spansProtoToSpans would panic if we received a nil span.

Marking the fields as non-nullable also means the generated Golang
code doesn't use pointers for these fields, reducing allocations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-02-03 13:49:22 +11:00
György Krajcsovits 2d9a9cbc08 Fix storage/remote/codec ignoreing histogram reset hint
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-01-24 12:56:30 +01:00
Marc Tudurí 721f33dbb0
histograms: Add remote-write support for Float Histograms (#11817)
* adapt code.go and write_handler.go to support float histograms
* adapt watcher.go to support float histograms
* wip adapt queue_manager.go to support float histograms
* address comments for metrics in queue_manager.go
* set test cases for queue manager
* use same counts for histograms and float histograms
* refactor createHistograms tests
* fix float histograms ref in watcher_test.go
* address PR comments

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-01-13 16:39:20 +05:30
Bryan Boreham ccea61c7bf
Merge pull request #11717 from bboreham/labels-abstraction
Add and use abstractions over labels.Labels
2022-12-20 17:23:39 +00:00
Sniper91 46fb802791
reset frameBytesLeft after writing (#11689)
Signed-off-by: sniper91 <kevinzhao91@outlook.com>

Signed-off-by: sniper91 <kevinzhao91@outlook.com>
2022-12-19 16:54:49 +01:00
Bryan Boreham 047585360b Update package storage/remote tests for new labels.Labels type
Use ScratchBuilder to create labels.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 3c7de69059 storage: allow re-use of iterators
Patterned after `Chunk.Iterator()`: pass the old iterator in so it
can be re-used to avoid allocating a new object.

(This commit does not do any re-use; it is just changing all the method
signatures so re-use is possible in later commits.)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-15 18:32:45 +00:00
Jesus Vazquez e934d0f011 Merge 'main' into sparsehistogram
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2022-10-05 22:14:49 +02:00
Bryan Boreham 3029320ce6 storage/remote: in tests use labels.FromStrings
And a few cases of `EmptyLabels()`.
Replacing code which assumes the internal structure of `Labels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-09 13:34:49 +02:00
Levi Harrison 0db6b072bc
Export histogramToHistogramProto() (#11046)
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2022-07-21 10:12:50 -04:00
Levi Harrison 08f3ddb864
Sparse histogram remote-write support (#11001) 2022-07-14 09:13:12 -04:00
beorn7 86cc83b13c storage: iterator fixes after merge
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-18 14:12:01 +01:00
beorn7 64c7bd2b08 Merge branch 'main' into sparsehistogram 2021-12-18 14:04:25 +01:00
beorn7 b042e29569 storage: expose bug in iterators #10027
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-16 12:02:15 +01:00
beorn7 68e02be963 Post-merge fixes
Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-30 17:20:28 +01:00
Björn Rabenstein b866db009b
storage: Fix and improve the Seek method of various iterators (#9878)
There was a subtle and nasty bug in listSeriesIterator.Seek.

In addition, the Seek call is defined to be a no-op if the current
position of the iterator is already pointing to a suitable
sample. This commit adds fast paths for this case to several
potentially expensive Seek calls.

Another bug was in concreteSeriesIterator.Seek. It always searched the
whole series and not from the current position of the iterator.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-29 15:17:56 +05:30
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Bryan Boreham 5afa606ecb
Remote-write: reuse memory for marshalling (#9412)
By holding a `proto.Buffer` per shard and passing it down to where
marshalling is done, we avoid creating a lot of garbage.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-10-29 14:44:40 -07:00
Serge Catudal d77c985f8c
Add initial support for exemplar to the remote write receiver endpoint (#9319)
* Add initial support for exemplar to the remote write receiver endpoint

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>

* Update storage remote write handler tests with exemplars

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>

* Update remote write handler in order to have a distinct checkAppendExemplarError function from scrape

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>
2021-09-21 14:53:27 -06:00
Tom Wilkie d479151f1f Various enhancements and refactorings for remote write receiver:
- Remove unrelated changes
- Refactor code out of the API module - that is already getting pretty crowded.
- Don't track reference for AddFast in remote write.  This has the potential to consume unlimited server-side memory if a malicious client pushes a different label set for every series.  For now, its easier and safer to always use the 'slow' path.
- Return 400 on out of order samples.
- Use remote.DecodeWriteRequest in the remote write adapters.
- Put this behing the 'remote-write-server' feature flag
- Add some (very) basic docs.
- Used named return & add test for commit error propagation

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-08 20:41:23 +00:00
gotjosh 4eca4dffb8
Allow metric metadata to be propagated via Remote Write. (#6815)
* Introduce a metadata watcher

Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage.

Signed-off-by: gotjosh <josue@grafana.com>

* Additional fixes after rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Rework samples/metadata metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use more descriptive variable names in MetadataWatcher collect.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix issues caused during rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix missing metric add and unneeded config code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix metrics and docs

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Replace assert with require

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Bring back max_samples_per_send metric

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-11-19 20:53:03 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122)
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110)
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Hu Shuai b962029031
Add a unit test for MergeLabels in storage/remote/codec.go. (#7499)
This PR is about adding a unit test for MergeLabels in storage/remote/codec.go.

Signed-off-by: Hu Shuai <hus.fnst@cn.fujitsu.com>
2020-07-04 22:17:19 -06:00
ZouYu 2b7437d60e
Fix some warnings: 'redundant type from array, slice, or map composite literal' (#7109)
Signed-off-by: ZouYu <zouy.fnst@cn.fujitsu.com>
2020-04-15 11:17:41 +01:00
Bartlomiej Plotka fe802f29c9 storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response.
This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on https://github.com/prometheus/prometheus/pull/5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-03-13 13:06:25 +00:00
George Felix 895abbb7d0 Replaced test validations with testutils on storage/remote/codec_test.go (#6097)
* Replaced test validations with testutils on storage/remote/codec_test.go

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* gofmt

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* Removed shouldPass assertion

Signed-off-by: George Felix <gfelixc@gmail.com>

* Fixes to improve readability

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* Fixes based on code review comments

Signed-off-by: George Felix <george.felix@ubeeqo.com>
2019-10-07 11:35:53 -06:00
Bartek Płotka 48b2c9c8ea
remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703)
Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488

Changes:
* Extended protobuf for chunked remote read and negotation.
* Added checksumed, chunked Writer/Reader.
* Added Server side implementation for chunked streamed remote-read.


Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-08-19 21:16:10 +01:00
Chris Marchbanks 529ccff07b
Remove all usages of stretchr/testify
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:49:27 -06:00
Vadym Martsynovskyy 8318aa2d5d Check for duplicate label names in remote read (#5829)
* Check for duplicate label names in remote read

Also add test to confirm that #5731 is fixed

* Use subtests in TestValidateLabelsAndMetricName

* Really check that expectedErr matches err

Signed-off-by: Vadym Martsynovskyy <vmartsynovskyy@gmail.com>
2019-08-07 16:13:10 +01:00
Matt Layher 302148fd69 *: apply gofmt -s
Signed-off-by: Matt Layher <mdlayher@gmail.com>
2019-01-16 17:28:14 -05:00
Tom Wilkie d3a1ff1abf
Reduce memory usage of remote read by reducing pointer usage. (#4655)
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-09-25 19:14:00 +01:00
Julius Volz 9f10c63cff
Fix remote read labelset corruption (#3456)
The labelsets returned from remote read are mutated in higher levels
(like seriesFilter.Labels()) and since the concreteSeriesSet didn't
return a copy, the external mutation affected the labelset in the
concreteSeries itself. This resulted in bizarre bugs where local and
remote series would show with identical label sets in the UI, but not be
deduplicated, since internally, a series might come to look like:

{__name__="node_load5", instance="192.168.1.202:12090", job="node_exporter", node="odroid", node="odroid"}

(note the repetition of the last label)
2017-11-12 00:47:47 +01:00
Tom Wilkie 2ae04d0e79 Add license header. 2017-10-26 11:09:16 +01:00
Tom Wilkie ee011d906d Port remote read server to 2.0. 2017-10-26 11:09:14 +01:00