Commit graph

200 commits

Author SHA1 Message Date
Nicolás Pazos 48f9285b11 define separate proto types for remote write 2.0
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-21 09:59:52 -03:00
Nicolás Pazos 66f93860ce lint fixes
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Nicolás Pazos 934de724ce lint
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Nicolás Pazos 25c8baeb6c cleanup rwSymbolTable
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Nicolás Pazos ec9300fc1a remove all new rw formats but the []string one
also adapt tests to the new format

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Nicolás Pazos 31d3956f47 new interning format based on []string indeces
Co-authored-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Callum Styan 3e48b8a582 fix minor lint issue + use labels Range function since it looks like
the tests fail to do `range labels.Labels` on CI

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Callum Styan 7630577850 more cleanup, address review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Callum Styan a8639ddb36 more cleanup, mostly linting fixes
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Callum Styan 766a12fb76 cleanup; remove some unused functions
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:29 -03:00
Callum Styan 4bdb73780c remove more format types we probably won't use
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:11 -03:00
Callum Styan 81c0d78ad4 remove formats we've decided not to use
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:11 -03:00
Callum Styan 97d0556584 test additional len and lenbytes formats
Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:11 -03:00
Callum Styan 8b0cda8bb3 Add bytes slice (instead of slice of 32bit vars) format for testing
Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:11 -03:00
Callum Styan 23c7936cee refactor new version flag to make it easier to pick a specific format
instead of having multiple flags, plus add new formats for testing

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:29:11 -03:00
Nicolás Pazos 8353c337b4 minimally-tested exemplar support for rw 1.1
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 8426752e0f remove all code from previous interning approach
the 'minimized' version is now the only v1.1 version

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos f46a104752 Use unsafe []byte->string cast to reuse buffer
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 5dbe3c1919 manually optimize varint marshaling
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 0584610080 Use two uint32 to encode (offset,leng)
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Callum Styan 2dacd72102 fix minor things
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Callum Styan 622a723626 add functionality for new minimized remote write request format
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 68a9f8bb54 lint
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
alexgreenbank 3f33175e0b Remove config, update proto
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
alexgreenbank d1d80b8bed Add 1.1 version handling code
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 26673bcbde Use github.com/golang/snappy
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos c710e19f9c Improve sender benchmarks and some allocations
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 9b4fbd9552 fix build
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos e70780ca86 refactor queue manager code to remove some duplication
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos e909eaea31 fix: queue manager to include float histograms in new requests
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Nicolás Pazos 12de4c4a62 remove some comented code
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:54 -03:00
Callum Styan ce1e2ad0b9 Implement code paths for new proto format
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:09 -03:00
Callum Styan 4c4b9aa471 replace snappy encoding library
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-12-19 14:27:09 -03:00
Levi Harrison dcaca86958
Update dependencies for 2.48 (#12964) 2023-10-15 10:53:59 -04:00
Paschalis Tsilias c173cd57c9
Add a header to count retried remote write requests (#12729)
Header name is `Retry-Attempt`, only set when >0.

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
2023-09-20 11:11:03 +01:00
Bryan Boreham d2ae8dc3cb remote-write: add http.resend_count tracing attribute
As recommended by the OpenTelemetry semantic conventions.

https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/http/#http-client
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-11 16:20:12 +00:00
Matthieu MOREL bae9a21200
Merge branch 'main' into linter/nilerr
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-19 19:56:39 +02:00
beorn7 5b53aa1108 style: Replace else if cascades with switch
Wiser coders than myself have come to the conclusion that a `switch`
statement is almost always superior to a statement that includes any
`else if`.

The exceptions that I have found in our codebase are just these two:

* The `if else` is followed by an additional statement before the next
  condition (separated by a `;`).
* The whole thing is within a `for` loop and `break` statements are
  used. In this case, using `switch` would require tagging the `for`
  loop, which probably tips the balance.

Why are `switch` statements more readable?

For one, fewer curly braces. But more importantly, the conditions all
have the same alignment, so the whole thing follows the natural flow
of going down a list of conditions. With `else if`, in contrast, all
conditions but the first are "hidden" behind `} else if `, harder to
spot and (for no good reason) presented differently from the first
condition.

I'm sure the aforemention wise coders can list even more reasons.

In any case, I like it so much that I have found myself recommending
it in code reviews. I would like to make it a habit in our code base,
without making it a hard requirement that we would test on the CI. But
for that, there has to be a role model, so this commit eliminates all
`if else` occurrences, unless it is autogenerated code or fits one of
the exceptions above.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:22:31 +02:00
Matthieu MOREL fb3eb21230 enable gocritic, unconvert and unused linters
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-13 19:20:22 +00:00
Arve Knudsen bc9a82f5a1
remote: Improve some comments (#12102)
Improve some comments in storage/remote/queue_manager.go, wrt. general
language and a typo.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-03-09 11:05:24 +00:00
Arve Knudsen 435b500de7
remote: Convert to RecoverableError using errors.As (#12103)
In storage/remote, try converting to RecoverableError using errors.As,
instead of through direct casting.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-03-08 13:58:09 -07:00
Julien Pivotto 475f9984d0
Merge pull request #11787 from damnever/perf/avoid-alloc-if-no-externallabels
Avoid allocation during remote write if external labels is empty
2023-02-22 23:38:21 +01:00
Marc Tudurí 721f33dbb0
histograms: Add remote-write support for Float Histograms (#11817)
* adapt code.go and write_handler.go to support float histograms
* adapt watcher.go to support float histograms
* wip adapt queue_manager.go to support float histograms
* address comments for metrics in queue_manager.go
* set test cases for queue manager
* use same counts for histograms and float histograms
* refactor createHistograms tests
* fix float histograms ref in watcher_test.go
* address PR comments

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-01-13 16:39:20 +05:30
Xiaochao Dong (@damnever) 2d61d012ff Avoid copy during remote write if external labels is empty
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2022-12-30 19:18:30 +08:00
Bryan Boreham abd9909595 Update package storage/remote for new labels.Labels type
`QueueManager.externalLabels` becomes a slice rather than a `Labels` so
we can index into it when doing the merge operation.

Note we avoid calling `Labels.Len()` in `labelProtosToLabels()`.
It isn't necessary - `append()` will enlarge the buffer and we're
expecting to re-use it many times.

Also, we now validate protobuf input before converting to Labels.
This way we can detect errors first, and we don't place unnecessary
requirements on the Labels structure.

Re-do seriesFilter using labels.Builder (albeit N^2).

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Ganesh Vernekar 648be89822
Merge remote-tracking branch 'upstream/main' into fix-conflict
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 14:20:02 +05:30
Jesus Vazquez 775d90d5f8
TSDB: Rename wal package to wlog (#11352)
The wlog.WL type can now be used to create a Write Ahead Log or a Write
Behind Log.

Before the prefix for wbl metrics was
'prometheus_tsdb_out_of_order_wal_' and has been replaced with
'prometheus_tsdb_out_of_order_wbl_'.

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2022-10-10 20:38:46 +05:30
Ganesh Vernekar f540c1dbd3
Add support for histograms in WAL checkpointing (#11210)
* Add support for histograms in WAL checkpointing

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-08-29 17:38:36 +05:30
Levi Harrison 0db6b072bc
Export histogramToHistogramProto() (#11046)
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2022-07-21 10:12:50 -04:00
Levi Harrison 08f3ddb864
Sparse histogram remote-write support (#11001) 2022-07-14 09:13:12 -04:00