Commit graph

197 commits

Author SHA1 Message Date
Callum Styan 934bb2dbc6 wip, rework compression testing after format PR was merged
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-23 19:25:00 -08:00
Nicolás Pazos 32eb50917e No Close method and basic pooling 2023-11-23 16:47:24 -08:00
alexgreenbank 0ab4808153 Add alternative compression algos to experiment 2023-11-23 16:45:10 -08:00
Callum Styan 788dfc8cb4 fix minor lint issue + use labels Range function since it looks like
the tests fail to do `range labels.Labels` on CI

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-23 11:55:56 -08:00
Callum Styan e2acded6ca more cleanup, address review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-23 11:31:10 -08:00
Callum Styan a3c6904243 more cleanup, mostly linting fixes
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-20 17:28:05 -08:00
Callum Styan a0cd7939e2 cleanup; remove some unused functions
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-20 16:51:53 -08:00
Callum Styan fb3ed046ac remove more format types we probably won't use
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-20 16:19:42 -08:00
Callum Styan bf773afd57 remove formats we've decided not to use
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-19 19:05:30 -08:00
Callum Styan e444439d59 test additional len and lenbytes formats
Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-15 10:04:38 -08:00
Callum Styan 1842c99f2f Add bytes slice (instead of slice of 32bit vars) format for testing
Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-13 21:53:44 -08:00
Callum Styan ed0c3b25c7 refactor new version flag to make it easier to pick a specific format
instead of having multiple flags, plus add new formats for testing

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-13 15:41:01 -08:00
Nicolás Pazos 4bdb865ae3 minimally-tested exemplar support for rw 1.1
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-11-09 17:00:16 -03:00
Nicolás Pazos 34f3f11ee2 remove all code from previous interning approach
the 'minimized' version is now the only v1.1 version

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-11-09 17:00:16 -03:00
Nicolás Pazos e2a5ea5397 Use unsafe []byte->string cast to reuse buffer
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-11-09 15:08:49 -03:00
Nicolás Pazos c9b6dddef9 manually optimize varint marshaling
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-11-09 15:05:16 -03:00
Nicolás Pazos 761efc860e Use two uint32 to encode (offset,leng)
Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
2023-11-09 15:03:18 -03:00
Callum Styan eb63f30459 fix minor things
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-09 10:35:43 -03:00
Callum Styan 3e5facc0c0 add functionality for new minimized remote write request format
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-09 10:35:43 -03:00
Nicolás Pazos 46b84ab3fb lint 2023-11-09 10:18:12 -03:00
alexgreenbank 8ab14f2456 Remove config, update proto 2023-11-09 10:18:12 -03:00
alexgreenbank 15c4d45635 Add 1.1 version handling code 2023-11-09 10:18:12 -03:00
Nicolás Pazos 44f166d066 Use github.com/golang/snappy 2023-11-09 10:18:12 -03:00
Nicolás Pazos c1593fd35a Improve sender benchmarks and some allocations 2023-11-09 10:18:12 -03:00
Nicolás Pazos b35ab6c080 fix build 2023-11-09 10:18:12 -03:00
Nicolás Pazos 2f815ee3dd refactor queue manager code to remove some duplication 2023-11-09 10:18:12 -03:00
Nicolás Pazos eebf7ac1fc fix: queue manager to include float histograms in new requests 2023-11-09 10:18:12 -03:00
Nicolás Pazos ed34405d68 remove some comented code 2023-11-09 10:18:12 -03:00
Callum Styan f1992591b2 Implement code paths for new proto format
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-09 10:18:12 -03:00
Callum Styan 7a862d265f replace snappy encoding library
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2023-11-09 10:18:12 -03:00
Levi Harrison dcaca86958
Update dependencies for 2.48 (#12964) 2023-10-15 10:53:59 -04:00
Paschalis Tsilias c173cd57c9
Add a header to count retried remote write requests (#12729)
Header name is `Retry-Attempt`, only set when >0.

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
2023-09-20 11:11:03 +01:00
Bryan Boreham d2ae8dc3cb remote-write: add http.resend_count tracing attribute
As recommended by the OpenTelemetry semantic conventions.

https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/http/#http-client
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-11 16:20:12 +00:00
Matthieu MOREL bae9a21200
Merge branch 'main' into linter/nilerr
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-19 19:56:39 +02:00
beorn7 5b53aa1108 style: Replace else if cascades with switch
Wiser coders than myself have come to the conclusion that a `switch`
statement is almost always superior to a statement that includes any
`else if`.

The exceptions that I have found in our codebase are just these two:

* The `if else` is followed by an additional statement before the next
  condition (separated by a `;`).
* The whole thing is within a `for` loop and `break` statements are
  used. In this case, using `switch` would require tagging the `for`
  loop, which probably tips the balance.

Why are `switch` statements more readable?

For one, fewer curly braces. But more importantly, the conditions all
have the same alignment, so the whole thing follows the natural flow
of going down a list of conditions. With `else if`, in contrast, all
conditions but the first are "hidden" behind `} else if `, harder to
spot and (for no good reason) presented differently from the first
condition.

I'm sure the aforemention wise coders can list even more reasons.

In any case, I like it so much that I have found myself recommending
it in code reviews. I would like to make it a habit in our code base,
without making it a hard requirement that we would test on the CI. But
for that, there has to be a role model, so this commit eliminates all
`if else` occurrences, unless it is autogenerated code or fits one of
the exceptions above.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:22:31 +02:00
Matthieu MOREL fb3eb21230 enable gocritic, unconvert and unused linters
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-13 19:20:22 +00:00
Arve Knudsen bc9a82f5a1
remote: Improve some comments (#12102)
Improve some comments in storage/remote/queue_manager.go, wrt. general
language and a typo.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-03-09 11:05:24 +00:00
Arve Knudsen 435b500de7
remote: Convert to RecoverableError using errors.As (#12103)
In storage/remote, try converting to RecoverableError using errors.As,
instead of through direct casting.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-03-08 13:58:09 -07:00
Julien Pivotto 475f9984d0
Merge pull request #11787 from damnever/perf/avoid-alloc-if-no-externallabels
Avoid allocation during remote write if external labels is empty
2023-02-22 23:38:21 +01:00
Marc Tudurí 721f33dbb0
histograms: Add remote-write support for Float Histograms (#11817)
* adapt code.go and write_handler.go to support float histograms
* adapt watcher.go to support float histograms
* wip adapt queue_manager.go to support float histograms
* address comments for metrics in queue_manager.go
* set test cases for queue manager
* use same counts for histograms and float histograms
* refactor createHistograms tests
* fix float histograms ref in watcher_test.go
* address PR comments

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-01-13 16:39:20 +05:30
Xiaochao Dong (@damnever) 2d61d012ff Avoid copy during remote write if external labels is empty
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2022-12-30 19:18:30 +08:00
Bryan Boreham abd9909595 Update package storage/remote for new labels.Labels type
`QueueManager.externalLabels` becomes a slice rather than a `Labels` so
we can index into it when doing the merge operation.

Note we avoid calling `Labels.Len()` in `labelProtosToLabels()`.
It isn't necessary - `append()` will enlarge the buffer and we're
expecting to re-use it many times.

Also, we now validate protobuf input before converting to Labels.
This way we can detect errors first, and we don't place unnecessary
requirements on the Labels structure.

Re-do seriesFilter using labels.Builder (albeit N^2).

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Ganesh Vernekar 648be89822
Merge remote-tracking branch 'upstream/main' into fix-conflict
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 14:20:02 +05:30
Jesus Vazquez 775d90d5f8
TSDB: Rename wal package to wlog (#11352)
The wlog.WL type can now be used to create a Write Ahead Log or a Write
Behind Log.

Before the prefix for wbl metrics was
'prometheus_tsdb_out_of_order_wal_' and has been replaced with
'prometheus_tsdb_out_of_order_wbl_'.

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2022-10-10 20:38:46 +05:30
Ganesh Vernekar f540c1dbd3
Add support for histograms in WAL checkpointing (#11210)
* Add support for histograms in WAL checkpointing

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-08-29 17:38:36 +05:30
Levi Harrison 0db6b072bc
Export histogramToHistogramProto() (#11046)
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2022-07-21 10:12:50 -04:00
Levi Harrison 08f3ddb864
Sparse histogram remote-write support (#11001) 2022-07-14 09:13:12 -04:00
Matej Gera 1dd247f68b
Remote Write: Rename confusing walDir parameter to dir (#10464)
* Rename walDir parameter to dir

Signed-off-by: Matej Gera <matejgera@gmail.com>

* Improve NewQueueManager comment

Signed-off-by: Matej Gera <matejgera@gmail.com>
2022-05-30 21:45:30 -07:00
Chris Marchbanks a11e73edda
Fix a deadlock between Batch and FlushAndShutdown (#10608)
If FlushAndShutdown is called with a full batchQueue, and then Batch is
called rather than the normal path of reading from a queue a deadlock
might be encountered. Rather than having FlushAndShutdown having
blocking code while holding a lock retry sending the batch every second.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2022-04-20 20:50:41 +02:00
beorn7 79376c1e94 Merge branch 'release-2.33' into beorn7/release 2022-03-08 17:42:49 +01:00