Commit graph

24 commits

Author SHA1 Message Date
Owen Williams a28d7865ad UTF-8: Add support for parsing UTF8 metric and label names
This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats.
This grammar will also be valid for non-UTF-8 names.
UTF-8 names will not be considered valid unless model.NameValidationScheme is changed.

This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/.

Part of https://github.com/prometheus/prometheus/issues/13095

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-02-15 14:34:37 -05:00
Daniel Kerbel b2185d96af
Consider storage.ErrTooOldSample as non-retryable
Signed-off-by: Daniel Kerbel <nmdanny@gmail.com>
2023-12-26 18:44:39 +02:00
Goutham 3048a88ae7
Add suffixes
Older version already did that. This upgrade needed manual opt-in

Signed-off-by: Goutham <gouthamve@gmail.com>
2023-11-15 15:52:18 +01:00
Matthieu MOREL 1ec6e407d0
ci(lint): enable errorlint on storage (#12935)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-10-31 12:15:30 +01:00
Goutham Veeramachaneni ad4f514e66
Add OTLP Ingestion endpoint (#12571)
* Add OTLP Ingestion endpoint

We copy files from the otel-collector-contrib. See the README in
`storage/remote/otlptranslator/README.md`.

This supersedes: https://github.com/prometheus/prometheus/pull/11965

Signed-off-by: gouthamve <gouthamve@gmail.com>

* Return a 200 OK

It is what the OTEL Golang SDK expect :(

https://github.com/open-telemetry/opentelemetry-go/issues/4363

Signed-off-by: Goutham <gouthamve@gmail.com>

---------

Signed-off-by: gouthamve <gouthamve@gmail.com>
Signed-off-by: Goutham <gouthamve@gmail.com>
2023-07-28 12:35:28 +02:00
LHHDZ 7d8f9b0978
remote-write receiver: reuse 'ref' to optimize multiple samples for same series (#12580)
reuse 'ref' to optimize multi samples processing efficiency

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-07-22 14:24:46 +01:00
Julien Pivotto 986fde06b2
Merge pull request #11688 from damnever/fix/datamodelvalidation-remotewriteapi
Validate the metric names and labels in the remote write handler
2023-07-04 13:52:02 +02:00
Jeanette Tan e9a1e26ab7 Perform integer/float histogram type checking on conversions, and use a consistent method for determining integer vs float histogram
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-04-22 02:27:15 +08:00
Justin Lei 83f43982c9 Add support for native histograms to concreteSeriesIterator
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-04-06 09:54:15 -07:00
Xiaochao Dong (@damnever) 2b7202c4cc Validate the metric names and labels in the remote write handler
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-04-05 19:09:05 +08:00
Marc Tudurí 721f33dbb0
histograms: Add remote-write support for Float Histograms (#11817)
* adapt code.go and write_handler.go to support float histograms
* adapt watcher.go to support float histograms
* wip adapt queue_manager.go to support float histograms
* address comments for metrics in queue_manager.go
* set test cases for queue manager
* use same counts for histograms and float histograms
* refactor createHistograms tests
* fix float histograms ref in watcher_test.go
* address PR comments

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-01-13 16:39:20 +05:30
Marc Tudurí 9474610baf
Support FloatHistogram in TSDB (#11522)
Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram.

Signed-off-by: Marc Tudurí <marctc@protonmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:25:07 +05:30
Julius Volz 1a2c645dfa Correctly handle error unwrapping in rules and remote write receiver
errors.Unwrap() actually dangerously returns nil if the error does not have an
Unwrap() method, which is the case in at least one of these places where I
noticed that no error was being logged at all when it should have.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2022-12-15 12:50:55 +01:00
beorn7 87351f2318 prompb: Modify layout of histograms
Note: This is deliberately an incompatible change. Since we have never
used histograms in remote read/write yet, there is no point in keeping
compatibility. This _is_, however, compatible to the state in the main
branch.

This commit flattens the bucket message into top-level fields. This
has the disadvantage of now having two triples of fields prefixed with
`negative_...` or `positive_...`. However, with this structure, we
save one tag on the wire. And, perhaps more importantly, we mirror the
structure of the `histogram.Histogram` Go type.

This commit also adjusts `repeated` fields to use names in the plural
form, as it is also the case for the fields that already existed.

This also adds a doc comment to `HistogramProtoToHistogram` and
changes its return type to a pointer (which is more convenient and
probably more efficient).

Signed-off-by: beorn7 <beorn@grafana.com>
2022-07-14 17:47:17 +02:00
Levi Harrison 08f3ddb864
Sparse histogram remote-write support (#11001) 2022-07-14 09:13:12 -04:00
Matthieu MOREL d56d0a9d52
(storage): move from github.com/pkg/errors to 'errors' and 'fmt' (#10946)
Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>

Co-authored-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>
2022-07-01 18:59:50 +02:00
Sebastian Rabenhorst d8b8678bd1
Log time series details for out-of-order samples in remote write receiver (#9894)
* Improved out-of-order sample logs in write handler

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

sign commit

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Inlined logAppendError

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Update storage/remote/write_handler.go

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

Fixed fmt

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Improved out-of-order sample logs in write handler

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

sign commit

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Inlined logAppendError

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
2021-12-08 15:07:51 +00:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Mateusz Gozdek 1a6c2283a3 Format Go source files using 'gofumpt -w -s -extra'
Part of #9557

Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
2021-11-02 19:52:34 +01:00
Serge Catudal d77c985f8c
Add initial support for exemplar to the remote write receiver endpoint (#9319)
* Add initial support for exemplar to the remote write receiver endpoint

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>

* Update storage remote write handler tests with exemplars

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>

* Update remote write handler in order to have a distinct checkAppendExemplarError function from scrape

Signed-off-by: Serge Catudal <serge.catudal@gmail.com>
2021-09-21 14:53:27 -06:00
Levi Harrison b5f6f8fb36 Switched to go-kit/log
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-11 12:28:36 -04:00
Tom Wilkie ce97cdd477
Move remote read handler to remote package. (#8536)
* Move remote read handler to remote package.

This follows the pattern I started with the remote write handler.  The api/v1 package is getting pretty cluttered.  Moving code to other packages helps reduce this size and also makes it reusable - eg Cortex doesn't do streaming remote writes yet, and will very soon.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Deal with a nil remoteReadHandler for tests.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Remove the global metrics.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Fix test.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-26 16:43:19 +00:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489)
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
fuling 47d7e3781f [fix] remote_storage : change "write_hander.go" to "write_handler.go"
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
2021-02-10 14:25:04 +08:00
Renamed from storage/remote/write_hander.go (Browse further)