We cannot just use prometheus/client_model directly because we want to
stay consistent with the use of gogo-protobuf. So this converts
metrics.proto to proto3 and edits it lightly so that it fits into
the framework how prometheus/prometheus handles protobuf.
Note that metrics.proto couldn't be merged into the prompb package
because prompb already has an Exemplar type, which is different from
the Exemplar type in metrics.proto. The directory structure seems to
play a role in the protobuf world, so I better kept it.
Signed-off-by: beorn7 <beorn@grafana.com>
* Write exemplars to the WAL and send them over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Update example for exemplars, print data in a more obvious format.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add metrics for remote write of exemplars.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Fix incorrect slices passed to send in remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* We need to unregister the new metrics.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Order of exemplar append vs write exemplar to WAL needs to change.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Condense sample/exemplar delivery tests to parameterized sub-tests
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename test methods for clarity now that they also handle exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename counter variable. Fix instances where metrics were not updated correctly
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Add exemplars to LoadWAL benchmark
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* last exemplars timestamp metric needs to convert value to seconds with
ms precision
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Process exemplar records in a separate go routine when loading the WAL.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Regenerate types proto with comments, update protoc version again.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Put remote write of exemplars behind a feature flag.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address some of Ganesh's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Move exemplar remote write feature flag to a config file field.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address Bartek's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address more reivew comments from Ganesh.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add exemplar total label length check.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address a few last review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
The goal is to remove almost all references to the
golang.org/x/net/context package.
github.com/gogo/protobuf => v1.2.1
google.golang.org/grpc => v1.19.1
github.com/grpc-ecosystem/grpc-gateway => v1.18.5
It also replaces github.com/cockroachdb/cmux by github.com/soheilhy/cmux
because of [1] which fixes#3909 incidentally.
[1] https://github.com/grpc/grpc-go/issues/2636
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* *: bump gRPC dependencies
This change updates the gRPC dependencies to more recent versions:
* github.com/gogo/protobuf => v1.2.0
* github.com/grpc-ecosystem/grpc-gateway => v1.6.3
* google.golang.org/grpc => v1.17.0
In addition scripts/genproto.sh leverages Go modules information instead of
hardcoding SHA1 commits. This ensures that the code is generated from
the exact same sources.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Run 'make proto' in CI
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Revert tabs -> spaces change
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Fix 'make proto' step
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* 'go get' grpc/protobuf dependencies
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Prepopulate cache with go mod download
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Current way of setting prometheus project to generate repo is tedious.
If you just go install packages described in README.md
it ends up with those (non buildable code)
```
../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
```
The issue is that the guide presented in proto folder is asking to
install some GO packages. It does not mention from what version and
there is strong mismatch between those used to generate proto and those
that are vendored and used to run the code later...
The particular outdated packages in vendor are:
`github.com/gogo/protobuf` and `google.golang.org/grpc`.
Just updating vendor would be not enough, as it would only tmp fix.
I propose adding script that will parse version from vendor.json and install correct version.
Changes:
- Added script that uses packages in *exactly* the same version the Go code is build against later.
- Regenerated proto with those pinned deps.
- Updated REAMDE.md
- Added `make proto`Current way of setting prometheus project to generate code from proto files is tedious.
If you just go install packages described in README.md, it ends up with this (non buildable code)
```
../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
```
The issue is that the guide presented in proto folder is asking to install some GO packages. It does not mention from what version and there is strong mismatch between those used to generate proto and those that are vendored and used to run the code later...
The particular outdated packages in vendor are:
`github.com/gogo/protobuf` and `google.golang.org/grpc`.
Just updating vendor would be not enough, as it would only tmp fix.
I propose adding script that will parse version from vendor.json and install correct version.
Changes:
- Added script that uses packages in *exactly* the same version the Go code is build against later.
- Regenerated proto with those pinned deps.
- Updated REAMDE.md
- Added `make proto`
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
