Commit graph

30 commits

Author SHA1 Message Date
Tom Wilkie d3a1ff1abf
Reduce memory usage of remote read by reducing pointer usage. (#4655)
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-09-25 19:14:00 +01:00
beorn7 4fb59d1e61 Remove use of deprecated prometheus.Handler
Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-09-17 13:05:43 +02:00
Julius Volz 8fbe1b5133
Handle a bunch of unchecked errors (#4461)
There are many more (mostly finalizers like Close/Stop/etc.), but most of
the others seemed like one couldn't do much about them anyway.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-08-17 17:24:35 +02:00
Harsh Agarwal 6a464ae174 expose log.level for promlog for remote_storage_adapter (#4195)
* expose log.level for promlog for remote_storage_adapter

Signed-off-by: sipian <cs15btech11019@iith.ac.in>

* replace flag description

Signed-off-by: Harsh Agarwal <cs15btech11019@iith.ac.in>
2018-07-22 16:11:38 +05:30
Peter Gallerani a9d5034add Fix missing 'msg' in remote storage adapter main.go .Log info message (#4377)
Signed-off-by: Peter Gallerani <peter.gallerani@gmail.com>
2018-07-12 20:54:21 +02:00
ferhat elmas ffa673f7d8 General simplifications (#3887)
Another try as in #1516
2018-02-26 07:58:10 +00:00
Matthew Pound e6dcc72f9a Fix instructions on updating prometheus.yml for Remote Write Adapter Example (#3422) 2017-11-07 20:02:43 +00:00
Krasi Georgiev 5d8f93a22a now using only github.com/gogo/protobuf
bumped all grpc-gateway packages to v1.2.2
updated and run  the denproto.sh script
2017-11-02 11:31:57 +00:00
Julius Volz 099df0c5f0 Migrate "golang.org/x/net/context" -> "context" (#3333)
In some places, where ctxhttp or gRPC are concerned, we still need to use the
old contexts.
2017-10-24 21:21:42 -07:00
Fabian Reinartz abf7c975c9 Merge branch 'master' into dev-2.0 2017-10-07 13:37:21 +02:00
Jack Neely 128b31d058 Log failure to send NaN values to remote store as Debug (#3235)
This was a warning and can be a frequent occurrence.  Let's not fill up
logs unless we are asked to.
2017-10-06 11:22:55 +01:00
Goutham Veeramachaneni 3f0267c548 Merge branch 'dev-2.0' into go-kit/log
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-15 23:15:27 +05:30
Goutham Veeramachaneni f5aed810f9 logging: Port to common/promlog
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-15 12:40:50 +05:30
Fabian Reinartz d21f149745 *: migrate to go-kit/log 2017-09-08 22:01:51 +05:30
Fabian Reinartz 87918f3097 Merge branch 'master' into dev-2.0 2017-09-04 14:09:21 +02:00
Julius Volz aa5cdcb11e Remove extra space in log output 2017-08-29 15:24:00 +02:00
gdmello 35c952e344 Added logging for remote storage adapter (#3106)
* Added logging for remote storage adapter on startup and on any error condition during /read or /write.

* CR feedback.
2017-08-29 15:22:56 +02:00
Tom Wilkie cf105f9d57 Update example remote adapters for change in proto location. 2017-07-19 16:39:02 +01:00
Julius Volz e0f046396a Fix InfluxDB retention policy usage in read adapter (#2781) 2017-05-29 16:24:24 +02:00
Tom Wilkie 3141a6b36b Compress remote storage requests and responses with unframed/raw snappy. (#2696)
* Compress remote storage requests and responses with unframed/raw snappy, for compatibility with other languages.

* Remove backwards compatibility code from remote_storage_adapter, update example_write_adapter

* Add /documentation/examples/remote_storage/example_write_adapter/example_writer_adapter to .gitignore
2017-05-10 16:42:59 +02:00
Brian Brazil 0e0fc5a7f4 Correct example name to adapter. (#2590) 2017-04-10 17:24:53 +01:00
Brian Brazil c813c824d4 Separate out remote read responses.
Fixes #2574
2017-04-06 15:49:47 +01:00
Julius Volz 3581057ea4 Update remote storage bridge README.md 2017-04-03 01:42:49 +02:00
Julius Volz b391cbb808 Add InfluxDB read-back support to remote storage bridge 2017-04-03 01:42:43 +02:00
Julius Volz 815762a4ad Move retrieval.NewHTTPClient -> httputil.NewClientFromConfig 2017-03-20 14:17:04 +01:00
Julius Volz beb3c4b389 Remove legacy remote storage implementations
This removes legacy support for specific remote storage systems in favor
of only offering the generic remote write protocol. An example bridge
application that translates from the generic protocol to each of those
legacy backends is still provided at:

documentation/examples/remote_storage/remote_storage_bridge

See also https://github.com/prometheus/prometheus/issues/10

The next step in the plan is to re-add support for multiple remote
storages.
2017-02-14 17:52:05 +01:00
Julius Volz b16371595d Add standalone remote storage bridge example
In preparation for removing specific remote storage implementations,
this offers an example of how to achieve the same in a separate process.
Rather than having three separate bridges for OpenTSDB, InfluxDB, and
Graphite, I decided to support all in one binary.

For now, this is in the example documenation directory, but perhaps we
will want to make a first-class project / repository out of it.
2017-02-01 13:22:41 +01:00
Julius Volz b5163351bf Simplify and fix remote write example
After removing gRPC, this can be simplified again. Also, the
configuration for the remote storage moved from flags to the config
file.
2016-10-05 17:53:01 +02:00
Tom Wilkie d83879210c Switch back to protos over HTTP, instead of GRPC.
My aim is to support the new grpc generic write path in Frankenstein.  On the surface this seems easy - however I've hit a number of problems that make me think it might be better to not use grpc just yet.

The explanation of the problems requires a little background.  At weave, traffic to frankenstein need to go through a couple of services first, for SSL and to be authenticated.  So traffic goes:

    internet -> frontend -> authfe -> frankenstein

- The frontend is Nginx, and adds/removes SSL.  Its done this way for legacy reasons, so the certs can be managed in one place, although eventually we imagine we'll merge it with authfe.  All traffic from frontend is sent to authfe.
- Authfe checks the auth tokens / cookie etc and then picks the service to forward the RPC to.
- Frankenstein accepts the reads and does the right thing with them.

First problem I hit was Nginx won't proxy http2 requests - it can accept them, but all calls downstream are http1 (see https://trac.nginx.org/nginx/ticket/923).  This wasn't such a big deal, so it now looks like:

    internet --(grpc/http2)--> frontend --(grpc/http1)--> authfe --(grpc/http1)--> frankenstein

Next problem was golang grpc server won't accept http1 requests (see https://groups.google.com/forum/#!topic/grpc-io/JnjCYGPMUms).  It is possible to link a grpc server in with a normal go http mux, as long as the mux server is serving over SSL, as the golang http client & server won't do http2 over anything other than an SSL connection.  This would require making all our service to service comms SSL.  So I had a go a writing a grpc http1 server, and got pretty far.  But is was a bit of a mess.

So finally I thought I'd make a separate grpc frontend for this, running in parallel with the frontend/authfe combo on a different port - and first up I'd need a grpc reverse proxy.  Ideally we'd have some nice, generic reverse proxy that only knew about a map from service names -> downstream service, and didn't need to decode & re-encode every request as it went through.  It seems like this can't be done with golang's grpc library - see https://github.com/mwitkow/grpc-proxy/issues/1.

And then I was surprised to find you can't do grpc from browsers! See http://www.grpc.io/faq/ - not important to us, but I'm starting to question why we decided to use grpc in the first place?

It would seem we could have most of the benefits of grpc with protos over HTTP, and this wouldn't preclude moving to grpc when its a bit more mature?  In fact, the grcp FAQ even admits as much:

> Why is gRPC better than any binary blob over HTTP/2?
> This is largely what gRPC is on the wire.
2016-09-15 23:21:54 +01:00
Julius Volz aa3f2b7216 Generic write cleanups and changes.
- fold metric name into labels
- return initialization errors back to main
- add snappy compression
- better context handling
- pre-allocation of labels
- remove generic naming
- other cleanups
2016-08-30 17:24:48 +02:00