mirror of
https://github.com/prometheus/prometheus.git
synced 2024-11-05 05:04:04 -08:00
89 lines
4.6 KiB
Markdown
89 lines
4.6 KiB
Markdown
|
# Go tracing and monitoring (Prometheus) for `net.Conn`
|
||
|
|
|
||
|
|
[](https://travis-ci.org/mwitkow/go-conntrack)
|
||
|
|
[](http://goreportcard.com/report/mwitkow/go-conntrack)
|
||
|
|
[](https://godoc.org/github.com/mwitkow/go-conntrack)
|
||
|
|
[](LICENSE)
|
||
|
|
|
||
|
|
[Prometheus](https://prometheus.io/) monitoring and [`x/net/trace`](https://godoc.org/golang.org/x/net/trace#EventLog) tracing wrappers `net.Conn`, both inbound (`net.Listener`) and outbound (`net.Dialer`).
|
||
|
|
|
||
|
|
## Why?
|
||
|
|
|
||
|
|
Go standard library does a great job of doing "the right" things with your connections: `http.Transport` pools outbound ones, and `http.Server` sets good *Keep Alive* defaults.
|
||
|
|
However, it is still easy to get it wrong, see the excellent [*The complete guide to Go net/http timeouts*](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/).
|
||
|
|
|
||
|
|
That's why you should be able to monitor (using Prometheus) how many connections your Go frontend servers have inbound, and how big are the connection pools to your backends. You should also be able to inspect your connection without `ssh` and `netstat`.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## How to use?
|
||
|
|
|
||
|
|
All of these examples can be found in [`example/server.go`](example/server.go):
|
||
|
|
|
||
|
|
### Conntrack Dialer for HTTP DefaultClient
|
||
|
|
|
||
|
|
Most often people use the default `http.DefaultClient` that uses `http.DefaultTransport`. The easiest way to make sure all your outbound connections monitored and trace is:
|
||
|
|
|
||
|
|
```go
|
||
|
|
http.DefaultTransport.(*http.Transport).DialContext = conntrack.NewDialContextFunc(
|
||
|
|
conntrack.DialWithTracing(),
|
||
|
|
conntrack.DialWithDialer(&net.Dialer{
|
||
|
|
Timeout: 30 * time.Second,
|
||
|
|
KeepAlive: 30 * time.Second,
|
||
|
|
}),
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Dialer Name
|
||
|
|
|
||
|
|
Tracked outbound connections are organised by *dialer name* (with `default` being default). The *dialer name* is used for monitoring (`dialer_name` label) and tracing (`net.ClientConn.<dialer_name>` family).
|
||
|
|
|
||
|
|
You can pass `conntrack.WithDialerName()` to `NewDialContextFunc` to set the name for the dialer. Moreover, you can set the *dialer name* per invocation of the dialer, by passing it in the `Context`. For example using the [`ctxhttp`](https://godoc.org/golang.org/x/net/context/ctxhttp) lib:
|
||
|
|
|
||
|
|
```go
|
||
|
|
callCtx := conntrack.DialNameToContext(parentCtx, "google")
|
||
|
|
ctxhttp.Get(callCtx, http.DefaultClient, "https://www.google.com")
|
||
|
|
```
|
||
|
|
|
||
|
|
### Conntrack Listener for HTTP Server
|
||
|
|
|
||
|
|
Tracked inbound connections are organised by *listener name* (with `default` being default). The *listener name* is used for monitoring (`listener_name` label) and tracing (`net.ServerConn.<listener_name>` family). For example, a simple `http.Server` can be instrumented like this:
|
||
|
|
|
||
|
|
```go
|
||
|
|
listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
|
||
|
|
listener = conntrack.NewListener(listener,
|
||
|
|
conntrack.TrackWithName("http"),
|
||
|
|
conntrack.TrackWithTracing(),
|
||
|
|
conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
|
||
|
|
httpServer.Serve(listener)
|
||
|
|
```
|
||
|
|
|
||
|
|
Note, the `TrackWithTcpKeepAlive`. The default `http.ListenAndServe` adds a tcp keep alive wrapper to inbound TCP connections. `conntrack.NewListener` allows you to do that without another layer of wrapping.
|
||
|
|
|
||
|
|
#### TLS server example
|
||
|
|
|
||
|
|
The standard lobrary `http.ListenAndServerTLS` does a lot to bootstrap TLS connections, including supporting HTTP2 negotiation. Unfortunately, that is hard to do if you want to provide your own `net.Listener`. That's why this repo comes with `connhelpers` package, which takes care of configuring `tls.Config` for that use case. Here's an example of use:
|
||
|
|
|
||
|
|
```go
|
||
|
|
listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
|
||
|
|
listener = conntrack.NewListener(listener,
|
||
|
|
conntrack.TrackWithName("https"),
|
||
|
|
conntrack.TrackWithTracing(),
|
||
|
|
conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
|
||
|
|
tlsConfig, err := connhelpers.TlsConfigForServerCerts(*tlsCertFilePath, *tlsKeyFilePath)
|
||
|
|
tlsConfig, err = connhelpers.TlsConfigWithHttp2Enabled(tlsConfig)
|
||
|
|
tlsListener := tls.NewListener(listener, tlsConfig)
|
||
|
|
httpServer.Serve(listener)
|
||
|
|
```
|
||
|
|
|
||
|
|
# Status
|
||
|
|
|
||
|
|
This code is used by Improbable's HTTP frontending and proxying stack for debuging and monitoring of established user connections.
|
||
|
|
|
||
|
|
Additional tooling will be added if needed, and contributions are welcome.
|
||
|
|
|
||
|
|
#License
|
||
|
|
|
||
|
|
`go-conntrack` is released under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.
|
||
|
|
|