mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-15 01:54:06 -08:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity
prometheus/scripts/remotewrite11-bench/README.md
Nicolás Pazos 07e21cbba3
Remote Write 1.1: e2e benchmarks (#13102) 
* Remote Write e2e benchmarks

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>

* Prometheus ports automatically assigned

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>

* make dashboard editable + more modular to different job label values

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Dashboard improvements

* memory stats
* diffs look at counter increases

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>

* run script: absolute path for config templates

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>

* grafana dashboard improvements

* show actual values of metrics
* add memory stats and diff

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>

* dashboard changes

Signed-off-by: Callum Styan <callumstyan@gmail.com>

---------

Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Callum Styan <callumstyan@gmail.com>
2023-11-08 11:21:25 -08:00

1.5 KiB
Raw Blame History

e2e Remote Write benchmark

This benchmark's purpose is to compare different versions of the remote write protocol. It runs multiple pairs of sender=>receiver prometheus instances and has the senders scrape an entire port-forwarded remote kubernetes namepsace.

Run

  1. Set envvars to port forward pods:

    export CONTEXT=my-k8-context
export NAMESPACE=my-namespace

    If desired, tweak the INSTANCES variable in the run.sh script.

  2. Run

./run.sh

Profiles

go tool pprof -seconds=240 http://localhost:9095/debug/pprof/profile
go tool pprof -seconds=240 http://localhost:9094/debug/pprof/profile

Stats

Grafana instance with provisiones datasource and dashboard

docker run --network host -v ${PWD}/local_grafana/:/etc/grafana/provisioning  --env GF_AUTH_ANONYMOUS_ENABLED=true --env GF_AUTH_ANONYMOUS_ORG_ROLE=Admin  --env GF_AUTH_BASIC_ENABLED=false --env ORG_ID=123 grafana/grafana:latest

http://localhost:9095/graph?g0.expr=sum%20by%20(job)%20(process_cpu_seconds_total%7Bjob%3D~%22(sender%7Creceiver).%2B%22%2C%7D)&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=15m

http://localhost:9095/graph?g0.expr=sum%20by%20(job)%20(prometheus_remote_storage_bytes_total%7Bjob%3D~%22(sender%7Creceiver).%2B%22%2C%7D)&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=15m

http://localhost:9095/graph?g0.expr=sum%20by%20(job)%20(go_memstats_alloc_bytes_total%7Bjob%3D~%22(sender%7Creceiver).%2B%22%2C%7D)&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=15m