Commit graph

62 commits

Author SHA1 Message Date
Ziqi Zhao 033c9b229d mirror metrics.proto change & generate code
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2024-01-24 14:01:23 +08:00
Bryan Boreham 096ec12912 Update comment about metadata in types.proto
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Arianna Vespri ffcc8a64a9 Add unit to metric family protobuf message
Signed-off-by: Arianna Vespri <arianna.vespri@yahoo.it>
2023-12-07 14:13:20 +01:00
beorn7 0eb0ca42c5 Update “conventional histogram” → “classic histogram”
Signed-off-by: beorn7 <beorn@grafana.com>
2023-11-29 15:22:58 +01:00
beorn7 a5abd92541 prompb: Remove gogoproto.nullable) = true] from created_timestamp
In proto3, this doesn't change anything. However, since the
`CreatedTimestamp` field is generated as a pointer
(`*types.Timestamp`), we are still able to detect the unset state.
(This is in contrast to the `timestamp_ms` field, which is a plain
int64, for which we cannot enforce generation as a pointer, see
comment updated in the previous commit for future actions.)

Signed-off-by: beorn7 <beorn@grafana.com>
2023-10-19 17:59:07 +02:00
Arthur Silva Sens 4b9c19fe55
Add created timestamps to prompb (#12936)
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
2023-10-10 08:54:49 +01:00
beorn7 c58e20ad0e histogram: Identify native histograms even without observations
Native histograms without observations and with a zero threshold of
zero look the same as classic histograms in the protobuf exposition
format. According to
https://github.com/prometheus/client_golang/issues/1127 , the idea is
to add a no-op span to those histograms to mark them as native
histograms. This commit enables Prometheus to detect that no-op span
and adds a doc comment to the proto spec describing the behavior.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-26 17:42:30 +02:00
Paschalis Tsilias 8a34c43515
Update prompb/README.md
Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu>
Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com>
2023-04-26 12:57:20 +03:00
Paschalis Tsilias 55626c6911 Fix final newline
Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2023-04-26 12:44:15 +03:00
Paschalis Tsilias 93f64754c0 Remove extra line
Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2023-04-26 12:43:19 +03:00
Paschalis Tsilias 417e847662
Update prompb/README.md
Co-authored-by: Julien Pivotto <roidelapluie@o11y.eu>
Signed-off-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com>
2023-04-25 17:17:22 +03:00
Paschalis Tsilias c06c02b3b1 Fix recommended protoc version
Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2023-04-25 16:27:39 +03:00
Jeanette Tan e9a1e26ab7 Perform integer/float histogram type checking on conversions, and use a consistent method for determining integer vs float histogram
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-04-22 02:27:15 +08:00
Justin Lei c16b6a0185 Handle native histograms in remote read
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-03-09 09:13:53 -08:00
Ganesh Vernekar 36b04efc7b
Merge pull request #11947 from prometheus/beorn7/protobuf
protobuf: Mark repeated fields as non-nullable
2023-02-09 12:43:32 +05:30
beorn7 1dd53843b6 protobuf: Mark repeated fields as non-nullable
Inspired by #11935, we can do it here, too.

Also, clean up indentation.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-02-07 15:02:03 +01:00
Charles Korn 0a1de58f7e
Mark Histogram.(Positive|Negative)Spans as non-nullable.
As far as I understand it, we'd never expect to receive a nil span,
and remote.spansProtoToSpans would panic if we received a nil span.

Marking the fields as non-nullable also means the generated Golang
code doesn't use pointers for these fields, reducing allocations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-02-03 13:49:22 +11:00
Giedrius Statkevičius d1d2566055
remote/read_handler: pool input to Marshal() (#11357)
* remote/read_handler: pool input to Marshal()

Use a sync.Pool to reuse byte slices between calls to Marshal() in the
remote read handler.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* remote: add microbenchmark for remote read handler

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2022-11-15 16:29:16 +01:00
Ganesh Vernekar bde500e690
Merge pull request #11420 from jesusvazquez/jvp/update-sparsehistogram-with-main-v2
Merge branch 'main' into sparsehistogram
2022-10-12 11:45:50 +05:30
Ganesh Vernekar 507bfa46fd
Fix HistogramChunk's AtFloatHistogram()
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-10-12 10:38:13 +05:30
beorn7 430bdc9dd0 prompb: Add note about experimental state of native histograms
Signed-off-by: beorn7 <beorn@grafana.com>
2022-10-11 14:08:40 +02:00
beorn7 c40b105efd histograms: Move to new exposition protobuf format
This is an incompatible protobuf change. Instrumented targets must
include https://github.com/prometheus/client_golang/pull/1092 to make
this work.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-07-19 18:11:33 +02:00
beorn7 87351f2318 prompb: Modify layout of histograms
Note: This is deliberately an incompatible change. Since we have never
used histograms in remote read/write yet, there is no point in keeping
compatibility. This _is_, however, compatible to the state in the main
branch.

This commit flattens the bucket message into top-level fields. This
has the disadvantage of now having two triples of fields prefixed with
`negative_...` or `positive_...`. However, with this structure, we
save one tag on the wire. And, perhaps more importantly, we mirror the
structure of the `histogram.Histogram` Go type.

This commit also adjusts `repeated` fields to use names in the plural
form, as it is also the case for the fields that already existed.

This also adds a doc comment to `HistogramProtoToHistogram` and
changes its return type to a pointer (which is more convenient and
probably more efficient).

Signed-off-by: beorn7 <beorn@grafana.com>
2022-07-14 17:47:17 +02:00
beorn7 d758198e37 prompb: Update exposition protobuf to include float and gauge histograms
Signed-off-by: beorn7 <beorn@grafana.com>
2022-06-29 18:49:58 +02:00
beorn7 d9d51c565c prompb: Add histograms to remote write/read protobufs
Signed-off-by: beorn7 <beorn@grafana.com>
2022-06-29 18:01:12 +02:00
beorn7 fb92a801f6 prompb: add lint ignores for metrics.proto
Signed-off-by: beorn7 <beorn@grafana.com>
2022-06-14 13:47:09 +02:00
beorn7 3bc711e333 Merge branch 'main' into sparsehistogram 2022-05-04 13:37:13 +02:00
Paschalis Tsilias 4deb1a90d2 Amend Protobuf docstrings referring to model/ packages
Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-04-16 00:37:16 +02:00
beorn7 4210aac74a Merge branch 'main' into sparsehistogram 2022-03-22 14:47:42 +01:00
Julien Pivotto e239e3ee8b
github action: Add buf-lint, fix buf (#10307)
* github action: Add buf-lint, fix buf

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2022-02-18 00:07:12 +01:00
Rémy Léone 4cc18b1109
add support for buf (#10112)
* add support for buf

Co-authored-by: Alex McKinney <alexmckinney01@gmail.com>
Signed-off-by: Rémy Léone <rleone@scaleway.com>
2022-02-16 18:30:10 +01:00
Björn Rabenstein df3b674f01
Add metrics.proto to other protobuf code (#8996)
We cannot just use prometheus/client_model directly because we want to
stay consistent with the use of gogo-protobuf. So this converts
metrics.proto to proto3 and edits it lightly so that it fits into
the framework how prometheus/prometheus handles protobuf.

Note that metrics.proto couldn't be merged into the prompb package
because prompb already has an Exemplar type, which is different from
the Exemplar type in metrics.proto. The directory structure seems to
play a role in the protobuf world, so I better kept it.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-06-28 11:16:06 +01:00
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
Simon Pasquier cf269b7ddd Update github.com/gogo/protobuf to v1.3.2
Fix for CVE-2021-3121

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2021-02-08 17:30:46 +01:00
gotjosh 4eca4dffb8
Allow metric metadata to be propagated via Remote Write. (#6815)
* Introduce a metadata watcher

Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage.

Signed-off-by: gotjosh <josue@grafana.com>

* Additional fixes after rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Rework samples/metadata metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use more descriptive variable names in MetadataWatcher collect.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix issues caused during rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix missing metric add and unneeded config code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix metrics and docs

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Replace assert with require

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Bring back max_samples_per_send metric

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-11-19 20:53:03 +05:30
Julien Pivotto e208afcc95
web: Remove APIv2 (#7935)
* web: Remove APIv2

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-09-15 09:30:55 +02:00
Luke Chen 7de6046d1e
update the protoc version to 3.12.3 in readme.md (#7931)
Signed-off-by: Luke Chen <showuon@gmail.com>
2020-09-14 09:08:27 +01:00
Bartlomiej Plotka e6d7cc5fa4
tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069)
* tsdb: Added ChunkQueryable implementations to db; unified compactor, querier and fanout block iterating.

Chained to https://github.com/prometheus/prometheus/pull/7059

* NewMerge(Chunk)Querier now takies multiple primaries allowing tsdb DB code to use it.
* Added single SeriesEntry / ChunkEntry for all series implementations.
* Unified all vertical, and non vertical for compact and querying to single
merge series / chunk sets by reusing VerticalSeriesMergeFunc for overlapping algorithm (same logic as before)
* Added block (Base/Chunk/)Querier for block querying. We then use populateAndTomb(Base/Chunk/) to iterate over chunks or samples.
* Refactored endpoint tests and querier tests to include subtests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments from Brian and Beorn.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed snapshot test and added chunk iterator support for DBReadOnly.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed race when iterating over Ats first.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed populate block tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed endpoints test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added test & fixed case of head open chunk.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed DBReadOnly tests and bug producing 1 sample chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added cases for partial block overlap for multiple full chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added extra tests for chunk meta after compaction.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed small vertical merge bug and added more tests for that.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-07-31 16:03:02 +01:00
Ganesh Vernekar c729df3d0f
Update dependencies (#7331)
* Update dependencies

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Update protos

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-06-04 11:25:56 +05:30
Julien Pivotto ef63d8d16d Update vendors
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-02-25 10:33:41 +01:00
Brian Brazil 38d32e0686 Don't sort postings if we only have one block.
Sorting the heads postings can be quite slow.
We only need sorted series when merging with another
querier, so only sort then.
This will make big queries that only touch the head faster,
though queries that touch both the head and a block will still
be the same speed. This probably won't help much with graphing
unless the range is under an hour, however it should make most
recording rules faster.

Add gaurantee that remote read streaming produces sorted series.

PromQL benchmarks for histograms show only 2-3% improvement, but
they're only over 1k series.

benchmark                                                old ns/op      new ns/op      delta
BenchmarkQuerierSelect/Head/1of1000000-4                 1375486282     507657736      -63.09%
BenchmarkQuerierSelect/Head/10of1000000-4                1387859004     507769850      -63.41%
BenchmarkQuerierSelect/Head/100of1000000-4               1387087935     506029110      -63.52%
BenchmarkQuerierSelect/Head/1000of1000000-4              1386869064     504521986      -63.62%
BenchmarkQuerierSelect/Head/10000of1000000-4             1386213685     505210422      -63.55%
BenchmarkQuerierSelect/Head/100000of1000000-4            1392754988     529842406      -61.96%
BenchmarkQuerierSelect/Head/1000000of1000000-4           1569414722     725059506      -53.80%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           1381019902     1370495863     -0.76%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          1375696209     1366789468     -0.65%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         1386009422     1364519297     -1.55%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        1377700532     1364486191     -0.96%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       1383539536     1369545314     -1.01%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      1410089163     1394731339     -1.09%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     1634744148     1581554956     -3.25%
BenchmarkQuerierSelect/Block/1of1000000-4                881741242      879839470      -0.22%
BenchmarkQuerierSelect/Block/10of1000000-4               880381562      882846038      +0.28%
BenchmarkQuerierSelect/Block/100of1000000-4              887519357      881016916      -0.73%
BenchmarkQuerierSelect/Block/1000of1000000-4             902194205      883433524      -2.08%
BenchmarkQuerierSelect/Block/10000of1000000-4            892321964      885130170      -0.81%
BenchmarkQuerierSelect/Block/100000of1000000-4           938604466      933527150      -0.54%
BenchmarkQuerierSelect/Block/1000000of1000000-4          1313510845     1295881124     -1.34%

benchmark                                                old allocs     new allocs     delta
BenchmarkQuerierSelect/Head/1of1000000-4                 4000056        4000018        -0.00%
BenchmarkQuerierSelect/Head/10of1000000-4                4000074        4000036        -0.00%
BenchmarkQuerierSelect/Head/100of1000000-4               4000254        4000216        -0.00%
BenchmarkQuerierSelect/Head/1000of1000000-4              4002054        4002016        -0.00%
BenchmarkQuerierSelect/Head/10000of1000000-4             4020054        4020016        -0.00%
BenchmarkQuerierSelect/Head/100000of1000000-4            4200054        4200016        -0.00%
BenchmarkQuerierSelect/Head/1000000of1000000-4           6000054        6000016        -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           4000071        4000071        +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          4000089        4000089        +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         4000269        4000269        +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        4002069        4002069        +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       4020069        4020069        +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      4200069        4200069        +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     6000069        6000069        +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4                6000023        6000022        -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4               6000059        6000058        -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4              6000419        6000418        -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4             6004019        6004018        -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4            6040019        6040018        -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4           6400019        6400018        -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4          10000020       10000019       -0.00%

benchmark                                                old bytes     new bytes     delta
BenchmarkQuerierSelect/Head/1of1000000-4                 229192200     176001176     -23.21%
BenchmarkQuerierSelect/Head/10of1000000-4                229193352     176002328     -23.21%
BenchmarkQuerierSelect/Head/100of1000000-4               229204872     176013848     -23.21%
BenchmarkQuerierSelect/Head/1000of1000000-4              229320072     176129048     -23.20%
BenchmarkQuerierSelect/Head/10000of1000000-4             230472072     177281048     -23.08%
BenchmarkQuerierSelect/Head/100000of1000000-4            241992072     188801048     -21.98%
BenchmarkQuerierSelect/Head/1000000of1000000-4           357192072     304001048     -14.89%
BenchmarkQuerierSelect/SortedHead/1of1000000-4           229193928     229193928     +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4          229195080     229195080     +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4         229206600     229206600     +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4        229321800     229321800     +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4       230473800     230473800     +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4      241993800     241993800     +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4     357193800     357193800     +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4                227201516     227201500     -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4               227202924     227202908     -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4              227217036     227217020     -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4             227358156     227358140     -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4            228769356     228769340     -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4           242881356     242881340     -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4          384001616     384001600     -0.00%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
Garrett 5a9c4acfbf Pushdown aggregator group by through read hint (#6401)
* Pushdown aggregator group by through read hint

Implement https://github.com/prometheus/prometheus/issues/6400

* add temporal aggregation pushdown support

Signed-off-by: xiancli <xiancli@ebay.com>
2019-12-05 14:06:28 +00:00
Bartek Płotka 48b2c9c8ea
remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703)
Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488

Changes:
* Extended protobuf for chunked remote read and negotation.
* Added checksumed, chunked Writer/Reader.
* Added Server side implementation for chunked streamed remote-read.


Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-08-19 21:16:10 +01:00
Julius Volz b5c833ca21
Update go.mod dependencies before release (#5883)
* Update go.mod dependencies before release

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Add issue for showing query warnings in promtool

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Revert json-iterator back to 1.1.6

It produced errors when marshaling Point values with special float
values.

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Fix expected step values in promtool tests after client_golang update

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Update generated protobuf code after proto dep updates

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2019-08-14 11:00:39 +02:00
Simon Pasquier 81c4248081
*: bump gRPC and protobuf dependencies (#5367)
The goal is to remove almost all references to the
golang.org/x/net/context package.

github.com/gogo/protobuf => v1.2.1
google.golang.org/grpc => v1.19.1
github.com/grpc-ecosystem/grpc-gateway => v1.18.5

It also replaces github.com/cockroachdb/cmux by github.com/soheilhy/cmux
because of [1] which fixes #3909 incidentally.

[1] https://github.com/grpc/grpc-go/issues/2636

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-04-04 11:55:32 +02:00
Callum Styan 5358f76c5c update remote write path proto so that Labels/Timeseries can't be nil (#4957)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-01-15 19:13:39 +00:00
Simon Pasquier 375ad1185c
*: bump gRPC dependencies (#5075)
* *: bump gRPC dependencies

This change updates the gRPC dependencies to more recent versions:

* github.com/gogo/protobuf => v1.2.0
* github.com/grpc-ecosystem/grpc-gateway => v1.6.3
* google.golang.org/grpc => v1.17.0

In addition scripts/genproto.sh leverages Go modules information instead of
hardcoding SHA1 commits. This ensures that the code is generated from
the exact same sources.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Run 'make proto' in CI

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Revert tabs -> spaces change

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Fix 'make proto' step

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* 'go get' grpc/protobuf dependencies

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Prepopulate cache with go mod download

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-15 15:32:05 +01:00
Frederic Branczyk 3287b9a4b8
Merge pull request #4579 from bwplotka/ensure-genproto-works
proto: Allow reproducible proto generation.
2018-11-06 15:46:56 +01:00
Bartek Plotka c3434a39b7 proto: Allow reproducible proto generation.
Current way of setting prometheus project to generate repo is tedious.
If you just go install packages described in README.md
it ends up with those (non buildable code)
```
../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
```
The issue is that the guide presented in proto folder is asking to
install some GO packages. It does not mention from what version and
there is strong mismatch between those used to generate proto and those
that are vendored and used to run the code later...
The particular outdated packages in vendor are:
`github.com/gogo/protobuf` and `google.golang.org/grpc`.

Just updating vendor would be not enough, as it would only tmp fix.
I propose adding script that will parse version from vendor.json and install correct version.

Changes:
- Added script that uses packages in *exactly* the same version the Go code is build against later.
- Regenerated proto with those pinned deps.
- Updated REAMDE.md
- Added `make proto`Current way of setting prometheus project to generate code from proto files is tedious.
If you just go install packages described in README.md, it ends up with this (non buildable code)
```
../../../prompb/remote.pb.go:61:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:109:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:164:34: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:214:27: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
../../../prompb/remote.pb.go:284:33: undefined: "github.com/prometheus/prometheus/vendor/github.com/gogo/protobuf/proto".InternalMessageInfo
```
The issue is that the guide presented in proto folder is asking to install some GO packages. It does not mention from what version and there is strong mismatch between those used to generate proto and those that are vendored and used to run the code later...
The particular outdated packages in vendor are:
`github.com/gogo/protobuf` and `google.golang.org/grpc`.

Just updating vendor would be not enough, as it would only tmp fix.
I propose adding script that will parse version from vendor.json and install correct version.

Changes:
- Added script that uses packages in *exactly* the same version the Go code is build against later.
- Regenerated proto with those pinned deps.
- Updated REAMDE.md
- Added `make proto`

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2018-10-05 10:22:02 +01:00
Tom Wilkie d3a1ff1abf
Reduce memory usage of remote read by reducing pointer usage. (#4655)
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-09-25 19:14:00 +01:00