Bartlomiej Plotka
34426766d8
Unify Iterator interfaces. All point to storage now.
...
This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.
* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.
No logic was changed, only different source of abstractions, so no need for benchmarks.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:54 +00:00
李国忠
3cd6a5b050
Storage concurrently tests and bug fix ( #6808 )
...
* Storage concurrently tests and bug fix
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
2020-02-12 08:58:52 +00:00
李国忠
40dd13b074
Storage concurrently ( #6770 )
...
* Storage concurrently
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
2020-02-11 17:19:34 +00:00
Peter Štibraný
08c5549055
Document that NewMergeSeriesSet expects individual sets to be sorted. ( #6718 )
...
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
2020-01-30 19:30:32 +05:30
Brian Brazil
38d32e0686
Don't sort postings if we only have one block.
...
Sorting the heads postings can be quite slow.
We only need sorted series when merging with another
querier, so only sort then.
This will make big queries that only touch the head faster,
though queries that touch both the head and a block will still
be the same speed. This probably won't help much with graphing
unless the range is under an hour, however it should make most
recording rules faster.
Add gaurantee that remote read streaming produces sorted series.
PromQL benchmarks for histograms show only 2-3% improvement, but
they're only over 1k series.
benchmark old ns/op new ns/op delta
BenchmarkQuerierSelect/Head/1of1000000-4 1375486282 507657736 -63.09%
BenchmarkQuerierSelect/Head/10of1000000-4 1387859004 507769850 -63.41%
BenchmarkQuerierSelect/Head/100of1000000-4 1387087935 506029110 -63.52%
BenchmarkQuerierSelect/Head/1000of1000000-4 1386869064 504521986 -63.62%
BenchmarkQuerierSelect/Head/10000of1000000-4 1386213685 505210422 -63.55%
BenchmarkQuerierSelect/Head/100000of1000000-4 1392754988 529842406 -61.96%
BenchmarkQuerierSelect/Head/1000000of1000000-4 1569414722 725059506 -53.80%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 1381019902 1370495863 -0.76%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 1375696209 1366789468 -0.65%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 1386009422 1364519297 -1.55%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 1377700532 1364486191 -0.96%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 1383539536 1369545314 -1.01%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 1410089163 1394731339 -1.09%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 1634744148 1581554956 -3.25%
BenchmarkQuerierSelect/Block/1of1000000-4 881741242 879839470 -0.22%
BenchmarkQuerierSelect/Block/10of1000000-4 880381562 882846038 +0.28%
BenchmarkQuerierSelect/Block/100of1000000-4 887519357 881016916 -0.73%
BenchmarkQuerierSelect/Block/1000of1000000-4 902194205 883433524 -2.08%
BenchmarkQuerierSelect/Block/10000of1000000-4 892321964 885130170 -0.81%
BenchmarkQuerierSelect/Block/100000of1000000-4 938604466 933527150 -0.54%
BenchmarkQuerierSelect/Block/1000000of1000000-4 1313510845 1295881124 -1.34%
benchmark old allocs new allocs delta
BenchmarkQuerierSelect/Head/1of1000000-4 4000056 4000018 -0.00%
BenchmarkQuerierSelect/Head/10of1000000-4 4000074 4000036 -0.00%
BenchmarkQuerierSelect/Head/100of1000000-4 4000254 4000216 -0.00%
BenchmarkQuerierSelect/Head/1000of1000000-4 4002054 4002016 -0.00%
BenchmarkQuerierSelect/Head/10000of1000000-4 4020054 4020016 -0.00%
BenchmarkQuerierSelect/Head/100000of1000000-4 4200054 4200016 -0.00%
BenchmarkQuerierSelect/Head/1000000of1000000-4 6000054 6000016 -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 4000071 4000071 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 4000089 4000089 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 4000269 4000269 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 4002069 4002069 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 4020069 4020069 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 4200069 4200069 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 6000069 6000069 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 6000023 6000022 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 6000059 6000058 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 6000419 6000418 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 6004019 6004018 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 6040019 6040018 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 6400019 6400018 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 10000020 10000019 -0.00%
benchmark old bytes new bytes delta
BenchmarkQuerierSelect/Head/1of1000000-4 229192200 176001176 -23.21%
BenchmarkQuerierSelect/Head/10of1000000-4 229193352 176002328 -23.21%
BenchmarkQuerierSelect/Head/100of1000000-4 229204872 176013848 -23.21%
BenchmarkQuerierSelect/Head/1000of1000000-4 229320072 176129048 -23.20%
BenchmarkQuerierSelect/Head/10000of1000000-4 230472072 177281048 -23.08%
BenchmarkQuerierSelect/Head/100000of1000000-4 241992072 188801048 -21.98%
BenchmarkQuerierSelect/Head/1000000of1000000-4 357192072 304001048 -14.89%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 229193928 229193928 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 229195080 229195080 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 229206600 229206600 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 229321800 229321800 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 230473800 230473800 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 241993800 241993800 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 357193800 357193800 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 227201516 227201500 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 227202924 227202908 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 227217036 227217020 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 227358156 227358140 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 228769356 228769340 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 242881356 242881340 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 384001616 384001600 -0.00%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
Thomas Jackson
91d7175eaa
Add storage.Warnings to LabelValues and LabelNames ( #5673 )
...
Fixes #5661
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2019-06-17 08:31:17 +01:00
Tom Wilkie
6e08029b56
Move err to be the last return value from storage.Select. ( #5054 )
...
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-01-02 11:10:13 +00:00
mknapphrt
f0e9196dca
Return warnings on a remote read fail ( #4832 )
...
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
2018-11-30 14:27:12 +00:00
Ganesh Vernekar
ca93fd544b
/api/v1/labels endpoint for getting all label names ( #4835 )
...
* vendor: update tsdb
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* /api/v1/labels endpoint
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* regex matchers for API
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add docs
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Matchers behaving as OR
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Removed the matchers
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* vendor: update tsdb using go mod
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* vendor update: tsdb
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Added LabelNames() to storage.Querier
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Test for api.labelNames
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-19 15:51:14 +05:30
Brian Brazil
78efdc6d6b
Avoid infinite loop on duplicate NaN values. ( #4275 )
...
Fixes #4254
NaNs don't equal themselves, so a duplicate NaN would
always hit the break statement and never get popped.
We should not be returning multiple data point for the same
timestamp, so don't compare values at all.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-06-18 17:34:08 +01:00
Tom Wilkie
ba418780be
Dedupe samples in the mergeIterator.
...
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2018-05-23 12:15:47 +01:00
Fabian Reinartz
7ccd4b39b8
*: implement query params
...
This adds a parameter to the storage selection interface which allows
query engine(s) to pass information about the operations surrounding a
data selection.
This can for example be used by remote storage backends to infer the
correct downsampling aggregates that need to be provided.
2018-02-13 12:17:22 +01:00
Tom Wilkie
749781edf3
Also, don't make a mergeSeriesSet if there is only one SeriesSet.
2018-01-25 11:17:16 +00:00
Tom Wilkie
48e39068bd
Don't allocate a mergeSeries if there is only one series to merge.
2018-01-25 11:11:55 +00:00
Ed Schouten
bb724f1bef
Deprecate DeduplicateSeriesSet() in favor of NewMergeSeriesSet().
...
Federation makes use of dedupedSeriesSet to merge SeriesSets for every
query into one output stream. If many match[] arguments are provided,
many dedupedSeriesSet objects will get chained. This has the downside of
causing a potential O(n*k) running time, where n is the number of series
and k the number of match[] arguments.
In the mean time, the storage package provides a mergeSeriesSet that
accomplishes the same with an O(n*log(k)) running time by making use of
a binary heap. Let's just get rid of dedupedSeriesSet and change all
existing callers to use mergeSeriesSet.
2017-12-10 19:51:20 +01:00
Fabian Reinartz
83cd270ea4
*: adapt to storage interface changes
2017-11-23 19:05:04 +01:00
Tom Wilkie
48a7a00a38
Fast path the merge querier ( #3358 )
...
* Fast path the merge querier such that it is completely removed from query path when there is no remote storage.
* Add NoopQuerier
* Add copyright notice.
* Avoid global, use a function.
2017-10-27 13:29:05 +02:00
Tom Wilkie
0e572686db
Revert "Bypass the fanout storage merging if no remote storage is configured."
2017-10-26 16:09:39 +01:00
Tom Wilkie
4bbef0ec30
Bypass the fanout storage merging if no remote storage is configured.
2017-10-23 21:34:53 +01:00
Thibault Chataigner
bf4a279a91
Remote storage reads based on oldest timestamp in primary storage ( #3129 )
...
Currently all read queries are simply pushed to remote read clients.
This is fine, except for remote storage for wich it unefficient and
make query slower even if remote read is unnecessary.
So we need instead to compare the oldest timestamp in primary/local
storage with the query range lower boundary. If the oldest timestamp
is older than the mint parameter, then there is no need for remote read.
This is an optionnal behavior per remote read client.
Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>
2017-10-18 12:08:14 +01:00
Julius Volz
9ef8518b37
Remove "package remote" garbage from license headers ( #3304 )
2017-10-17 02:26:38 +01:00
Julius Volz
f7e8348a88
Re-add contexts to storage.Storage.Querier() ( #3230 )
...
* Re-add contexts to storage.Storage.Querier()
These are needed when replacing the storage by a multi-tenant
implementation where the tenant is stored in the context.
The 1.x query interfaces already had contexts, but they got lost in 2.x.
* Convert promql.Engine to use native contexts
2017-10-04 21:04:15 +02:00
Fabian Reinartz
d21f149745
*: migrate to go-kit/log
2017-09-08 22:01:51 +05:30
Fabian Reinartz
0efecea6d4
Adapt storage APIs to uint64 references
2017-09-07 14:14:41 +02:00
Tom Wilkie
994a7f27d6
Propagate errors through mergeSeriesSet correctly.
2017-07-13 15:02:01 +01:00
Tom Wilkie
2e0d8487e3
Return zeros if At() is called after Next() returns false.
2017-07-13 14:40:29 +01:00
Tom Wilkie
014bd31a86
Remove unnecessary whitespace changes, add comment.
2017-07-13 11:26:46 +01:00
Tom Wilkie
98ac07f86a
Add unit test for the merging on the read path.
2017-07-13 11:05:38 +01:00
Tom Wilkie
96e25adc8d
Introduce 'primary' storage in fanout, and have Add return the ref from the primary.
...
Also, ensure all append batches are rolled back when a commit or rollback fails.
2017-07-12 15:51:05 +01:00
Tom Wilkie
db8128ceeb
Add label set as first parameter to AddFast, ingored by TSDB adapter.
2017-07-12 15:20:12 +01:00
Tom Wilkie
2dda5775e3
Initial port of remote storage to v2.
2017-07-12 12:27:57 +01:00
