mudhorn/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2024-11-14 09:34:05 -08:00
Code Issues Actions 8 Packages Projects Releases Wiki Activity
8600 commits 248 branches 440 tags 656 MiB
Commit graph

44 commits

Author SHA1 Message Date
Chris Marchbanks bedcd88343
Compress records before checking segment size (#8501) 
Right now a new segment might be created unnecessarily if the
uncompressed record would not fit, but after compression (typically
reducing record size in half) it would.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2021-02-17 18:45:49 +05:30
Li Zhijian 9602933db2
Cleanup tempfiles in tests (#8179) 
* test: cleanup tempdir for TestBlockWriter

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>

* test: cleanup tempdir for TestLogPartialWrite

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
 2021-01-15 13:49:57 +05:30
Marco Pracucci 63be30dcee
Fixed WAL corruption on partial writes within a page (#8125) 
* Fixed WAL corruption on partial writes

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed variable

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-10-29 16:07:03 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122) 
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-29 09:43:23 +00:00
Bartlomiej Plotka 3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. (#8066) 
* MultiError: Refactored MultiError for more concise and safe usage.

* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename package to merrors. (: In different PR.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed review comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix after rebase.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-28 15:24:58 +00:00
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110) 
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-22 11:00:08 +02:00
Brian Brazil 073e93c768
Gracefully handle unknown WAL record types. (#8004) 
As we're looking to expand what's in the WAL,
having old Prometheus servers ignore the new record types
rather than treating them as corruption allows for better
upgrade/downgrade paths.

Adjust some tests accordingly, so they're still testing what they're
meant to test.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-05 10:09:59 +01:00
zhulongcheng 5ea6bdca1f
tsdb: sync temporary checkpoint directory before rename (#7004) 
Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>
 2020-09-07 20:04:49 +05:30
Ganesh Vernekar 2255b6f62f
Refactor WAL.Segments method to be part of the wal package (#6477) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-09-01 14:46:57 +05:30
gastonqiu 5a7d398d19
Chore: Log segment number when segment read failed (#7687) 
* Chore: Log segment number when segment read failed

To manually fix the WAL files, it is good to know where the corrupt
happened so we should log the segment number when the read failed.

Related Issue #7506

Signed-off-by: gaston.qiu <gaston.qiu@umbocv.com>
 2020-08-13 11:01:21 -07:00
Julien Pivotto 62805b2fe9
tsdb: test for leaks (#7566) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-07-21 10:08:06 +02:00
Zhou Hao ddedf454d0
add os.RemoveAll err verification (#7540) 
* add os.RemoveAll err verification for watcher_test

Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>

* add os.RemoveAll err verification for db_test

Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>

* add os.RemoveAll err verification for write_test

Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>

* add os.RemoveAll err verification for queue_manager_test

Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>

* tsdb/wal/watcher_test: add close operation before delete

Signed-off-by: Zhou Hao <zhouhao@cn.fujitsu.com>
 2020-07-17 11:47:32 +05:30
Ganesh Vernekar ea013343ca
Log when starting to create a checkpoint (#7581) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-07-15 19:15:37 +05:30
Marek Slabicki 8224ddec23
Capitalizing first letter of all log lines (#7043) 
Signed-off-by: Marek Slabicki <thaniri@gmail.com>
 2020-04-11 09:22:18 +01:00
Brad Walker 3348930df5
Replace fileutil.ReadDir with ioutil.ReadDir (#7029) (#7033) 
* tsdb: Replace fileutil.ReadDir with ioutil.ReadDir (#7029)

Signed-off-by: Brad Walker <brad@bradmwalker.com>

* tsdb: Remove fileutil.ReadDir (#7029)

Signed-off-by: Brad Walker <brad@bradmwalker.com>
 2020-04-06 19:04:20 +05:30
Ben Kochie 269e7c8091
Fix golint issues. 
Signed-off-by: Ben Kochie <superq@gmail.com>
 2020-03-23 20:38:43 +01:00
Ganesh Vernekar e64a149984
Close Head in DBReadOnly.FlushWAL (#7022) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-03-23 14:49:44 +05:30
zhulongcheng e813f60fd6
tsdb: fix sequence check for WAL segments (#7032) 
Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>
 2020-03-23 13:16:28 +05:30
Julien Pivotto f1984bb007
Merge pull request #7025 from prometheus/release-2.17 
Merge release 2.17 into master
 2020-03-21 21:39:07 +01:00
Callum Styan f802f1e8ca
Fix bug with WAL watcher and Live Reader metrics usage. (#6998) 
* Fix bug with WAL watcher and Live Reader metrics usage.

Calling NewXMetrics when creating a Watcher or LiveReader results in a
registration error, which we're ignoring, and as a result other than the
first Watcher/Reader created, we had no metrics for either. So we would
only have metrics like Watcher Records Read for the first remote write
config in a users config file.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2020-03-20 17:34:15 +01:00
zhulongcheng 5f5c7a4477
tsdb: sort checkpoints by segment number (#6987) 
Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>
 2020-03-18 20:40:41 +05:30
李国忠 261cbab8e9
remove Unused parameter 'reg' in wal.Open function (#6941) 
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
 2020-03-10 11:01:47 +05:30
Julien Pivotto 52630ad0c7 Make head Postings only return series in time range 
benchmark                                                old ns/op      new ns/op      delta
BenchmarkQuerierSelect/Head/1of1000000-8                 405805161      120436132      -70.32%
BenchmarkQuerierSelect/Head/10of1000000-8                403079620      120624292      -70.07%
BenchmarkQuerierSelect/Head/100of1000000-8               404678647      120923522      -70.12%
BenchmarkQuerierSelect/Head/1000of1000000-8              403145813      118636563      -70.57%
BenchmarkQuerierSelect/Head/10000of1000000-8             405020046      125716206      -68.96%
BenchmarkQuerierSelect/Head/100000of1000000-8            426305002      175808499      -58.76%
BenchmarkQuerierSelect/Head/1000000of1000000-8           619002108      567013003      -8.40%
BenchmarkQuerierSelect/SortedHead/1of1000000-8           1276316086     120281094      -90.58%
BenchmarkQuerierSelect/SortedHead/10of1000000-8          1282631170     121836526      -90.50%
BenchmarkQuerierSelect/SortedHead/100of1000000-8         1325824787     121174967      -90.86%
BenchmarkQuerierSelect/SortedHead/1000of1000000-8        1271386268     121025117      -90.48%
BenchmarkQuerierSelect/SortedHead/10000of1000000-8       1280223345     130838948      -89.78%
BenchmarkQuerierSelect/SortedHead/100000of1000000-8      1271401620     243635515      -80.84%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-8     1360256090     1307744674     -3.86%
BenchmarkQuerierSelect/Block/1of1000000-8                748183120      707888498      -5.39%
BenchmarkQuerierSelect/Block/10of1000000-8               741084129      716317249      -3.34%
BenchmarkQuerierSelect/Block/100of1000000-8              722157273      735624256      +1.86%
BenchmarkQuerierSelect/Block/1000of1000000-8             727587744      731981838      +0.60%
BenchmarkQuerierSelect/Block/10000of1000000-8            727518578      726860308      -0.09%
BenchmarkQuerierSelect/Block/100000of1000000-8           765577046      757382386      -1.07%
BenchmarkQuerierSelect/Block/1000000of1000000-8          1126722881     1084779083     -3.72%

benchmark                                                old allocs     new allocs     delta
BenchmarkQuerierSelect/Head/1of1000000-8                 4000018        24             -100.00%
BenchmarkQuerierSelect/Head/10of1000000-8                4000036        82             -100.00%
BenchmarkQuerierSelect/Head/100of1000000-8               4000216        625            -99.98%
BenchmarkQuerierSelect/Head/1000of1000000-8              4002016        6028           -99.85%
BenchmarkQuerierSelect/Head/10000of1000000-8             4020016        60037          -98.51%
BenchmarkQuerierSelect/Head/100000of1000000-8            4200016        600047         -85.71%
BenchmarkQuerierSelect/Head/1000000of1000000-8           6000016        6000016        +0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-8           4000055        28             -100.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-8          4000073        87             -100.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-8         4000253        630            -99.98%
BenchmarkQuerierSelect/SortedHead/1000of1000000-8        4002053        6036           -99.85%
BenchmarkQuerierSelect/SortedHead/10000of1000000-8       4020053        60054          -98.51%
BenchmarkQuerierSelect/SortedHead/100000of1000000-8      4200053        600074         -85.71%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-8     6000053        6000053        +0.00%
BenchmarkQuerierSelect/Block/1of1000000-8                6000021        6000021        +0.00%
BenchmarkQuerierSelect/Block/10of1000000-8               6000057        6000057        +0.00%
BenchmarkQuerierSelect/Block/100of1000000-8              6000417        6000417        +0.00%
BenchmarkQuerierSelect/Block/1000of1000000-8             6004017        6004017        +0.00%
BenchmarkQuerierSelect/Block/10000of1000000-8            6040017        6040017        +0.00%
BenchmarkQuerierSelect/Block/100000of1000000-8           6400017        6400017        +0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-8          10000018       10000018       +0.00%

benchmark                                                old bytes     new bytes     delta
BenchmarkQuerierSelect/Head/1of1000000-8                 176001177     1392          -100.00%
BenchmarkQuerierSelect/Head/10of1000000-8                176002329     4368          -100.00%
BenchmarkQuerierSelect/Head/100of1000000-8               176013849     33520         -99.98%
BenchmarkQuerierSelect/Head/1000of1000000-8              176129056     321456        -99.82%
BenchmarkQuerierSelect/Head/10000of1000000-8             177281049     3427376       -98.07%
BenchmarkQuerierSelect/Head/100000of1000000-8            188801049     35055408      -81.43%
BenchmarkQuerierSelect/Head/1000000of1000000-8           304001059     304001049     -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-8           229192188     2488          -100.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-8          229193340     5568          -100.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-8         229204860     35536         -99.98%
BenchmarkQuerierSelect/SortedHead/1000of1000000-8        229320060     345104        -99.85%
BenchmarkQuerierSelect/SortedHead/10000of1000000-8       230472060     3894672       -98.31%
BenchmarkQuerierSelect/SortedHead/100000of1000000-8      241992060     40511632      -83.26%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-8     357192060     357192060     +0.00%
BenchmarkQuerierSelect/Block/1of1000000-8                227201516     227201506     -0.00%
BenchmarkQuerierSelect/Block/10of1000000-8               227203057     227203041     -0.00%
BenchmarkQuerierSelect/Block/100of1000000-8              227217161     227217165     +0.00%
BenchmarkQuerierSelect/Block/1000of1000000-8             227358279     227358289     +0.00%
BenchmarkQuerierSelect/Block/10000of1000000-8            228769485     228769475     -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-8           242881487     242881477     -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-8          384001705     384001705     +0.00%

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-02-20 22:41:46 +01:00
johncming c30abf1e2b tsdb/wal: remove unused argument. 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-02-15 21:54:09 +08:00
John McBride 669592a2c4 Exports metric for WAL write errors (#6647) 
* Exports metric for WAL write errors

Signed-off-by: John McBride <jpmmcbride@gmail.com>

* Correct name for counter

Signed-off-by: John McBride <jpmmcbride@gmail.com>

* Move WAL write failure to wal.go

Signed-off-by: John McBride <jpmmcbride@gmail.com>

* WAL write fail metric moved to Log for external consumers

Signed-off-by: John McBride <jpmmcbride@gmail.com>
 2020-01-17 12:12:04 -08:00
Julien Pivotto 201491fd1b Remove old checkpoint dir if it still exists (#6621) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-01-14 09:35:24 +00:00
Brian Brazil 536d416299
Fix tsdb code and tests to work on Windows. (#6547) 
Add back Windows CI, we lost it when tsdb was merged into the prometheus
repo. There's many tests failing outside tsdb, so only test tsdb for
now.

Fixes #6513

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-01-04 14:55:02 +00:00
Josh Soref 91d76c8023 Spelling (#6517) 
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-01-02 15:54:09 +01:00
Callum Styan 6a24eee340 Simplify duration check for watcher WAL replay. 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2019-11-26 16:53:11 -08:00
Callum Styan 2d3ce3916c Log the start and end of the WAL replay within the WAL watcher. 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2019-11-19 14:21:13 -08:00
Tom Wilkie de0a772b8e Port tsdb to use pkg/labels. (#6326) 
* Port tsdb to use pkg/labels.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Get tests passing.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Remove useless cast.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Appease linters.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-11-18 11:53:33 -08:00
Dipack P Panjabi ce7bab04dd Compute WAL size and use it during retention size checks (#5886) 
* Compute WAL size and account for it when applying the retention settings.

Signed-off-by: Dipack P Panjabi <dpanjabi@hudson-trading.com>
 2019-11-12 09:40:16 +07:00
Chris Marchbanks 8df4bca470
Garbage collect asynchronously in the WAL Watcher 
The WAL Watcher replays a checkpoint after it is created in order to
garbage collect series that no longer exist in the WAL. Currently the
garbage collection process is done serially with reading from the tip of
the WAL which can cause large delays in writing samples to remote
storage just after compaction occurs.

This also fixes a memory leak where dropped series are not cleaned up as
part of the SeriesReset process.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2019-10-07 14:36:10 -06:00
陈谭军 103f26d188 fix the wrong word (#6069) 
Signed-off-by: chentanjun <2799194073@qq.com>
 2019-09-30 09:54:55 -06:00
AllenZMC 1e62435960 fix wrong spells in live_reader.go (#5899) 
* fix wrong spells in live_reader.go
* fix wrong spells in lex.go

Signed-off-by: czm <zhongming.chang@daocloud.io>
 2019-09-21 16:36:33 +03:00
johncming 612f9cb361 tsdb/wal: pull out wal metrics separately as tsdb.DB (#5957) 
Signed-off-by: johncming <johncming@yahoo.com>
 2019-09-19 14:24:34 +03:00
Callum Styan 3344bb5c33 Move WAL watcher code to tsdb/wal package. (#5999) 
* Move WAL watcher code to tsdb/wal package.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix tests after moving WAL watcher code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Lint fixes.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
 2019-09-19 14:45:41 +05:30
Erfan Besharat 9336c01dfd Add methods to fetch page's buf data in tsdb WAL (#5967) 
* move the WAL page buf reset in its own func

Signed-off-by: Erfan Besharat <erbesharat@gmail.com>
 2019-08-30 11:38:36 +03:00
陈谭军 50d453b3c3 fix-up tsdb-typo (#5954) 
Signed-off-by: chentanjun <2799194073@qq.com>
 2019-08-28 14:43:02 +01:00
johncming 7d43feb03f tsdb/wal: some small refactoring for easier reading (#5930) 
Signed-off-by: johncming <johncming@yahoo.com>
 2019-08-22 16:12:59 +03:00
Bartek Plotka f0863a604e Removed extra tsdb/testutil after merge. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2019-08-14 10:12:32 +01:00
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-08-13 14:04:14 +05:30
Ganesh Vernekar 7cf09b0395
Moving tsdb into its own subdirectory 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-08-13 13:58:49 +05:30