prometheus/docs/migration.md

---
title: Migration
sort_rank: 10
---

# Prometheus 3.0 migration guide

In line with our [stability promise](https://prometheus.io/docs/prometheus/latest/stability/),
the Prometheus 3.0 release contains a number of backwards incompatible changes.
This document offers guidance on migrating from Prometheus 2.x to Prometheus 3.0 and newer versions.

## Flags

- The following feature flags have been removed and they have been added to the 
  default behavior of Prometheus v3:
  - `promql-at-modifier`
  - `promql-negative-offset`
  - `remote-write-receiver`
  - `new-service-discovery-manager`
  - `expand-external-labels`
    Environment variable references `${var}` or `$var` in external label values 
    are replaced according to the values of the current environment variables.  
    References to undefined variables are replaced by the empty string.
    The `$` character can be escaped by using `$$`.
  - `no-default-scrape-port`
    Prometheus v3 will no longer add ports to scrape targets according to the 
    specified scheme. Target will now appear in labels as configured.
    If you rely on scrape targets like 
    `https://example.com/metrics` or `http://exmaple.com/metrics` to be 
    represented as `https://example.com/metrics:443` and 
    `http://example.com/metrics:80` respectively, add them to your target URLs
    - `agent`
      Instead use the dedicated `--agent` cli flag.

  Prometheus v3 will log a warning if you continue to pass these to 
  `--enable-feature`.

## Configuration

- The scrape job level configuration option `scrape_classic_histograms` has been 
  renamed to `always_scrape_classic_histograms`. If you use the 
  `--enable-feature=native-histograms` feature flag to ingest native histograms 
  and you also want to ingest classic histograms that an endpoint might expose 
  along with native histograms, be sure to add this configuration or change your 
  configuration from the old name.
- The `http_config.enable_http2` in `remote_write` items default has been 
  changed to `false`. In Prometheus v2 the remote write http client would 
  default to use http2. In order to parallelize multiple remote write queues 
  across multiple sockets its preferable to not default to http2.
  If you prefer to use http2 for remote write you must now set 
  `http_config.enable_http2: true` in your `remote_write` configuration section.

## PromQL

- The `.` pattern in regular expressions in PromQL matches newline characters. 
  With this change a regular expressions like `.*` matches strings that include 
  `\n`. This applies to matchers in queries and relabel configs. For example the 
  following regular expressions now match the accompanying strings, wheras in 
  Prometheus v2 these combinations didn't match.

| Regex      | Additional matches  |
| -----      | ------              |
| ".*"       | "foo\n", "Foo\nBar" |
| "foo.?bar" | "foo\nbar"          |
| "foo.+bar" | "foo\nbar"          |

  If you want Prometheus v3 to behave like v2 did, you will have to change your 
  regular expressions by replacing all `.` patterns with `[^\n]`, e.g.  
  `foo[^\n]*`.
- Lookback and range selectors are left open and right closed (previously left 
  closed and right closed). This change affects queries when the evaluation time 
  perfectly aligns with the sample timestamps. For example assume querying a 
  timeseries with even spaced samples exactly 1 minute apart. Before Prometheus 
  3.x, range query with `5m` will mostly return 5 samples. But if the query 
  evaluation aligns perfectly with a scrape, it would return 6 samples. In 
  Prometheus 3.x queries like this will always return 5 samples.
  This change has likely few effects for everyday use, except for some sub query 
  use cases.
  Query front-ends that align queries usually align sub-queries to multiples of 
  the step size. These sub queries will likely be affected.
  Tests are more likely to affected. To fix those either adjust the expected 
  number of samples or extend to range by less then one sample interval.
- The `holt_winters` function has been renamed to `double_exponential_smoothing` 
  and is now guarded by the `promql-experimental-functions` feature flag.
  If you want to keep using holt_winters, you have to do both of these things:
    - Rename holt_winters to double_exponential_smoothing in your queries.
    - Pass `--enable-feature=promql-experimental-functions` in your Prometheus 
      cli invocation..

## Scrape protocols
Prometheus v3 is more strict concerning the Content-Type header received when
scraping. Prometheus v2 would default to the standard Prometheus text protocol
if the target being scraped did not specify a Content-Type header or if the
header was unparsable or unrecognised. This could lead to incorrect data being
parsed in the scrape. Prometheus v3 will now fail the scrape in such cases.

If a scrape target is not providing the correct Content-Type header the
fallback protocol can be specified using the fallback_scrape_protocol
parameter. See [Prometheus scrape_config documentation.](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)

This is a breaking change as scrapes that may have succeeded with Prometheus v2
may now fail if this fallback protocol is not specified.

## Miscellaneous

### TSDB format and downgrade
The TSDB format has been changed in Prometheus v2.55 in preparation for changes 
to the index format. Consequently a Prometheus v3 tsdb can only be read by a 
Prometheus v2.55 or newer.
Before upgrading to Prometheus v3 please upgrade to v2.55 first and confirm 
Prometheus works as expected. Only then continue with the upgrade to v3.

### TSDB Storage contract
TSDB compatible storage is now expected to return results matching the specified 
selectors. This might impact some third party implementations, most likely 
implementing `remote_read`.
This contract is not explicitly enforced, but can cause undefined behavior.

### UTF-8 names
Prometheus v3 supports UTF-8 in metric and label names. This means metric and
label names can change after upgrading according to what is exposed by
endpoints. Furthermore, metric and label names that would have previously been
flagged as invalid no longer will be.

Users wishing to preserve the original validation behavior can update their
prometheus yaml configuration to specify the legacy validation scheme:

```
global:
  metric_name_validation_scheme: legacy
```

Or on a per-scrape basis:

```
scrape_configs:
  - job_name: job1
    metric_name_validation_scheme: utf8
  - job_name: job2
    metric_name_validation_scheme: legacy
```

### Log message format
Prometheus v3 has adopted `log/slog` over the previous `go-kit/log`. This 
results in a change of log message format. An example of the old log format is:
```
ts=2024-10-23T22:01:06.074Z caller=main.go:627 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2024-10-23T22:01:06.074Z caller=main.go:671 level=info msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=91d80252c3e528728b0f88d254dd720f6be07cb8-modified)"
ts=2024-10-23T22:01:06.074Z caller=main.go:676 level=info build_context="(go=go1.23.0, platform=linux/amd64, user=, date=, tags=unknown)"
ts=2024-10-23T22:01:06.074Z caller=main.go:677 level=info host_details="(Linux 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 gigafips (none))"
```

a similar sequence in the new log format looks like this:
```
time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/user/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:640 msg="No time or size retention was set so using the default time retention" duration=15d
time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/user/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:681 msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=7c7116fea8343795cae6da42960cacd0207a2af8)"
time=2024-10-24T00:03:07.542+02:00 level=INFO source=/home/user/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:686 msg="operational information" build_context="(go=go1.23.0, platform=linux/amd64, user=, date=, tags=unknown)" host_details="(Linux 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 gigafips (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
```

### `le` and `quantile` label values
In Prometheus v3, the values of the `le` label of classic histograms and the 
`quantile` label of summaries are normalized upon ingestions. In Prometheus v2 
the value of these labels depended on the scrape protocol (protobuf vs text 
format) in some situations. This led to label values changing based on the 
scrape protocol. E.g. a metric exposed as `my_classic_hist{le="1"}` would be 
ingested as `my_classic_hist{le="1"}` via the text format, but as 
`my_classic_hist{le="1.0"}` via protobuf. This changed the identity of the 
metric and caused problems when querying the metric.
In Prometheus v3 these label values will always be normalized to a float like 
representation. I.e. the above example will always result in 
`my_classic_hist{le="1.0"}` being ingested into prometheus, no matter via which 
protocol. The effect of this change is that alerts, recording rules and 
dashboards that directly reference label values as whole numbers such as 
`le="1"` will stop working.

Ways to deal with this change either globally or on a per metric basis:

- Fix references to integer `le`, `quantile` label values, but otherwise do
nothing and accept that some queries that span the transition time will produce
inaccurate or unexpected results.
_This is the recommended solution._
- Use `metric_relabel_config` to retain the old labels when scraping targets.
This should **only** be applied to metrics that currently produce such labels.

```yaml
    metric_relabel_configs:
      - source_labels:
          - quantile
        target_label: quantile
        regex: (\d+)\.0+
      - source_labels:
          - le
          - __name__
        target_label: le
        regex: (\d+)\.0+;.*_bucket
```

# Prometheus 2.0 migration guide

For the Prometheus 1.8 to 2.0 please refer to the [Prometheus v2.55 documentation](https://prometheus.io/docs/prometheus/2.55/migration/).