How to Upgrade Prometheus Version

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何升级prometheus版本

| username: db_user

[TiDB Usage Environment] Production Environment / Testing / Poc Production
[TiDB Version] v4.0.13

The Prometheus version is relatively old, which may affect the amount of disk space occupied by WAL and also impact the memory used for queries (mainly because of the monitoring data from TiKV).

Here are a few approaches:

  1. Reduce the collection frequency of TiKV.
  2. Use remote_read and remote_write to handle it, but large queries might still occupy local memory. Has anyone tried this?
  3. Use the solution mentioned in the post https://asktug.com/t/topic/512888.
  4. Upgrade Prometheus and test it out, but I haven’t seen any upgrade cases.

I would like to ask if there are any standalone upgrade cases, whether memory usage has significantly improved after the upgrade, and how to upgrade.

2
| username: Kongdom | Original post link

Is it similar to deploying a set by yourself?

3
| username: db_user | Original post link

It should be different. I just want to switch to the official Prometheus package because the official one is too old. I also want to manage it with TiUP.

4
| username: xingzhenxiang | Original post link

TiDB has corresponding versions of Prometheus for different versions. I’m not sure if setting up an offline mode and replacing Prometheus would work.

5
| username: db_user | Original post link

The versions of Prometheus I have are all 2.8.1. I want to directly replace the Prometheus package under .tiup with the tar.gz package from GitHub, but I’m not sure if it will have any impact.

6
| username: xingzhenxiang | Original post link

I don’t know how the comparison is done online, but you can give it a try. Anyway, the monitoring can be scaled down, and then you can set up a new one.

7
| username: DBRE | Original post link

Upgrading TiDB to version 6.5.0 is also possible, and the corresponding Prometheus version is 2.27.1.

8
| username: db_user | Original post link

I just don’t want to upgrade the whole system. The changes are quite significant.

9
| username: db_user | Original post link

I’ll try directly replacing the compressed package later.

10
| username: xingzhenxiang | Original post link

I have used this article to migrate the monitoring component to another machine, hope it helps you as well (I did not migrate historical data, basically just scaled down first, then scaled up)

11
| username: db_user | Original post link

Thank you for the reply. I understand how to migrate, but I’m not quite sure what other methods can be used to reduce Prometheus’s memory usage.