Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: 如何升级prometheus版本
[TiDB Usage Environment] Production Environment / Testing / Poc Production
[TiDB Version] v4.0.13
The Prometheus version is relatively old, which may affect the amount of disk space occupied by WAL and also impact the memory used for queries (mainly because of the monitoring data from TiKV).
Here are a few approaches:
- Reduce the collection frequency of TiKV.
- Use remote_read and remote_write to handle it, but large queries might still occupy local memory. Has anyone tried this?
- Use the solution mentioned in the post https://asktug.com/t/topic/512888.
- Upgrade Prometheus and test it out, but I haven’t seen any upgrade cases.
I would like to ask if there are any standalone upgrade cases, whether memory usage has significantly improved after the upgrade, and how to upgrade.
Is it similar to deploying a set by yourself?
It should be different. I just want to switch to the official Prometheus package because the official one is too old. I also want to manage it with TiUP.
TiDB has corresponding versions of Prometheus for different versions. I’m not sure if setting up an offline mode and replacing Prometheus would work.
The versions of Prometheus I have are all 2.8.1. I want to directly replace the Prometheus package under .tiup with the tar.gz package from GitHub, but I’m not sure if it will have any impact.
I don’t know how the comparison is done online, but you can give it a try. Anyway, the monitoring can be scaled down, and then you can set up a new one.
Upgrading TiDB to version 6.5.0 is also possible, and the corresponding Prometheus version is 2.27.1.
I just don’t want to upgrade the whole system. The changes are quite significant.
I’ll try directly replacing the compressed package later.
I have used this article to migrate the monitoring component to another machine, hope it helps you as well (I did not migrate historical data, basically just scaled down first, then scaled up)
Thank you for the reply. I understand how to migrate, but I’m not quite sure what other methods can be used to reduce Prometheus’s memory usage.