Prometheus Still Alerts After Node is Successfully Scaled Down via TiUP

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 节点通过tiup正常缩容后Prometheus依然告警问题

| username: TiDBer_E5BPfmPA

[Test Environment for TiDB] Testing
[TiDB Version] v6.4.0
[Encountered Issue: Issue Phenomenon and Impact]
After completing the deployment of a new cluster using tiup (the new cluster includes alertmanager, Prometheus, etc.), and after scaling in/out the TiDB instance using tiup, Prometheus still reports alerts for the downscaled nodes.

[Attachment: Screenshot]

| username: Rilakkuma | Original post link

Check the template file .tiup/storage/cluster/clusters/${clustername}/config-cache/run_prometheus_{ip}_{port}.sh to see if it still contains information about the nodes that have been scaled down. If it does, delete that information and then restart Prometheus.

| username: buptzhoutian | Original post link

At that time, the audit log for the scale-in operation should still be there, right? Please post it for us to see. TiUP should update the Prometheus targets once.
tiup cluster audit <ID>

| username: redgame | Original post link

Check the Prometheus configuration file to confirm whether the newly added TiDB instance has been correctly added.

| username: Raymond | Original post link

I encountered an alert from node_exporter after scaling down the cluster nodes before. You should reload Prometheus, and it should be fine.

| username: Jellybean | Original post link

Was there any interruption during the process of scaling up or down? In theory, tiup should also automatically update the monitoring components.