Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: 节点通过tiup正常缩容后Prometheus依然告警问题
[Test Environment for TiDB] Testing
[TiDB Version] v6.4.0
[Encountered Issue: Issue Phenomenon and Impact]
After completing the deployment of a new cluster using tiup (the new cluster includes alertmanager, Prometheus, etc.), and after scaling in/out the TiDB instance using tiup, Prometheus still reports alerts for the downscaled nodes.
[Attachment: Screenshot]
Check the template file .tiup/storage/cluster/clusters/${clustername}/config-cache/run_prometheus_{ip}_{port}.sh to see if it still contains information about the nodes that have been scaled down. If it does, delete that information and then restart Prometheus.
At that time, the audit log for the scale-in operation should still be there, right? Please post it for us to see. TiUP should update the Prometheus targets once.
tiup cluster audit <ID>
Check the Prometheus configuration file to confirm whether the newly added TiDB instance has been correctly added.
I encountered an alert from node_exporter after scaling down the cluster nodes before. You should reload Prometheus, and it should be fine.
Was there any interruption during the process of scaling up or down? In theory, tiup should also automatically update the monitoring components.