Forced Offline Down Node, Alarm Still Reported, How to Eliminate

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 强制下线Down节点，告警仍然报出来，如何消除

| username: 扬仔_tidb

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Symptoms and Impact]
[Resource Configuration]

image
image1380×164 64.9 KB

There was a CDC node previously deployed, and the machine has been released.
Executed command to force offline: tiup cluster xxx scale-in ip:8300 --force
Now the cluster is in this state, but Prometheus still alerts about Ping and node loss for this machine. How can I eliminate this alert?
image
image926×301 103 KB

image
image1380×702 83.4 KB

2
| username: tidb菜鸟一只 | Original post link

You need to restart Prometheus.

3
| username: 扬仔_tidb | Original post link

After forcing it offline, I restarted it this way.

4
| username: tidb菜鸟一只 | Original post link

Check if there is corresponding node information under /u01/tidb-deploy/prometheus-9090/conf/prometheus.yml. Also, check if it is present in tiup cluster edit-config. If not, reload Prometheus and see if the configuration file gets refreshed.

5
| username: 小龙虾爱大龙虾 | Original post link

You need to use tiup cluster reload tidb-xxx -R prometheus because this will refresh the configuration file.

6
| username: dba远航 | Original post link

You need to manually execute tiup cluster reload tidb-xxx -R prometheus to take it offline.

7
| username: Kongdom | Original post link

Manually execute the reload command to reload the configuration

tiup cluster reload tidb-xxx -R prometheus

8
| username: swino | Original post link

Try reloading it.

9
| username: TiDB_C罗 | Original post link

I simply added silence in alertmanager.

10
| username: andone | Original post link

Filter the alert levels.