Forced Offline Down Node, Alarm Still Reported, How to Eliminate

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 强制下线Down节点,告警仍然报出来,如何消除

| username: 扬仔_tidb

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5
[Reproduction Path] Operations performed that led to the issue
[Encountered Issue: Symptoms and Impact]
[Resource Configuration]


There was a CDC node previously deployed, and the machine has been released.
Executed command to force offline: tiup cluster xxx scale-in ip:8300 --force
Now the cluster is in this state, but Prometheus still alerts about Ping and node loss for this machine. How can I eliminate this alert?

| username: tidb菜鸟一只 | Original post link

You need to restart Prometheus.

| username: 扬仔_tidb | Original post link

After forcing it offline, I restarted it this way.

| username: tidb菜鸟一只 | Original post link

Check if there is corresponding node information under /u01/tidb-deploy/prometheus-9090/conf/prometheus.yml. Also, check if it is present in tiup cluster edit-config. If not, reload Prometheus and see if the configuration file gets refreshed.

| username: 小龙虾爱大龙虾 | Original post link

You need to use tiup cluster reload tidb-xxx -R prometheus because this will refresh the configuration file.

| username: dba远航 | Original post link

You need to manually execute tiup cluster reload tidb-xxx -R prometheus to take it offline.

| username: Kongdom | Original post link

Manually execute the reload command to reload the configuration

tiup cluster reload tidb-xxx -R prometheus

| username: swino | Original post link

Try reloading it.

| username: TiDB_C罗 | Original post link

I simply added silence in alertmanager.

| username: andone | Original post link

Filter the alert levels.