After deleting drainer, alert manager is still alarming

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: drainer删除后,alert manager还一直在告警

| username: TiDBer_OB4kHrS7

[TiDB Usage Environment] Production Environment
[TiDB Version] V5.3.3
[Reproduction Path] After using tiup to delete the drainer service, there are still alarms. How can I remove the drainer service alarms?
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]
image

| username: 胡杨树旁 | Original post link

Try restarting Grafana?

| username: 啦啦啦啦啦 | Original post link

show drainer status; Check the drainer status.

| username: TiDBer_OB4kHrS7 | Original post link

I have restarted it, but the alarm still comes out.

| username: TiDBer_OB4kHrS7 | Original post link

Check the drainer status, this drainer is no longer displayed.

| username: weixiaobing | Original post link

You should reload Prometheus, as the alert data is collected from Prom.

| username: srstack | Original post link

tiup cluster reload tidb-prd-bill -N prometheus

| username: TiDBer_OB4kHrS7 | Original post link

I’ve already restarted everything, but the alerts still come up.

| username: TiDBer_OB4kHrS7 | Original post link

What is the difference between reload and restart?

| username: Rilakkuma | Original post link

Check the tiup template at .tiup/storage/cluster/clusters/${cluster_name}/config-cache/prometheus_${ip}_${port}.yml to see if the deleted drainer is still there. If it is, manually delete it and then run tiup reload prometheus.

| username: huhaifeng | Original post link

“Reload” means uploading the configuration file first, then restarting; “Restart” means just restarting.

| username: TiDBer_OB4kHrS7 | Original post link

You need to manually delete the file located at .tiup/storage/cluster/clusters/${cluster_name}/config-cache/prometheus_${ip}_${port}.yml. Additionally, you need to manually delete the file located at /data/tidb/tidb-deploy/prometheus-9090/conf/prometheus.yml. After deleting these two files, the alerts will no longer appear. Thank you all for your responses.

| username: srstack | Original post link

Just perform a reload operation.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.