After deleting the alert monitoring item and restarting Alertmanager, the alert will still be reported

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 删掉告警监控项后重启altermanager, 告警还是会上报

| username: Hacker_qeCXjAN2

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] V5.0.0
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Problem Phenomenon and Impact]
After deleting some unimportant alert configurations in the alert rule and restarting alertmanager, these alerts are still being reported. Do I need to restart other roles as well? Thank you!
[Resource Configuration]
[Attachments: Screenshots / Logs / Monitoring]

| username: 胡杨树旁 | Original post link

After modifying certain alert items, it seems that there is no need to restart alertmanager. You just need to modify the configuration file in alertmanager. This is how we operate with the 5.4.0 version of the cluster. I’m not sure if the 5.0.0 version of the cluster is the same.

| username: Fly-bird | Original post link

No need to restart.

| username: Hacker_qeCXjAN2 | Original post link

The current issue is that even after deleting certain monitoring items, the alerts are still being reported.

| username: Hacker_qeCXjAN2 | Original post link

Not deleting, just modifying the alert level will still report the same. Looking at the alertmanager log, it will automatically load the configuration file.

| username: 像风一样的男子 | Original post link

Try restarting Prometheus.

| username: chenhanneu | Original post link

Prometheus matches records according to rule rules and sends alerts to Alertmanager. Normally, after modifying the rule, manually reloading it will take effect with curl -XPOST localhost:9090/-/reload. However, the Prometheus startup script for TiDB does not configure --web.enable-lifecycle, so the process must be restarted to take effect.

| username: Hacker_qeCXjAN2 | Original post link

Thank you! I restarted Prometheus yesterday, but it didn’t take effect for a minute or two after the restart. I thought it didn’t work, but later I saw that it did.

| username: Hacker_qeCXjAN2 | Original post link

It took effect, thank you!

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.