A CDC synchronization task has been removed but is still triggering alerts

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: cdc某一个同步任务已经remove,还在报警

| username: 路在何chu

[TiDB Usage Environment] Production Environment 4013
[Reproduction Path] What operations were performed when the issue occurred
Constantly alarming
Warning: cdc_checkpoint_high_delay

cluster: tidb-nova-prod instance: 10.115.27.112: 8300 values: 10422.803999900818 status: firing start_time: 2024-01-16 12:05:33 +08:00 end_time: 0001-01-01 08:05:43 +08:05
[Encountered Issue: Issue Phenomenon and Impact]
The task was removed at 12 o’clock
The monitoring is still alarming

| username: 小龙虾爱大龙虾 | Original post link

It is recommended to suppress the alert first, and it will be fine once the data retention period has passed. This changefeed should no longer have relevant monitoring metrics, but the alert expression will continue to trigger because it queries an instant vector.
Alert expression: ticdc_owner_checkpoint_ts_lag > 600

| username: 路在何chu | Original post link

Oh, then we can only wait for him to clean it up himself.

| username: wangccsy | Original post link

Is the memory buffer not completed?

| username: yiduoyunQ | Original post link

For issues with lower versions of ticdc, you can manually clean up Prometheus metrics using the API method.

| username: dba远航 | Original post link

Try restarting it.