CDC Processor Resolved TS Delay Exceeds 5 Minutes

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: cdc processor resolved ts delay more than 5 minutes

| username: li190100183

[TiDB Usage Environment] Production Environment
[TiDB Version] tidb 5.2.4
[Reproduction Path] Operations performed that led to the issue
The write speed from TiDB CDC to Kafka suddenly dropped, causing downstream synchronization delays.
[Encountered Issue: Symptoms and Impact]

cdc_resolvedts_high_delay
Status: firing, Level: critical
cdc processor resolved ts delay more than 5 minutes
cluster: tidb-order, instance: 10.220.33.74:8300, values: 1552.5, first alert: 2022-11-15T12:52:35

[2022/11/15 13:28:40.068 +08:00] [WARN] [manager.go:374] [“flush row changed events too slow”] [duration=10.314253056s] [changefeed=lucky-order-os-2-kafka]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

image
image898×396 76.5 KB

2
| username: forever | Original post link

Check if there are any long transactions.

3
| username: li190100183 | Original post link

There are no long transactions, but yesterday we upgraded from TiDB 5.0.3 to 5.2.4.

4
| username: forever | Original post link

Wait for the expert to take a look, I can’t figure this out after the upgrade :joy: