5.1.0 TiCDC Incremental Replication from One Cluster to Another TiDB Cluster - TiCDC Not Progressing

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 5.1.0 ticdc 增量从一个集群到另一个tidb集群 ticdc不推进

| username: tug_twf

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 5.1.0
[Reproduction Path] Restore to a new instance using BR, then incremental synchronization based on the TS point causes TiCDC to get stuck
[Encountered Problem: Phenomenon and Impact]

[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]

| username: tug_twf | Original post link

Tasks cannot be paused or deleted.

| username: xfworld | Original post link

It looks a bit messy, I suggest organizing it a bit…

| username: TIDB-Learner | Original post link

Is it because the source is not configured correctly or because it hasn’t been consumed?

| username: gcworkerishungry | Original post link

Eliminate and confirm each step one by one.

| username: WalterWj | Original post link

  1. It is unexpected for a current CDC task to block other tasks. This is a known issue in version 5.1 and requires an upgrade. It is resolved in versions 6.5 and above.
  2. In cases where CDC has a large number of table IDs, initialization and scheduling are serialized, leading to a long metadata loading time for over 80,000 tables, causing new tasks to remain in initialization. Optimizations were made in versions 6.5-7.1, and the issue is completely resolved in version 7.5.

Solutions:

  1. Upgrade the cluster to a new version.
  2. If upgrading is not possible:
    1. Stop the CDC process.
    2. The current method of deleting tasks normally does not work; use etcdctl to directly delete CDC-related metadata in PD.
    3. After deletion, restart the CDC service.
    4. In the future, split the synchronization tasks, and it is recommended that a single task should not exceed 2000 table IDs.

Recommendations:

  1. For lower versions like 5.1, consider trying binlog. For CDC, it is recommended to use version 6.1+.
  2. Version 5.1 will reach EOL in June 2024. It is recommended to upgrade as soon as possible.