TiCDC stops tasks every night at 2 AM

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TICDC 每天晚上2点就停止了任务

| username: TiDBer_wKmPLRQo

【TiDB Environment】Production
【TiDB Version】V5.3.1
【Issue Encountered】
TICDC stops the task every night at 2 AM, but the status remains normal. However, the TSO does not move and stays at 2 AM.
【Attachments】

  • Error Log
    [2022/07/06 02:00:00.824 +08:00] [INFO] [region_worker.go:243] [“single region event feed disconnected”] [changefeed=lt-replication-task] [regionID=25669] [requestID=165802] [span=“[7480000000000006ff615f720000000000fa, 7480000000000006ff615f730000000000fa)”] [checkpoint=434384142370603012] [error=“[CDC:ErrEventFeedEventError]epoch_not_match:<current_regions:<id:25963 start_key:"t\200\000\000\000\000\000\005\377~\000\000\000\000\000\000\000\370" end_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25964 store_id:4 > peers:<id:25965 store_id:5 > peers:<id:25966 store_id:7 > > current_regions:<id:25669 start_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25670 store_id:4 > peers:<id:25848 store_id:5 > peers:<id:25866 store_id:7 > > > : epoch_not_match:<current_regions:<id:25963 start_key:"t\200\000\000\000\000\000\005\377~\000\000\000\000\000\000\000\370" end_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25964 store_id:4 > peers:<id:25965 store_id:5 > peers:<id:25966 store_id:7 > > current_regions:<id:25669 start_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25670 store_id:4 > peers:<id:25848 store_id:5 > peers:<id:25866 store_id:7 > > > “]
    [2022/07/06 02:00:00.824 +08:00] [INFO] [region_worker.go:243] [“single region event feed disconnected”] [changefeed=lt-replication-task] [regionID=25669] [requestID=165804] [span=”[7480000000000006ff745f720000000000fa, 7480000000000006ff745f730000000000fa)”] [checkpoint=434384142370603012] [error="[CDC:ErrEventFeedEventError]epoch_not_match:<current_regions:<id:25963 start_key:"t\200\000\000\000\000\000\005\377~\000\000\000\000\000\000\000\370" end_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25964 store_id:4 > peers:<id:25965 store_id:5 > peers:<id:25966 store_id:7 > > current_regions:<id:25669 start_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25670 store_id:4 > peers:<id:25848 store_id:5 > peers:<id:25866 store_id:7 > > > : epoch_not_match:<current_regions:<id:25963 start_key:"t\200\000\000\000\000\000\005\377~\000\000\000\000\000\000\000\370" end_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25964 store_id:4 > peers:<id:25965 store_id:5 > peers:<id:25966 store_id:7 > > current_regions:<id:25669 start_key:"t\200\000\000\000\000\000\006\377\233\000\000\000\000\000\000\000\370" region_epoch:<conf_ver:2279 version:844 > peers:<id:25670 store_id:4 > peers:<id:25848 store_id:5 > peers:<id:25866 store_id:7 > > > "]
2
| username: HACK | Original post link

Has the downstream data been synchronized? Is there any scheduled task in the upstream environment?

3
| username: cs58_dba | Original post link

Generally, when encountering this situation, you need to check if there were any large IO reads and writes at that time, as there might be other data extraction threads.

4
| username: xuexiaogang | Original post link

Is the source database running normally? Are there any locks or waiting events?

5
| username: Meditator | Original post link

It feels like there is some scheduled task that is increasing the load on certain nodes.

6
| username: wfxxh | Original post link

I encountered a similar issue before, and it was because of a large deletion that took a very long time to synchronize.

7
| username: Tank001 | Original post link

Are there long transactions causing deadlocks?

8
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.