TiCDC Abnormal State: Table Sink Stuck

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc异常状态 table sink stuck

| username: TiDBer_Jack

【TiDB Usage Environment】Production Environment
【TiDB Version】v6.5.9
【Reproduction Path】Operations that led to the issue
Kafka disk was full, causing data to not be stored in Kafka
【Encountered Issue: Problem Phenomenon and Impact】
The CDC task stopped because the disk was full, resulting in a warning error
{
“upstream_id”: 7358011227008144308,
“namespace”: “default”,
“id”: “sms-record-sync”,
“state”: “warning”,
“checkpoint_tso”: 449549867346296837,
“checkpoint_time”: “2024-05-05 16:10:49.728”,
“error”: {
“time”: “2024-05-08T11:51:07.591737751+08:00”,
“addr”: “172.22.147.102:8300”,
“code”: “CDC:ErrProcessorUnknown”,
“message”: “table sink stuck”
}
}

【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

image
image1180×675 53.3 KB

【Attachments: Screenshots/Logs/Monitoring】
The error logs related to CDC are as follows
[2024/05/08 11:55:38.178 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7269] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.161.103:20160] [storeID=2] [streamID=19852] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:480\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:654\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.178 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7269] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.145.102:20160] [storeID=3] [streamID=19886] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:480\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:654\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.178 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7269] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.131.104:20160] [storeID=11] [streamID=19897] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:300\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func3\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:651\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.178 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7269] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.131.101:20160] [storeID=10] [streamID=19869] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:480\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:654\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.178 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7269] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.145.105:20160] [storeID=1] [streamID=19833] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:480\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:654\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.236 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7140] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.131.104:20160] [storeID=11] [streamID=19707] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:300\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func3\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:651\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.236 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7140] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.131.101:20160] [storeID=10] [streamID=19690] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:300\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func3\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:651\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.237 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7140] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.145.105:20160] [storeID=1] [streamID=19670] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:300\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func3\n\tgithub.com/pingcap/tiflow/cdc/kv/region_worker.go:651\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 11:55:38.237 +08:00] [ERROR] [client.go:1068] [“region worker exited with error”] [namespace=default] [changefeed=sms-record-sync] [tableID=7140] [tableName=itnio_sms_record.tbsendrcd] [store=172.22.161.103:20160] [storeID=2] [streamID=19675] [error=“context canceled”]

2
| username: Jellybean | Original post link

Has this issue been resolved? If not, prioritize addressing this issue first, and then execute resume to restore the synchronization task.

3
| username: TiDBer_Jack | Original post link

The Kafka issue has been resolved by expanding the disk. However, the problem of the table sink getting stuck occurred when resuming.

4
| username: kkpeter | Original post link

Remember to recreate a task at the checkpoint.

5
| username: Jellybean | Original post link

Go to the ticdc log directory and check the logs for errors where “table sink stuck” appears, and obtain more detailed log information.

6
| username: TiDBer_Jack | Original post link

The logs are as follows:
[2024/05/08 17:20:39.388 +08:00] [WARN] [feed_state_manager.go:575] [“changefeed meets a warning”] [warning=“{"time":"2024-05-08T17:20:39.352566396+08:00","addr":"172.22.163.103:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:20:46.139 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"warning","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:20:39.352566396+08:00","addr":"172.22.163.103:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:20:53.538 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"warning","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:20:39.352566396+08:00","addr":"172.22.163.103:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:28:46.033 +08:00] [WARN] [manager.go:289] [“Sink manager backend sink fails”] [namespace=default] [changefeed=sms-record-sync] [factoryVersion=1] [error=“table sink stuck”] [errorVerbose=“table sink stuck\ngithub.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).GetTableStats\n\tgithub.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:1038\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).GetTableStatus\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:429\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*table).getTableStatus\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/table.go:72\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).handleMessageHeartbeat\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:269\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).handleMessage\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:249\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).Tick\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:206\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:703\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:597\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:133\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:290\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:550\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:382\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 17:28:46.437 +08:00] [WARN] [feed_state_manager.go:481] [“processor reports a warning”] [namespace=default] [changefeed=sms-record-sync] [captureID=f724604d-e235-44bb-902d-c19bc8961504] [warning=“{"time":"2024-05-08T17:28:46.267991696+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:28:46.438 +08:00] [WARN] [feed_state_manager.go:575] [“changefeed meets a warning”] [warning=“{"time":"2024-05-08T17:28:46.267991696+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:31:02.988 +08:00] [WARN] [feed_state_manager.go:481] [“processor reports a warning”] [namespace=default] [changefeed=sms-record-sync] [captureID=29f337b7-84a2-4352-aa0e-7336bd5e6a9c] [warning=“{"time":"2024-05-08T17:31:02.942752472+08:00","addr":"172.22.147.102:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:31:02.988 +08:00] [WARN] [feed_state_manager.go:575] [“changefeed meets a warning”] [warning=“{"time":"2024-05-08T17:31:02.942752472+08:00","addr":"172.22.147.102:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:35:08.705 +08:00] [WARN] [manager.go:289] [“Sink manager backend sink fails”] [namespace=default] [changefeed=sms-record-sync] [factoryVersion=2] [error=“table sink stuck”] [errorVerbose=“table sink stuck\ngithub.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).GetTableStats\n\tgithub.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:1038\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).GetTableStatus\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:429\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*table).getTableStatus\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/table.go:72\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).handleMessageHeartbeat\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:269\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).handleMessage\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:249\ngithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent.(*agent).Tick\n\tgithub.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent/agent.go:206\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:703\ngithub.com/pingcap/tiflow/cdc/processor.(*processor).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/processor.go:597\ngithub.com/pingcap/tiflow/cdc/processor.(*managerImpl).Tick\n\tgithub.com/pingcap/tiflow/cdc/processor/manager.go:133\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:290\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:550\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:382\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.5.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594”]
[2024/05/08 17:35:08.993 +08:00] [WARN] [feed_state_manager.go:481] [“processor reports a warning”] [namespace=default] [changefeed=sms-record-sync] [captureID=f724604d-e235-44bb-902d-c19bc8961504] [warning=“{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:35:08.993 +08:00] [WARN] [feed_state_manager.go:575] [“changefeed meets a warning”] [warning=“{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}”]
[2024/05/08 17:40:23.747 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"warning","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:40:27.560 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"warning","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:40:31.050 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"warning","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:40:34.448 +08:00] [INFO] [changefeed.go:723] [“changefeed closed”] [namespace=default] [changefeed=sms-record-sync] [status=“{"checkpoint-ts":449549867346296837,"min-table-barrier-ts":449558191387967491,"admin-job-type":0}”] [info=“{"upstream-id":7358011227008144308,"namespace":"default","changefeed-id":"sms-record-sync","sink-uri":"kafka://b-1.itniokafka3.1uw2xm.c10.kafka.us-west-2.amazonaws.com:9092,b-2.itniokafka3.1uw2xm.c10.kafka.us-west-2.amazonaws.com:9092,b-3.itniokafka3.1uw2xm.c10.kafka.us-west-2.amazonaws.com:9092/sms-record-sync?kafka-version=2.8.1\u0026max-message-bytes=67108864\u0026partition-num=60\u0026protocol=canal-json\u0026replication-factor=1","create-time":"2024-04-28T15:51:25.988067401+08:00","start-ts":449391017571647493,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"enable-old-value":true,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["itnio_sms_record.tbsendrcd"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"transaction-atomicity":"","protocol":"canal-json","dispatchers":[{"matcher":["itnio_sms_record.tbsendrcd"],"dispatcher":"","partition":"rowid","topic":""}],"csv":null,"column-selectors":null,"schema-registry":"","encoder-concurrency":0,"terminator":"\r\n","date-separator":"","enable-partition-separator":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"warning","error":null,"warning":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"},"creator-version":"v6.5.9","epoch":449618776523276334}”] [isRemoved=false]
[2024/05/08 17:40:36.738 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"stopped","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:40:41.588 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"stopped","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]
[2024/05/08 17:40:42.688 +08:00] [INFO] [api.go:188] [“List changefeed successfully!”] [info=“{"upstream_id":7358011227008144308,"namespace":"default","id":"sms-record-sync","state":"stopped","checkpoint_tso":449549867346296837,"checkpoint_time":"2024-05-05 16:10:49.728","error":{"time":"2024-05-08T17:35:08.896949106+08:00","addr":"172.22.133.101:8300","code":"CDC:ErrProcessorUnknown","message":"table sink stuck"}}”]

7
| username: TiDBer_Jack | Original post link

In my current state, data is still being sent to Kafka, but the checkpoint_tso position hasn’t changed at all.

8
| username: TiDBer_Jack | Original post link

I checked the logs and found an ALTER statement. The table has 900 million records. Could this be the cause of the problem?

9
| username: zhh_912 | Original post link

I suggest checking the logs.

10
| username: 小于同学 | Original post link

Very likely.

11
| username: TiDBer_Jack | Original post link

Tested and found that it’s not an issue with the alter statement. A developer executed an update on the entire table, which caused Kafka to be overwhelmed and led to subsequent issues.

12
| username: kkpeter | Original post link

That’s impressive.

13
| username: kkpeter | Original post link

It seems to be a large transaction, not refreshing because it hasn’t finished syncing.

14
| username: zhaokede | Original post link

Seeing the cause of the problem, it seems that we still need to monitor such major transactions and large DDL operations regularly.