TiCDC Repeatedly Restarting

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc 反复重启

| username: GreenGuan

I encountered an issue today where all ticdc’s resolve tso stopped progressing, and the tidb logs also stopped progressing. Has anyone experienced this before? The version is v5.4.1.

The error is as follows:

[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:452] ["[etcd worker] ============State Commit============="] [committed=false]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:456] ["[etcd worker] ==========Failed Etcd Txn Cmps=========="]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc001\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc001\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc002\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc002\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc007\" mod_revision:103984752 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc007\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-alltab\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-alltab\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:VALUE key:\"/tidb/cdc/meta/ticdc-delete-etcd-key-count\" value:\"2395\" "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:462] ["[etcd worker] ============End Failed Etcd Txn Cmps============="]
[2022/09/21 11:53:32.583 +08:00] [INFO] [etcd_worker.go:444] ["[etcd worker] ==========Update State to ETCD=========="]
2
| username: xfworld | Original post link

These logs are all info, not errors.

3
| username: GreenGuan | Original post link

Uh, I checked and there are no WARN logs. I tried to remove the changefeed and encountered an error:

[2022/09/21 14:12:39.806 +08:00] [WARN] [changefeed.go:349] [“changefeed is removed, but state is not complete”] [state=“{"ID":"mediav-base-alltab","Info":null,"Status":null,"TaskPositions":{},"TaskStatuses":{},"Workloads":{}}”]

4
| username: GreenGuan | Original post link

Deleting tasks has caused some protocols to start reporting this issue:

[2022/09/21 14:37:48.698 +08:00] [PANIC] [canal.go:167] ["unsigned value not in type uint64"] [column="{\"name\":\"xxx\",\"type\":8,\"flag\":225,\"value\":null}"] [stack="github.com/pingcap/tiflow/cdc/sink/codec.getJavaSQLType\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:167\
github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildColumn\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:280\
github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildRowData\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:306\
github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).newFlatMessageForDML\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:178\
github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).AppendRowChangedEvent\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:321\
github.com/pingcap/tiflow/cdc/sink.(*mqSink).runWorker\
\tgithub.com/pingcap/tiflow/cdc/sink/mq.go:338\
github.com/pingcap/tiflow/cdc/sink.(*mqSink).run.func1\
\tgithub.com/pingcap/tiflow/cdc/sink/mq.go:267\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57"]
5
| username: HACK | Original post link

-try using -force to delete it

6
| username: GreenGuan | Original post link

Upgrading the TiDB version resolved the issue.

7
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.