Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: Ticdc同步到kafka数据报错
[TiDB Usage Environment] Production Environment / Testing
[TiDB Version] v5.1.0
[Encountered Issues]
- Ticdc synchronization to Kafka reports the following error (Figure 1), but Kafka can receive the data (two tasks were created, one normal and one abnormal). However, after running for a while, the error message appears as shown in Figure 2.
Figure 1:
Figure 2:
- The created synchronization task is synchronized to 6 partitions, but the data received by the Kafka end only goes into one partition. Is there a problem with the command? Or does the configuration file need to add worker-num = 16?
tiup ctl:v5.1.0 cdc changefeed create --pd=“http://x.x.x.x:2379” --sink-uri=“kafka://y.y.y.y:9092,y.y.y.y:9092,y.y.y.y:9092/gotion_gdmo_battery_produce_result_data?&partition-num=6&protocol=canal-json” --changefeed-id=“produce-kafka-changefeed” --config cdc-config_produce.toml
tiup ctl:v5.1.0 cdc changefeed create --pd=“http://x.x.x.x:2379” --sink-uri=“kafka://y.y.y.y:9092,y.y.y.y:9092,y.y.y.y:9092/gotion_gdmo_battery_produce_in_result_data?&partition-num=6&protocol=canal-json” --changefeed-id=“producein-kafka-changefeed” --config cdc-config_produce_in.toml
Configuration file:
[tidb@formal-tiup ~]$ cat cdc-config_produce.toml
[filter]
rules = [‘xx.produ…’]
Logs:
Question 1:
In the first image, it only shows that the message push failed…
In the second image, it shows a network connection timeout. You need to check if there were any network issues at the time the problem occurred or if the Kafka service was functioning properly.
Question 2:
The rules for Kafka push are handled according to the configuration process.
https://docs.pingcap.com/zh/tidb/stable/manage-ticdc#自定义-kafka-sink-的-topic-和-partition-的分发规则
It is recommended to check the distribution rules to see if they are configured according to the documentation and if they meet the requirements of the scenario.
I modified the push rules as follows and encountered an error. My version is 5.1, and I tried to configure it according to 6.1.
The official documentation for version 5.1 does not mention the ability to write topic information. Is it not supported?
TiCDC Maintenance Operations and Task Management | PingCAP Docs
For version 5.1, you can only configure multiple sinks to distribute to different topics.
Topic distribution configuration is supported only after version 6.x…
What we need is to distribute the data to different partitions of the topic, not to multiple topics.
That should be able to meet your needs, but the features will have different performances and results for different processing procedures.
You can refer to this:
After repeated testing, the CDC synchronization is now normal. The main issue was with the configuration file. The correct configuration for version 5.1.0 is as follows:
|
username: jingyesi3401
|
username: 
