Error in Ticdc Synchronization to Kafka

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Ticdc同步到kafka数据报错

| username: jingyesi3401

[TiDB Usage Environment] Production Environment / Testing

[TiDB Version] v5.1.0

[Encountered Issues]

  1. Ticdc synchronization to Kafka reports the following error (Figure 1), but Kafka can receive the data (two tasks were created, one normal and one abnormal). However, after running for a while, the error message appears as shown in Figure 2.
    Figure 1:

Figure 2:

  1. The created synchronization task is synchronized to 6 partitions, but the data received by the Kafka end only goes into one partition. Is there a problem with the command? Or does the configuration file need to add worker-num = 16?
    tiup ctl:v5.1.0 cdc changefeed create --pd=“http://x.x.x.x:2379” --sink-uri=“kafka://y.y.y.y:9092,y.y.y.y:9092,y.y.y.y:9092/gotion_gdmo_battery_produce_result_data?&partition-num=6&protocol=canal-json” --changefeed-id=“produce-kafka-changefeed” --config cdc-config_produce.toml

tiup ctl:v5.1.0 cdc changefeed create --pd=“http://x.x.x.x:2379” --sink-uri=“kafka://y.y.y.y:9092,y.y.y.y:9092,y.y.y.y:9092/gotion_gdmo_battery_produce_in_result_data?&partition-num=6&protocol=canal-json” --changefeed-id=“producein-kafka-changefeed” --config cdc-config_produce_in.toml

Configuration file:
[tidb@formal-tiup ~]$ cat cdc-config_produce.toml
rules = [‘xx.produ…’]


| username: xfworld | Original post link

Question 1:
In the first image, it only shows that the message push failed…
In the second image, it shows a network connection timeout. You need to check if there were any network issues at the time the problem occurred or if the Kafka service was functioning properly.

Question 2:
The rules for Kafka push are handled according to the configuration process.自定义-kafka-sink-的-topic-和-partition-的分发规则

It is recommended to check the distribution rules to see if they are configured according to the documentation and if they meet the requirements of the scenario.

| username: jingyesi3401 | Original post link

I modified the push rules as follows and encountered an error. My version is 5.1, and I tried to configure it according to 6.1.

The official documentation for version 5.1 does not mention the ability to write topic information. Is it not supported?
TiCDC Maintenance Operations and Task Management | PingCAP Docs

| username: xfworld | Original post link

For version 5.1, you can only configure multiple sinks to distribute to different topics.

Topic distribution configuration is supported only after version 6.x…

| username: jingyesi3401 | Original post link

What we need is to distribute the data to different partitions of the topic, not to multiple topics.

| username: xfworld | Original post link

That should be able to meet your needs, but the features will have different performances and results for different processing procedures.

You can refer to this:

| username: jingyesi3401 | Original post link

After repeated testing, the CDC synchronization is now normal. The main issue was with the configuration file. The correct configuration for version 5.1.0 is as follows: