Issue with ticdc synchronization to Kafka - Part 2

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc同步到kafka报错问题二

| username: xxxxxxxx

Encountering an error with TiCDC syncing to Kafka in TiDB version 4.0.13.

Background Description
The TiDB cluster needs to sync data change records to downstream Kafka via TiCDC. It is known that Kafka’s max.message.bytes configuration limits messages to 4MB. The current TiCDC configuration is as follows, but it still reports the error [kafka server: Message was too large, server rejected it to avoid allocation error]. There are 9 target tables.

      "max-batch-size": "1",
      "max-message-bytes": "1048576"

Additionally, I saw in the official documentation that this error cannot be controlled from the TiCDC side and needs to be limited from the Kafka side. However, Kafka is managed by another department, and it is known that the configuration cannot be changed. I would like to ask how to troubleshoot this issue and what other configurations can be optimized.

| username: 像风一样的男子 | Original post link

You can check Kafka to see the content and size of each message sent to identify which data is relatively large. If you can’t change the parameters on the Kafka side, you can split the CDC synchronization task into multiple synchronization tasks for execution.

| username: xxxxxxxx | Original post link

Kafka belongs to the parent company, and as a subsidiary, we cannot access it or request configuration changes.

| username: 像风一样的男子 | Original post link

You can send data to him, so you must have the Kafka address. Use this command to view the topic’s data: kafka-console-consumer.sh --bootstrap-server ip:9092 --topic topicname

| username: 像风一样的男子 | Original post link

If it really doesn’t work, you can start 9 CDC tasks and configure one task for each table.