Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: ticdc同步到kafka配置问题
TiDB 4.0.13 version
Question: Kafka limits the network packet size to 4MB, but the ticdc configuration is as follows (one task is a single table), and it still triggers [Message was too large, server rejected it to avoid allocation error.]
“max-batch-size”: “1”,
“max-message-bytes”: “1048576”
I would like to ask if the max-batch-size parameter in this document TiCDC 运维操作及任务管理 | PingCAP 归档文档站 is effective for the Maxwell protocol, because the document indicates that it is only effective for default. If it is not effective for the Maxwell protocol, is the default value 16?
Can the downstream Kafka configuration max.bytes be increased?
Based on the information you provided, the max-batch-size parameter in TiCDC is used to specify the maximum number of data changes in each Kafka message. However, according to the TiDB documentation, the max-batch-size parameter currently only takes effect when the Kafka protocol is set to open-protocol[1]. For the Maxwell protocol, the max-batch-size parameter may not be effective.
Regarding the default value, the TiDB documentation does not explicitly mention what the default value is for the Maxwell protocol. Therefore, I suggest you look for the default value settings for the Maxwell protocol in the TiDB configuration file or refer to the official Maxwell documentation for more information on the default values.
If you are encountering issues with Kafka messages being too large and getting rejected, you can try adjusting the max-batch-size parameter in TiCDC to ensure that the number of data changes in each Kafka message does not exceed Kafka’s limits.
I hope this information is helpful to you. If you have any other questions, please feel free to ask.
[1]: manage-ticdc
Kafka is the company’s global standardized configuration and does not support configuration changes for the time being, so it’s a bit awkward. Try changing the message to 256KB first.
|
username: xxxxxxxx
|
username: 
|
username: 
|
username: