Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: ticdc 到kafka同步报错问题
tidb 4.0.13
Recently, the online ticdc task encountered the following error:
The max-message-bytes for ticdc is set to 4m, and kafka max.message.bytes is also 4m, with max-batch-size using the default value (the official documentation states that this version is 16). Although the error was resolved after updating max-message-bytes to 2m, I would like to understand how the configuration parameters of ticdc limit the package size and why this error occurred.
In native Kafka parameters, the message size constraint on the producer side is max.request.size, while the scope of message.max.bytes is based on the Kafka service side.
- I would like to confirm whether the max-message-bytes parameter in the ticdc configuration corresponds to the kafka max.message.bytes (topic-level configuration) parameter.
- Does ticdc have a corresponding parameter to support limiting kafka’s max.request.size parameter?
- When configuring max-message-bytes=2097152, the result shows an opts value of 4m. What is the difference between these two?
Based on the information you provided, it can be seen that the reason for the TiCDC error is that the message size exceeds the limit of max-message-bytes. The value of the max-message-bytes parameter in TiCDC should not be greater than the max.message.bytes parameter value of the Kafka Topic. Therefore, you need to modify the TiCDC and Kafka parameter values according to the size of a single row of data to meet the above conditions.
To answer your questions:
-
Yes, the max-message-bytes parameter in the TiCDC configuration task corresponds to the max.message.bytes (topic-level configuration) parameter in Kafka. The max-message-bytes parameter in TiCDC is used to limit the message size sent to Kafka.
-
TiCDC does not have a corresponding parameter to limit Kafka’s max.request.size parameter. max.request.size is a parameter used by the Kafka producer side to constrain the message size, which TiCDC cannot directly control.
-
max-message-bytes=2097152 is the parameter value set when configuring the TiCDC task, while opts shows 4m. The difference between these two parameters lies in the unit. The unit of max-message-bytes is bytes, while opts shows the value in MB. Therefore, 4m corresponds to 4194304 bytes, which is greater than the max-message-bytes=2097152 you set.
Currently, our Kafka’s max.message.bytes is limited to 4MB. We can ensure that a single row of data will definitely not exceed 4MB. However, configuring cdc max-message-bytes to 4MB, 2MB, or 1MB all results in errors (the ticdc task involves 9 tables, and each table’s single row does not exceed 4MB, but there are changes in each table at the same time). So, I would like to ask for suggestions on how to configure ticdc to avoid this error.
The image is not visible. Please provide the text you need translated.
As long as cdc max-message-bytes * max-batch-size <= Kafka’s max.message.bytes, this error will not occur?
It seems to mean this inside.
However, in actual use, an error still occurs, which is quite frustrating.
It might be a critical state, try changing it to 3.
I have already changed it to 2, but it still reports an error. Frustrating.
|
username: xxxxxxxx
|
username: 
|
username: 