TiDB CDC Write to Kafka Configuration Compression

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Tidb cdc 写入kafka 配置压缩

| username: 表渣渣渣

[TiDB Usage Environment] Production Environment
[TiDB Version] 5.4.1

[Encountered Problem: Phenomenon and Impact]

TiDB CDC writing to Kafka without compression configured, leading to Kafka traffic reaching its limit and data consumption anomalies.

[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

image
image929×380 44.1 KB

Checked the documentation but couldn’t find the relevant configuration. After thinking it over, decided to directly test by adding the configuration (not sure which one, so added both) as follows:
&kafka.compression.type=snappy&compression.type=snappy

Not sure if it will be effective.

2
| username: 表渣渣渣 | Original post link

Observed for a day, no effect.

3
| username: Meditator | Original post link

Check the official documentation. The Kafka sink URL does not provide the compression.type parameter; it is likely hardcoded in TiCDC. You can raise an issue.

4
| username: 表渣渣渣 | Original post link

Thank you, much appreciated.

5
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.