Compatibility of TiCDC and TiDB Binlog Synchronization Formats to Kafka

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC和tidb Binlog 同步到kafka格式兼容性

| username: TiDBer_yUoxD0vR

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Problem Phenomenon and Impact]
Previously, we used TiDB Binlog to synchronize data to Kafka. After upgrading to TiDB 6.1.6, we need to use TiCDC to synchronize to Kafka. The message protocols for TiCDC output to Kafka include canal-json, open-protocol, canal, avro, and maxwell. Which one is the same as the TiDB Binlog format? Can the development team use it without changing the code?
The development team mentioned that when TiDB Binlog synchronized to Kafka, they consumed Kafka using PingCAP’s Go package in bytes format.
When I use the kafka-console-consumer.sh command line to consume, it looks different from the TiCDC output.

Kafka consumption result when TiDB Binlog to Kafka:

TiCDC canal-json format to Kafka:


TiCDC open-protocol format to Kafka:

TiCDC canal format to Kafka:

TiCDC maxwell format to Kafka:

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: xfworld | Original post link

Well, it won’t be the same…

It is recommended to use the open-protocol.
You can compare the differences between the events output by binlog and ticdc in the official documentation and make adjustments accordingly.

Binlog documentation:

Ticdc documentation:

| username: TiDBer_yUoxD0vR | Original post link

Are there any examples of upgrading from TiDB Binlog to TiCDC? Since TiDB 5, it is not recommended to use TiDB Binlog anymore. Replacing TiDB Binlog with TiCDC should be a common requirement, right?

| username: xfworld | Original post link

I don’t think so…

Binlog can be output to a file, and it is mostly used for file backups.

TiCDC, on the other hand, is used more for real-time scenarios, such as:

  • Downstream same-source database
  • Data subscription triggers
  • Multi-center remote locations
| username: TiDBer_yUoxD0vR | Original post link

We use TiDB Binlog to synchronize to Kafka and downstream TiDB.
The older version of TiDB doesn’t have TiCDC. If we upgrade TiDB, this incompatibility makes it difficult to upgrade.

| username: xfworld | Original post link

Not compatible…

There are also differences between major versions of TiDB, so make sure to conduct POC and environment testing before upgrading. This will make your upgrade plan more comprehensive.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.