There is a data incremental synchronization requirement

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 有个数据增量同步需求

| username: zhanggame1

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 7.1
[Encountered Problem: Problem Description and Impact]
Currently, there is a requirement to synchronize two tables in real-time to a certain interface, with the requirement to only synchronize newly added data.

The development plan is to add a unique auto-increment ID column to the table and use a program to determine and fetch the incremental ID values for synchronization.

I am considering using TiCDC to Kafka, and then synchronizing to the interface. Is this feasible?

2
| username: hey-hoho | Original post link

Feasible, write a consumer program to read data from Kafka, which contains both DDL and DML types, and use them as needed for greater flexibility.

3
| username: zhanggame1 | Original post link

No need to synchronize DDL, just add new data.

4
| username: redgame | Original post link

Feasible.

5
| username: hey-hoho | Original post link

You can filter out DDL on the consumer side. You can refer to the message format of the event:

6
| username: tidb菜鸟一只 | Original post link

ticdc+kafka is very suitable for handling this.

7
| username: yulei7633 | Original post link

TiCDC is a good choice.

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.