TiCDC can only synchronize incremental data to Kafka. How should the previous full data be handled?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC只能同步增量数据到kafka。。之前的全量数据该如何处理

| username: TiDBer_HErMeXDz

TiCDC can only synchronize incremental data to Kafka. How should the previous full data be handled?

| username: 像风一样的男子 | Original post link

Wouldn’t it be better to use tools like Dumpling to export the data in CSV format for downstream?

| username: TiDBer_HErMeXDz | Original post link

Is there an existing tool for downstream heterogeneous databases (ES) without R&D participation and without developing tools?

| username: tidb菜鸟一只 | Original post link

If there aren’t many tables, you can try using DataX.

| username: TiDBer_嘎嘣脆 | Original post link

Export, import, and then record the TS.

| username: 有猫万事足 | Original post link

Exporting dumpling to CSV, just refer to this article.

| username: zhanggame1 | Original post link

There are quite a few data synchronization tools that support heterogeneous environments.

| username: Hacker_PtIIxHC1 | Original post link

Use BR or Dumpling for full export and import.

| username: DBAER | Original post link

DataX is pretty good.

| username: TiDBer_QYr0vohO | Original post link

Use BR for full import.

| username: TiDBer_vJGTQABF | Original post link

If it’s isomorphic, you can consider using CloudCanal.

| username: Soysauce520 | Original post link

It seems like you want to do heterogeneous migration and replace TiDB :rofl:

| username: changpeng75 | Original post link

Kafka cannot replace TiDB. If you are switching to another database, you don’t need Kafka as an intermediary.

| username: Hacker_QGgM2nks | Original post link

Use dumpling or other tools to initialize the data.

| username: xiaoqiao | Original post link

First, perform a full export to the target database, then use CDC for incremental updates.

| username: 健康的腰间盘 | Original post link

CC or DataX

| username: TiDBer_fbU009vH | Original post link

You can consider using DataX. I have done historical data migration for enterprise-level databases. For TiDB, you can use DataX’s MySQL plugin, as TiDB supports MySQL statements.

| username: RyanHowe | Original post link

Use BR for full backup, and there will be a backup timestamp during the backup. Then, when creating the CDC synchronization task, specify this timestamp as the start-ts to continue data synchronization from the time of the full backup.

| username: miya | Original post link

It mainly depends on the amount of data. If the data volume is small, you can also update a specific field, such as the update time, to trigger data synchronization.

| username: TIDB-Learner | Original post link

BR, etc.