Does TiDB support migrating both full and incremental data across different databases within the same cluster?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB 支持同一个集群内,不同数据库,迁移全量以及增量数据吗

| username: TiDBer_lVKhiLUB

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issues: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: xfworld | Original post link

Full and incremental backups are currently supported by different components and cannot be handled all at once.

For full backups, you can use BR (Backup & Restore). It performs physical backups and restores, with some limitations, but it should meet your requirements.
For incremental backups, you can use TiCDC, which can satisfy distribution and customization requirements.

| username: TiDBer_lVKhiLUB | Original post link

Can DM do it? The business background is that there is already a database A in TiDB, and some data needs to be migrated from the old MySQL database to database A. To avoid impacting the production environment, we want to back up database A as database B, and any new data in database A will also be synchronized to database B. Then, migrate the MySQL data to database B. Later, we will directly use database B to replace database A.

| username: xfworld | Original post link

TiDB Data Migration (DM) is a convenient data migration tool that supports full data migration and incremental data synchronization from databases compatible with the MySQL protocol (MySQL, MariaDB, Aurora MySQL) to TiDB. Using the DM tool helps simplify the data migration process and reduce the operational costs of data migration.

The positioning is very clear… and the description is also very clear…

| username: TiDBer_lVKhiLUB | Original post link

Understood, we can only proceed with migrating the TiDB cluster to another TiDB cluster scenario.

| username: tidb菜鸟一只 | Original post link

Isn’t this just synchronizing MySQL data to TiDB? You can totally use DM for that.

| username: Kongdom | Original post link

Migration within the same cluster should not use BR, right?

| username: 胡杨树旁 | Original post link

DM from MySQL to TiDB should be feasible, but within the same TiDB cluster, migrating data from database A to database B using binlog is uncertain.

| username: 考试没答案 | Original post link

I remember it can be used. There is a filter function. You can export and import it. I tested it without any issues.

| username: Kongdom | Original post link

BR should not be able to specify the database name and can only restore to an empty database.

| username: 考试没答案 | Original post link

Are you saying this???

| username: db_user | Original post link

My understanding, which I am not sure if it’s correct:

Understanding part:
The full data of TiDB’s B database consists of the data from TiDB’s A database plus some data from MySQL.
The incremental data of TiDB’s B database consists of the incremental data from TiDB’s A database.

Method (which might be a bit cumbersome):

  1. Set a large GC time for TiDB and record the current TSO.
  2. Create an A database on MySQL and copy TiDB’s A database to MySQL’s A database.
  3. Use TiCDC to synchronize TiDB’s A database to MySQL’s A database.
  4. Use DM’s routing feature to synchronize MySQL’s A database to TiDB’s B database (set safe-mode to true initially).

However, this process is quite lengthy, and it depends on the size of the A database, as migrating it might be cumbersome.

| username: TiDBer_lVKhiLUB | Original post link

The data volume in database A is in the hundreds of millions. It’s considered relatively large.

| username: db_user | Original post link

How large is database A? If it’s within 50G, I think this method is still feasible. If it’s too large, it might be a bit slow, and it might be better to consider other methods.

| username: TiDBer_lVKhiLUB | Original post link

Database A is the production environment database with incremental business data. Database B must ensure it has all the data for easier switching later. Currently, Database A has 90GB of data.

| username: Hacker007 | Original post link

You can use DM for full and incremental synchronization of database A, and use TiCDC to synchronize from TiDB to database B.