DM Synchronization Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm同步问题

| username: mono

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.5
[Encountered Problem: Problem Phenomenon and Impact]
Synchronizing data from MySQL 5.7 to the TiDB cluster using DM. Currently, the running DM task is synchronizing some tables from the MySQL database, such as mydb.t1 and mydb.t2. Now there is a new requirement to add a table, such as mydb.t3. How can I gracefully add this table to the synchronization?

| username: forever | Original post link

Do new tables need historical full data?

| username: mono | Original post link

Yes, it is needed. My idea is to stop the task, migrate the historical data, then add the new table in the configuration, and restart the task. I wonder if there is a better way?

| username: 江湖故人 | Original post link

Isn’t it elegant enough to add a new synchronization task?

| username: Kongdom | Original post link

Are you suggesting that by using regular expressions, you can achieve a one-time setup, so that when a new table is added, there’s no need to change the configuration and it will be synchronized directly?

| username: wangccsy | Original post link

Learning, learning.

| username: dba远航 | Original post link

Stop the task. Then add the new table in the configuration. Restart the task.

| username: tidb菜鸟一只 | Original post link

If you don’t want to start a new task, you can add this table to the existing task. Alternatively, you can create a new task and specify this table separately.

| username: zzsh | Original post link


| username: Jellybean | Original post link

The approach is still base data + incremental data synchronization.

If there is no historical data to synchronize, meaning only incremental data is needed, it is relatively simple. You can directly modify the configuration in the original task, add the new tables to be synchronized, and then restart the synchronization.

If there is historical base data to synchronize, you can create a new synchronization task. In the task configuration file, enable the base data synchronization feature to achieve historical data migration, while also configuring normal real-time synchronization.

| username: 有猫万事足 | Original post link

Create a new task, don’t touch the old one.

| username: andone | Original post link

Create a new task.

| username: 麻烦是朋友 | Original post link

Create a new task.

Agree with the above +1

| username: Hacker007 | Original post link

This can only create new tasks; tasks cannot be modified.

| username: mono | Original post link

Experts, there might be too many new tasks. There will be more tables added later, which could lead to resource wastage. The process involves stopping tasks, migrating historical data, and then starting tasks again. This has already been implemented. Thank you all for your suggestions!

| username: TiDBer_QHSxuEa1 | Original post link

Here is an article for reference. While we are producing data synchronization ourselves, there are still write scenarios for upstream and downstream data. If a table is overly concentrated in one task, the entire task may be interrupted due to an error in synchronizing a single table. Therefore, we have split the number of tables to reduce the impact of task interruptions.
Column - Adding New Synchronization Tables to Existing Migration Tasks with DM | TiDB Community