Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: DM全量同步，怎样改为增量同步
【TiDB Usage Environment】Production Environment / Testing / PoC
【TiDB Version】V7.5
【Reproduction Path】Currently using DM to synchronize from upstream MySQL to TiDB, full synchronization is occupying disk space. Now, full synchronization is not needed, only incremental synchronization is required. Can the current synchronization task be stopped, the target end data cleared, and then start synchronizing incremental data from the current GTID position?
【Encountered Problem: Problem Phenomenon and Impact】
【Resource Configuration】Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】
Actually, you can retain the incremental synchronization logic and periodically delete the expired backup files.
You can just add an incremental synchronization task.
How to delete expired backup files? Should I directly delete the data in the table?
When adding an incremental synchronization task, should the existing data in the table be directly cleared?
I haven’t practiced it either. See if this document can help you:
TiDB Log Backup and PITR Guide | PingCAP Documentation Center
Of course, it won’t clear. Incremental synchronization will only sync the latest data.
If you only perform incremental synchronization, data inconsistencies may occur and result in errors. You still need to consider what your requirements are.
For example:
Updating a row before incremental synchronization may result in an error due to missing data.
First, there will be full synchronization, and later it will be changed to incremental synchronization. The data should theoretically be consistent.
Is your DM task configuration file set to mode: full?
- If it is full, then the full import is one-time, and after the task is completed, it will be gone. If you want to start an incremental task later, you can just start it directly.
- If it is all, then after the full import ends, it will automatically start incremental synchronization without any other operations.
DM 任务完整配置文件介绍 | PingCAP 文档中心
task-mode: all # Task mode, can be set to “full” - “only perform full data migration”, “incremental” - “Binlog real-time synchronization”, “all” - “full + Binlog real-time synchronization”
I am currently using full + incremental synchronization. To save hard disk space, I only need incremental synchronization. If I don’t clear the original data, the goal won’t be achieved, and there’s no need to change anything else.
This might indeed exist. Without the previous data, updates will indeed report errors. It depends on the requirements and how long ago the data being updated is.
Your initial approach is roughly correct.
Modify the task configuration to incremental. Stop the task, delete the target data, and then start the task again.
However, there is a problem. As someone mentioned earlier, after deleting the target-end data, if there are delete and update operations later and the target-end data does not exist, the DM task will stop here.
Enabling safe mode can solve this problem.
Okay, I’ll test it another day.
No, there should be basic data of level 8.
|
username: TiDBer_OB4kHrS7
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: