Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: MySQL数据同步至TiDB，使用DM工具，在load阶段会大量消耗下游IO，能不能降速，或者有没有其他更好的替代方案
Synchronizing MySQL data to TiDB through DM, the load-related parameters are as follows:
loader:
pool-size: 2
dir: ./dumped_data
import-mode: sql
on-duplicate: replace
When the synchronization task reaches the Load stage, the IO of some downstream TiDB nodes is almost fully utilized:
The write traffic of the server disk is also very high:
Currently, the downstream TiDB is already running production business, and the impact of full IO is quite significant. The official documentation seems to indicate that there are no good throttling measures for the Load stage of DM. What optimizations can be done in this situation?
Is manual import done using myloader? Because the downstream is running business, during peak periods, Load needs to be paused for a while, and many import methods do not support pausing.
Additionally, DM’s all mode can handle both full and incremental imports. With manual import, you still need to handle the incremental part yourself, and there will be many systems to migrate later, which will increase the operational workload 
Adjusting the load size to 1 doesn’t work, if it still doesn’t work, you’ll have to do it manually. There’s no other way.
Please provide the version information of dm/tidb and the specific configuration of the related nodes. Also, please send the detailed configuration of dm.
Additionally, from the screenshot, are tidb/tikv running on the same node?
During the load phase, dm executes SQL statements, so there shouldn’t be such high disk IO operations. The increased IO should be network IO (writing to TiKV). Was there any index addition operation during the period shown in the screenshot?
Before version 6.2.0, DM could not configure the concurrency of load (using lightning). After version 6.2, it can be configured through pool-size. If the version is lower, it is recommended to upgrade.
dm is v6.5.0
TiDB is v6.1.2
TiDB and TiKV are deployed independently, with one TiKV server deploying 3 TiKV nodes, each node having an independent 2T NVMe disk.
No DDL operations have been performed on the downstream TiDB during this period.
Traffic on one of the TiKV servers:
IO usage rate on the same server:
The attachment is the complete configuration of DM:
dm configuration.txt (2.6 KB)
We previously found that the migration speed with DM was very slow. Using our own DTS tool was much faster. There are also many commercial data synchronization tools available that you can look into.
|
username: HHHHHHULK
|
username: 
|
username: 
|
username: