Severe DM Synchronization Delay: How to Speed Up Synchronization

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: DM同步延迟严重,怎么加快同步速度

| username: Hacker_PtIIxHC1

Using DMv2.0 to synchronize data from MySQL to TiDB (v4.0.8), due to the large amount of upstream changes, the downstream TiDB experiences significant delays. How can the synchronization speed be improved?

| username: 像风一样的男子 | Original post link

The configuration file for the task has the following parameters, which can be appropriately increased based on the hardware:
syncers:
global:
worker-count: 16
batch: 100

| username: DBAER | Original post link

The documentation has a troubleshooting approach, can we refer to it?

| username: TiDBer_QYr0vohO | Original post link

It seems that your version is very low. You can check if there are any configuration options for threads.

| username: Hacker_PtIIxHC1 | Original post link

After adjusting these two parameters, it feels like there’s no change. :joy:

| username: 像风一样的男子 | Original post link

Check your server resources, such as CPU, memory, and I/O. Are they reaching their limits?

| username: tidb菜鸟一只 | Original post link

Are you using incremental or full + incremental? If the data volume is relatively large, it’s better to use dumpling + lightning for full and DM for incremental. DM can handle up to 30~50 GiB/h at most.

| username: xiaoqiao | Original post link

What is the scenario, and what kind of business is typically upstream?

| username: zhang_2023 | Original post link

Identify the bottleneck. If there are no issues with IO, CPU, memory, or physical resources, increase the number of threads.

| username: 友利奈绪 | Original post link

We need to consider network I/O, physical resources, and other issues from multiple aspects.

| username: Kamner | Original post link

Refer to the performance test report:

Additionally, it seems that your DM version is a bit outdated; you might want to upgrade it.

| username: Hacker_PtIIxHC1 | Original post link

Sure, please provide the Chinese text you need translated into English.

| username: kelvin | Original post link

Make sure that IO, CPU, memory, and physical resources are not an issue before increasing the number of threads.

| username: TiDBer_嘎嘣脆 | Original post link

Upgrade the version.

| username: 呢莫不爱吃鱼 | Original post link

Let’s upgrade first.