Does DM synchronization execute in binlog order?

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm同步是否按binlog顺序执行?

| username: TiDBer_QHSxuEa1

Is DM synchronization executed in the order of binlog? I read the official documentation rather roughly, and it says that to improve QPS, some DML operations will be merged for execution. Currently, some phenomena have appeared that cannot be traced to the cause.

Phenomenon: There is a conflict between the primary key and the business primary key during DM synchronization. It has been confirmed that the downstream data is not generated downstream.
Table primary key PRI1(id), UNK(columnA, columnB);

Checking MySQL’s binlog, I found that upstream frequently executes delete and insert operations almost simultaneously in a short period.
delete(1, aa, 123);
insert(2, aa, 123);

delete(2, aa, 123);
insert(3, aa, 123);

delete(3, aa, 123);
insert(4, aa, 123);


This cycle repeated about 100 times within a minute.

| username: okenJiang | Original post link

At the very least, you should post the DM error.

| username: onlyacat | Original post link

Which version of DM?

| username: TiDBer_QHSxuEa1 | Original post link

Isn’t it just a regular unique key duplicate entry conflict error? The error itself doesn’t reveal the problem. Both skipping the binlog or using the resume-task safe mode can restart the synchronization task. What I want to know now is why there is a conflict error in the downstream synchronization when the upstream executes smoothly.

| username: TiDBer_QHSxuEa1 | Original post link

Is it viewed like this?

| username: dba远航 | Original post link

DM has parameters that can be adjusted to skip, right?

| username: TiDBer_QHSxuEa1 | Original post link

It can be skipped, but I want to know the cause. The downstream confirmed that no data was written, and the upstream MySQL was also executing normally. DM suddenly reported a primary key conflict. Is it really caused by the DM tool?

| username: Jasper | Original post link

It does not guarantee complete order, but in cases where conflicts might occur, it will ensure order. For more details, you can refer to this: Data Migration 中的 DML 同步机制 | PingCAP 文档中心
Please provide the full DM configuration file for review.

| username: 舞动梦灵 | Original post link

If you ensure there are no duplicates, directly execute resume-task. In my environment, if DM has a long delay, it often reports primary key conflicts during synchronization, even though there aren’t any. I have to manually keep executing resume. Directly resuming synchronization works fine.