Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: dm同步是否按binlog顺序执行？
Is DM synchronization executed in the order of binlog? I read the official documentation rather roughly, and it says that to improve QPS, some DML operations will be merged for execution. Currently, some phenomena have appeared that cannot be traced to the cause.
Phenomenon: There is a conflict between the primary key and the business primary key during DM synchronization. It has been confirmed that the downstream data is not generated downstream.
Table primary key PRI1(id), UNK(columnA, columnB);
Checking MySQL’s binlog, I found that upstream frequently executes delete and insert operations almost simultaneously in a short period.
delete(1, aa, 123);
insert(2, aa, 123);
delete(2, aa, 123);
insert(3, aa, 123);
delete(3, aa, 123);
insert(4, aa, 123);
delete…
insert…
This cycle repeated about 100 times within a minute.
At the very least, you should post the DM error.
Isn’t it just a regular unique key duplicate entry conflict error? The error itself doesn’t reveal the problem. Both skipping the binlog or using the resume-task safe mode can restart the synchronization task. What I want to know now is why there is a conflict error in the downstream synchronization when the upstream executes smoothly.
DM has parameters that can be adjusted to skip, right?
It can be skipped, but I want to know the cause. The downstream confirmed that no data was written, and the upstream MySQL was also executing normally. DM suddenly reported a primary key conflict. Is it really caused by the DM tool?
It does not guarantee complete order, but in cases where conflicts might occur, it will ensure order. For more details, you can refer to this: Data Migration 中的 DML 同步机制 | PingCAP 文档中心
Please provide the full DM configuration file for review.
If you ensure there are no duplicates, directly execute resume-task. In my environment, if DM has a long delay, it often reports primary key conflicts during synchronization, even though there aren’t any. I have to manually keep executing resume. Directly resuming synchronization works fine.
|
username: TiDBer_QHSxuEa1
|
username: 
|
username: 
|
username: 
|
username: 
|
username: