Technical Consultation on Duplicate Key in DM: Core Principles of DM Causality (Conflict Detection)

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm 出现 duplicate key ,关于 DM Causality(冲突检测)核心原理的技术咨询

| username: hzc989

【TiDB Usage Environment】Disaster Recovery Environment
【TiDB Version】7.5.1
【Reproduction Path】Not yet discovered, as it is sporadic
【Encountered Problem: Phenomenon and Impact】

  1. As shown in the figure, the problem encountered is a DM error: duplicated entry.
  2. Preliminary investigation suggests that the preceding transaction 61662813 involved a batch delete, but due to an invalid connection, it was not executed, causing subsequent insertions of related row data to fail.
  3. Without discussing why the invalid connection occurred, I am curious as this seems to contradict the replication principle mentioned in the DM documentation, Causality. These conflicting rows should be grouped and executed serially, right?

Causality uses a union-find-like algorithm to classify each DML, grouping interrelated DMLs together. For the specific algorithm, refer to Parallel Execution of DML.

  1. Since it is a MySQL with very low business volume being synchronized to TiDB, our current approach is to disable concurrency and set worker-count to allow serial execution.

【Resource Configuration】N/A
【Attachments: Screenshots/Logs/Monitoring】

| username: RenlySir | Original post link

Disabling concurrency, can it solve the problem of delete failure followed by an insert error due to duplicated entry?

| username: hzc989 | Original post link

Good question, not 100% sure, but I think it should be: the previous transaction failed due to an invalid connection. In single-threaded mode, does it directly apply the following transaction without retrying?

| username: Jasper | Original post link

If safe-mode is enabled, insert will be converted to replace, so there will be no primary key conflict issues.
You can refer to the following link:

| username: hzc989 | Original post link

I know about this. But we don’t want to enable it here…

| username: redgame | Original post link

Reading the documentation might solve it.

| username: TiDBer_aaO4sU46 | Original post link

We just used safe mode.