DM Synchronization Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm同步错误

| username: TiDBer_OB4kHrS7

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] V5.3.3
[Reproduction Path] After expanding the hard disk upstream, an instance switch occurred, and the binlog log changed from mysql-bin.000450 to mysql-bin.000006.
[Encountered Problem: Problem Phenomenon and Impact] DM synchronization task terminated
[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]


Parsing mysql-bin.000448 cannot find 69352200. From the application log, it appears that mysql-bin.000448 log has already been completed, and parsing cannot find 69352200 in mysql-bin.000448, unable to skip the current error.

| username: 大飞哥online | Original post link

startlocation 69352201 gtid is 183031071
endlocation 69352200 gtid is 183031072
How did this flip :joy:

| username: TiDBer_OB4kHrS7 | Original post link

I don’t know, the source side expanded the size of a hard disk, causing an instance switch, and the binlog logs have all changed. Now, even skipping errors doesn’t work, there are a bunch of errors. Let’s just resynchronize.

| username: TiDBer_OB4kHrS7 | Original post link

Use handle-error to skip errors, there are a bunch of errors.

| username: TiDBer_OB4kHrS7 | Original post link

Sorry, I can’t assist with that.

| username: Fly-bird | Original post link

Kick this faulty node out of the cluster?

| username: TiDBer_OB4kHrS7 | Original post link

Removing the node, it can’t synchronize anymore, right? There are currently 6 tasks, all of which are either many-to-one or one-to-one, and they all report this error.

| username: TiDBer_OB4kHrS7 | Original post link

After the upstream cloud database expands its hard drive, it will verify the data and generate a verification database. Then, DM synchronization will encounter a series of issues mentioned above.

| username: TiDBer_OB4kHrS7 | Original post link

| username: TiDB_C罗 | Original post link

Create a new task configuration file, use incremental mode, and specify the file-pos

task-mode: incremental

mysql-instances:
  - source-id: "xxx"
    
    meta:
      binlog-name: mysql-bin.000448
      binlog-pos: 69352001

Check the position from this error or the table in dm_meta.

| username: 像风一样的男子 | Original post link

It is recommended to skip MySQL upstream and write directly to TiDB, which is more convenient.

| username: TiDBer_OB4kHrS7 | Original post link

Configured the incremental task and enabled it, but it didn’t report any errors or proceed with synchronization. Later, the DM task was directly recreated.

| username: TiDBer_OB4kHrS7 | Original post link

The upstream is MySQL, so it’s skipped, and DM is not needed.

| username: 有猫万事足 | Original post link

Set up a whitelist so that unplanned create/drop database/table operations are not synchronized. There’s no need to follow the upstream changes.

| username: 有猫万事足 | Original post link

The reason is that the four tables under dm_meta with the task name prefix were not deleted.

I guess you directly used dmctl stop-task to delete the original task, which will cause this issue when adding an incremental task.

The correct process is dmctl stop-task → delete the four tables under dm_meta with the task name prefix → submit the incremental task.

| username: zhanggame1 | Original post link

Reconfigure DM, as it supports both full and incremental.

| username: TiDBer_OB4kHrS7 | Original post link

The task metadata table has indeed not been deleted.

| username: mono | Original post link

Are there meta configuration items in the documentation? Please share the link!

| username: TiDB_C罗 | Original post link

mysql-instances:
  -
    source-id: "mysql-replica-01"           # Corresponds to `source-id` in source.toml
    meta:                                   # The starting position of binlog migration when `task-mode` is `incremental` and the `checkpoint` of the downstream database does not exist; if the checkpoint exists, it will be based on the `checkpoint`. If neither the `meta` item nor the `checkpoint` of the downstream database exists, the migration will start from the latest binlog position of the upstream.
      binlog-name: binlog.000001
      binlog-pos: 4
      binlog-gtid: "03fc0263-28c7-11e7-a653-6c0b84d59f30:1-7041423
| username: Hacker007 | Original post link

In theory, business tables shouldn’t have such operations. If it really exists, then manually execute it in TiDB as well.