Experts, look here! DM synchronization error occurs after enabling GTID upstream

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 大佬们,看过来!!!DM同步,上游开启GTID之后发生报错

| username: TiDBer_STGGd1J1

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.51
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachment: Screenshot/Log/Monitoring]
The original data source did not have GTID enabled. After enabling it and reconfiguring the data source with GTID enabled, an error occurred.

| username: WalterWj | Original post link

The link you provided appears to be a specific topic on the AskTUG forum. Unfortunately, I cannot access external content directly. Please provide the text you need translated, and I will translate it for you.

| username: TiDBer_STGGd1J1 | Original post link

My operation here is to first stop all tasks on the current data source, then destroy the data source, modify the data source configuration, restart the data source, and then start the tasks.

| username: TiDBer_STGGd1J1 | Original post link

The error message indicates an abnormal GTID encoding.

| username: dba-kit | Original post link

It depends on the steps you took to enable GTID upstream. Check how that string of all zeros GTID was generated.

| username: dba-kit | Original post link

When enabling GTID in MySQL, there are three stages, two of which seem to be compatibility modes, which may result in abnormal GTID formats. (You can use mysqlbinlog to parse the upstream MySQL binlog to see what the GTID actually is.)
I suggest that DM does not enable GTID initially, and then switch to GTID mode after this period of changing GTID has passed.

| username: TiDBer_STGGd1J1 | Original post link

I have now operated to leave the gtid column in syncer_checkpoint empty and then restarted the task. There is no such error now, but will this behavior cause data inaccuracies?

| username: DBAER | Original post link

How did you generate this GTID with -000? It’s amazing.

| username: chris-zhang | Original post link

Is the generated GTID format abnormal? Use mysqlbinlog to parse the binlog and see what the GTID actually is?

| username: 小于同学 | Original post link

The generated GTID format is abnormal, right?

| username: kelvin | Original post link

GTID encoding anomaly

| username: 这里介绍不了我 | Original post link

Amazing, this string of zeros.

| username: TiDBer_rvITcue9 | Original post link

Marking this to keep an eye on it.

| username: TiDBer_STGGd1J1 | Original post link

The task is synchronizing normally. The source undergoes a master-slave switch, changing from the original master data source to the backup data source. After stopping the task, shutting down the data source, modifying the data source configuration, enabling GTID, and restarting the task, it is generated.

| username: knull | Original post link

Suspect it’s the default value.
Because it wasn’t enabled at the beginning, the checkpoint didn’t have a GTID;
After enabling it later, since the checkpoint hasn’t been updated, the value obtained is the default value (empty), so it’s all zeros.

| username: knull | Original post link

Suggestion: You can check the logs to see what exactly caused the error.

| username: WalterWj | Original post link

@TiDBer_STGGd1J1 Is there any follow-up? For example, first disable DM GTID, let MySQL run for a while, and then enable DM GTID again?

| username: TiDBer_STGGd1J1 | Original post link

Yes, I have tried the method you mentioned, but it still reports this kind of error.

| username: TiDBer_STGGd1J1 | Original post link

The current solution is to first clear the GTID column in the sync table of the corresponding task in the dm_meta database, and then restart the task.

| username: mono | Original post link

This is indeed strange. It’s already at the sync stage. Has anyone encountered this before? If it were me, I would try to see if I could skip it. :joy: