Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: DM增加一个过滤规则需要重启任务吗？可以在线增加吗？
After DM is deployed, when using gh-ost to add an index online on a remote MySQL, synchronization fails. According to the documentation, you can add the parameter online-ddl, which will convert gh-ost operations into create index. I want to add this parameter and then add a filter to exclude create index. Can this parameter be added online?
online-ddl: true
sql-pattern: [“^CREATE\s+DEFINER”, “^DROP\s+EVENT”, “^CREATE\s+INDEX”]
This is the modification to the task. It cannot be added online.
However, when you stop the task, the binlog position recorded in dm_meta is still there.
So for the same task, you can stop the task, make the modifications, and then start the task again.
If the task name remains unchanged, it should continue to sync.
This is equivalent to starting the sync from the binlog position recorded in dm_meta.
In my task-name, the binlog is already specified for a certain time. If it stops at the start, won’t it start syncing from the specified date, causing duplication? According to you, it will start from the recorded position. So, what if I delete the binlog information in the task? Will it automatically find and sync from dm_meta?
So how to change it? No need to restart. The DM version is 5.3.0.
Will this method work for me? Export the configuration, modify it, and then import it.
Many tools have not been personally used.
I haven’t tried it, but this approach imports and exports the data source together. I feel the operation might be even more significant than the one mentioned above.
What are you concerned about? Are you worried about long downtime for DM? Data inconsistency?
Even when using DM OpenAPI to update tasks, there are still the following considerations.
Note
When updating the synchronization task configuration, you must ensure that the task is in a paused state, has reached the incremental synchronization stage, and only certain fields can be updated.
I’m afraid that updating will resynchronize the previously written pos’ value, or that stopping and then starting might cause data loss. It has already been stopped twice recently, affecting business twice. 
There are no issues within three minutes. According to what you said, stopping and then starting might cause data loss.
The content in the dm_meta database will remain unchanged. As long as you do not forcibly specify the binlog position in the task configuration, you do not need to worry about this.
If it was originally there and you don’t modify it, there shouldn’t be any problem. Or should you delete these two lines?
To be on the safe side, delete it. All task configurations on my side are for full synchronization.
Without deleting the metadata table corresponding to the task in the dm_meta database, restarting the task will start from the incremental position where it was stopped. It will not go through the full synchronization process again.
I haven’t tried incremental, but I feel you don’t need to specify it.
My current setup is already specified. The original DM has crashed, and a new setup has been built on a new server. It is specified to start from the binlog and pos values where the previous DM crashed. According to your suggestion, if I stop it, should I delete the specified values? Or should I leave them as is? I feel like they need to be deleted.
What you wrote is very difficult for me to understand.
The original DM has already crashed, and a new setup was built on the new server.
Did the DM-worker crash, and you expanded with a new worker? Or did you directly establish a new DM cluster, changing everything from master to worker?
Previously, the dm-worker crashed, and the old version was 1.0.6, which couldn’t be fixed. A new set of servers was installed with DM version 5.3, including new master and worker. In the task configuration, the binlog and pos values from the time of the previous crash were added:
- source-id: "db31"
block-allow-list: "bw-rule-1"
filter-rules: ["filter1"]
meta:
binlog-name: "mysql-db31-bin.019590"
binlog-pos: 436561655
Check if there is a {taskname}_syncer_checkpoint table under dm_meta. Open it and see if there are records similar to the ones above.
If there are, you don’t need to specify the binlog position in the task configuration.
If not, then you need to.
There are several tables with binlog information corresponding to the task name. So, does “not needed” mean that I should delete the previously written binlog information if I want it to start successfully? I’ll deploy another test setup to try it out.
Then it’s not necessary. Anyway, my configuration has always been full updates. As long as these tables are still there, stopping the task and then starting it again won’t cause it to start from full synchronization; it will start from incremental synchronization.
Okay. Mine is not full, so it’s hard to estimate. Based on yours, I definitely need to delete the binlog information configuration after stop.
I’ll try both solutions.
One is to modify the configuration file. Stop and then start.
The other is an export and import. I’ll try both and see the effects.
Thanks, brother.
|
username: 舞动梦灵
|
username: 
|
username: 
|
username: