Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: DM worker节点的高可用
[TiDB Usage Environment] Production Environment / Testing / POC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
Migrating from DataX to DM requires long-term synchronization from MySQL to TiDB. Initially, it was filtered by DB, but over time, the number of upstream DBs will increase significantly, and the number of DM workers will correspondingly increase. DM master can have three nodes to ensure its high availability, but how many free worker nodes should be reserved to ensure high availability? Suppose 20 workers correspond to 20 DBs, which may be in one MySQL instance or multiple MySQL instances. These workers are distributed across 4 physical machines, with 5 worker instances on each physical machine. To ensure that the failure of one physical machine does not affect the worker tasks, the workers on that machine need to migrate to other physical machines. So, how many free workers should be reserved on each physical machine? Is there a formula?
When the number of deployed DM-worker nodes exceeds the number of upstream MySQL/MariaDB nodes, the DM-worker nodes exceeding the number of upstream nodes will be idle by default. If a DM-worker node goes offline or experiences network isolation with the DM-master leader, DM-master can automatically schedule the data migration tasks related to the original DM-worker node to other idle DM-worker nodes (if the original DM-worker node is in a network isolation state, it will automatically stop the related data migration tasks); if there are no idle DM-worker nodes available for scheduling, the data migration tasks related to the original DM-worker will be temporarily suspended until an idle DM-worker node becomes available, at which point the tasks will automatically resume.
The number of workers is only related to the MySQL instance, and the worker is the slave of the upstream MySQL.
DM-worker is a component of DM (Data Migration) responsible for executing specific data migration tasks.
Its main functions are as follows:
* Register as a slave of a MySQL or MariaDB server.
The current situation is, assuming you have 7 upstream MySQL instances. To meet your needs, you would need 3 physical machines and 15 workers. Just ensure that the number of free workers is always greater than 5.
It is indeed necessary for free to be no less than 5.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
|
username: TiDB_C罗
|
username: 
|
username: 
|
username: 
|
username: 
|
username: