What parameters can be used to speed up region balance migration when expanding a TiKV cluster?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv集群扩容，有什么参数可以加速region平衡迁移速度的？

| username: 扬仔_tidb

【TiDB Usage Environment】Production Environment
【TiDB Version】5.3.0
【Encountered Problem: Phenomenon and Impact】

Background:

The old TiKV server has disk issues and is planned to be smoothly replaced, so a new node is added to replace the old node.

Configuration and Current Status of New and Old Machines

The original TiKV had three nodes, each with 1.4 million regions. The newly expanded node has a configuration of 16c/32g. Currently, during the balancing process, the CPU usage of the new node is only 7%, while the CPU usage of the old TiKV nodes is 50%. At the current speed, it is estimated to take 37 days to complete the balancing.

Questions

  1. Are there any parameters that can speed up region balancing? The cluster is idle at night, so this parameter can be increased at night and decreased during the day.
  2. Additionally, if the cluster performance degrades significantly during the balancing process, can region migration be paused?
2
| username: zhanggame1 | Original post link

You can view the PD scheduling configuration by executing the following query:

select * from INFORMATION_SCHEMA.CLUSTER_CONFIG t
where t.`KEY` like '%schedule-limit%'

Increase the default values of leader-schedule-limit (default is 4) and region-schedule-limit (default is 2048).

Modification commands:

set config pd `schedule.leader-schedule-limit`=8;
set config pd `schedule.region-schedule-limit`=4096;
3
| username: caiyfc | Original post link

You can refer to the parameter modification methods in this article: Column - Steps to Scale Down TiKV in a Production TiDB Cluster | TiDB Community

4
| username: cy6301567 | Original post link

Good

5
| username: redgame | Original post link

config set max-pending-peer-count 16
config set replica-schedule-limit 64

6
| username: tidb菜鸟一只 | Original post link

The main ones are these three:
leader-schedule-limit
region-schedule-limit
store limit
The specific values depend on your configuration.

7
| username: 扬仔_tidb | Original post link

Thank you all for completing the balance in 20 hours. Here are some pictures:

image
image1380×668 66.1 KB

image
image1125×782 62.2 KB

Old node offline
image
image738×752 27.6 KB

8
| username: Kongdom | Original post link

:astonished: 20 hours? The data volume doesn’t seem particularly large. However, it’s best to use acceleration parameters during off-peak business hours.

9
| username: 扬仔_tidb | Original post link

This is the final region balance diagram.
There is one thing I don’t understand: the new node has a 3TB disk, while the old node has a 2TB disk.

Why does the new machine have more regions than the old machine? Shouldn’t they be balanced equally?

10
| username: Kongdom | Original post link

That’s correct. Previously, with 3 nodes and 3 replicas, each node had 2TB, so the total data volume was 2TB multiplied by 3, which is 6TB. Now, with 2 nodes and 3 replicas, the total data volume remains the same, but the number of nodes has decreased. So, 6TB divided by 2 means each node now has 3TB.

11
| username: 像风一样的男子 | Original post link

Each store node has a Region score, and PD will balance the regions of each node based on this Region score.

12
| username: ljluestc | Original post link
  1. balance-region-scheduler-limit: This parameter can control the speed of region balance migration. By default, the concurrent number of balance migrations per node is 8. You can adjust this parameter as needed, increasing it to speed up balance migration, but be careful not to set it too high to avoid negatively impacting cluster performance.
  2. max-rebalance-region-scheduler-count: This parameter limits the number of schedulers for balancing regions. You can try increasing the value of this parameter to allow more region balance migrations to occur simultaneously.
  3. leader-schedule-limit: The leader scheduler also affects balance. You can consider increasing the value of this parameter as needed to speed up leader scheduling and balancing.