Is there a way to evenly distribute all region leaders of a table across all stores in bulk?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 是否有将一表所有region leader批量均分至所有store的方法?

| username: Qiuchi

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 6.5.0
[Reproduction Path] What operations were performed to cause the issue
After creating a new table and using split region to pre-allocate 100 regions, I found that all regions were located on a single store. Although PD scheduling balanced the total number of regions across all stores, the regions of the new table were not distributed. Is there an operation, similar to pd add transfer leader, that can manually distribute the leaders across different stores?

| username: TiDBer_QKDdYGfz | Original post link

The document says so.

| username: Qiuchi | Original post link

This is talking about the behavior of split region, not where the region is placed after splitting.

| username: tidb菜鸟一只 | Original post link

Currently, there is no such operation. In the situation you mentioned, if the leader distribution of a certain table is scattered, the distribution of other tables will also be affected. By default, it is overall balanced.

| username: Qiuchi | Original post link

Got it, but in our scenario, we are actually creating a new table. Someone just mentioned that we can use pre-split region to create the table, so that the leader will be balanced after creation. However, I found that the table only has one region after creation.

The reason is that pre-split can only be used together with SHARD_ROW_ID_BITS.

| username: jetora | Original post link

>> operator show                                        // Show all operators
>> operator show admin                                  // Show all admin operators
>> operator show leader                                 // Show all leader operators
>> operator show region                                 // Show all Region operators
>> operator add add-peer 1 2                            // Add a replica of Region 1 on store 2
>> operator add add-learner 1 2                         // Add a learner replica of Region 1 on store 2
>> operator add remove-peer 1 2                         // Remove a replica of Region 1 on store 2
>> operator add transfer-leader 1 2                     // Transfer the leader of Region 1 to store 2
>> operator add transfer-region 1 2 3 4                 // Transfer Region 1 to stores 2, 3, 4
>> operator add transfer-peer 1 2 3                     // Transfer the replica of Region 1 from store 2 to store 3
>> operator add merge-region 1 2                        // Merge Region 1 with Region 2
>> operator add split-region 1 --policy=approximate     // Split Region 1 into two Regions based on approximate values
>> operator add split-region 1 --policy=scan            // Split Region 1 into two Regions based on precise scan values
>> operator remove 1                                    // Remove the scheduling operation of Region 1
>> operator check 1                                     // Check the status of the operator related to Region 1
| username: xfworld | Original post link

Region and store, there’s another control point through label scheduling. It’s a bit complicated, you can give it a try.

| username: 小龙虾爱大龙虾 | Original post link

TiDB HTTP API has this operation:

15. Scatter regions of the specified table, add a scatter-range scheduler for the PD and the range is the same as the table range.
curl http://{TiDBIP}:10080/tables/{db}/{table}/scatter
Hint: On a partitioned table, use the table(partition) pattern as the table name, test(p1) for example.
Note: The scatter-range scheduler may conflict with the global scheduler, do not use it for long periods on larger tables.
16. Stop scattering the regions, disable the scatter-range scheduler for the specified table.
curl http://{TiDBIP}:10080/tables/{db}/{table}/stop-scatter
Hint: On a partitioned table, use the table(partition) pattern as the table name, test(p1) for example.
| username: 小龙虾爱大龙虾 | Original post link

Sometimes stopping scatter doesn’t work well; you can use pd-ctl to clear the corresponding scheduling stop.

| username: Qiuchi | Original post link

Is the process here to add a label to each store and then use placement rules to restrict it?

| username: xfworld | Original post link

Yes, fix some regions on certain TiKV nodes.

When there are enough TiKV nodes, we want to control the distribution of regions to make it more reasonable. For example, planning this in different ways such as splitting by physical nodes, splitting by racks, splitting by network, etc.

| username: Qiuchi | Original post link

I think it’s doable, but it does come at a pretty high cost :joy:

| username: xfworld | Original post link

If you have enough time, you can give it a try.

| username: Qiuchi | Original post link

Tested and works, but the description is a bit off-putting… What would be the consequences of using it on a large table?