TiKV Node Expansion: Unbalanced Regions

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv扩容节点 不均衡region

| username: fly4310862

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] 4.0.8
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]

There are 4 TiKV nodes. After successfully expanding to 2 more nodes, the monitoring also shows that there are currently 6 TiKV nodes. However, the data regions from the previous 4 nodes have not been balanced to the new 2 nodes. What could be the possible reasons for this?

[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]

| username: redgame | Original post link

Do you have any other information? Like logs or something?

| username: tidb菜鸟一只 | Original post link

Let’s see the Grafana screenshot. Hasn’t even one region been balanced? Please share the cluster topology.

| username: xfworld | Original post link

Are all the configurations of TiKV nodes consistent? Or are they different?

| username: zhanggame1 | Original post link

Please send a screenshot of the region, then send the capacity of each TiKV. Finally, check if there are any error logs.

| username: xingzhenxiang | Original post link

Overview → Please take a look at the screenshots of the leaders and regions under TiKV.

| username: WalterWj | Original post link

Version 4.0 is almost out of maintenance, upgrade quickly.

| username: Anna | Original post link

I didn’t see your logs and screenshots.

| username: ljluestc | Original post link

Here are some potential reasons to consider:

Automatic balancing configuration: Check if the TiKV cluster has automatic balancing enabled. If it is disabled, the cluster will not automatically balance data regions among nodes. You can enable automatic balancing by adjusting the relevant configuration parameters.

Region splitting and merging settings: TiKV uses region splitting and merging operations to balance data among nodes. Improper configuration of splitting and merging settings may lead to uneven data distribution. Ensure that the splitting and merging configurations are appropriately set for your workload.

Leader balancing: In addition to data balancing, TiKV also balances leaders across nodes. If leader balancing is not enabled or configured correctly, it may result in uneven leader distribution, which in turn affects data balance. Check the leader balancing settings and ensure they are configured correctly.

Hotspot imbalance: TiKV automatically detects hotspots (regions with high read/write traffic) and attempts to move them to different nodes for load balancing. If there are hotspots on the original 4 nodes, they may prevent data from balancing to new nodes. Monitor the hotspot status and take appropriate measures to resolve hotspot issues, such as adjusting the workload or TiKV configuration.

Network or hardware limitations: Network or hardware issues may affect the data balancing process. Ensure that the network connections between nodes are stable and that the hardware resources (CPU, memory, storage) on all nodes are sufficient to meet the workload requirements.