Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: KV节点快速缩融方案
[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
tidb 6.5.2
[Reproduction Path] What operations were performed when the issue occurred
KV node shrinkage, data distribution process is relatively slow, is it possible to directly take the KV node offline and let TiDB internally balance it automatically; can this method quickly take it offline
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
I thought you had it, but it turns out you were just following a plan. Hanghang then…
pkv went offline immediately after a split, very fast.
Your idea is great, even better than the official one.
Could you please explain in detail? What does it mean when pkv cannot be found? I have two kv instances on one machine, and quickly taking them offline might result in losing replicas.
You can first evict all the leaders on the TiKV nodes to other TiKV nodes using the pd-ctl command.
scheduler add evict-leader-scheduler 1: Add a scheduler to remove all leaders from Store 1.
Is this what pkv is talking about?
To speed up scaling down, you can refer to the solution for adjusting PD parameters in the column.
- Modify PD parameters to speed up the balance process
/tidb-data1/pd/tidb-deploy/pd-2379/bin/pd-ctl -i -u http://127.0.0.1:2379
# View parameters
» config show » store limit (Keep a copy of the original cluster parameters, so you can quickly revert to the original parameters if any issues arise)
# Modify parameters
» config set max-pending-peer-count 256 (Control the upper limit of pending peers for a single store to prevent a large number of regions with lagging logs on some nodes. To speed up replica addition or balance, you can appropriately increase this value. Setting it to 0 means no limit.)
» config set replica-schedule-limit 512 (Control the number of replica scheduling tasks that can be performed simultaneously. This configuration mainly controls the scheduling speed when a node goes down or is taken offline. The larger the value, the faster the scheduling. Setting it to 0 disables scheduling. Replica scheduling is costly, so it is generally not recommended to set this value too high, but since this is a test cluster, setting it higher can speed up the process.)
» store limit all 800 add-peer (Set the speed limit for adding peers to all stores to 800 per minute)
» store limit all 20000 remove-peer (Set the speed limit for removing peers from all stores to 20000 per minute)
# If there are any issues, revert to the original parameters
Is this what pkv is talking about?
https://docs.pingcap.com/zh/tidb/stable/partitioned-raft-kv#partitioned-raft-kv
With pkv, it should be done in a few minutes.
If using s3kv, it would probably be at the second level.
Isn’t this the experimental stage?
Warning
Partitioned Raft KV is currently an experimental feature and is not recommended for use in production environments. This feature may change or be removed without prior notice. If you find any bugs, please report them by raising an issue on GitHub.
It is recommended to follow the formal process to avoid issues, especially in production.
When performing a scale-down operation on TiKV, if data migration is too slow, you should prioritize addressing the issue of slow Region data migration rather than other issues. Hastily taking a TiKV node offline, although it won’t result in data loss, will cause noticeable jitter in the cluster, affecting the service.
Here are some approaches to handle this:
-
Method 1: Adjust the scheduling parameters of PD to appropriately speed up the normal migration scheduling process. (Recommended)
-
Method 2: Evict the Leaders on the TiKV node to be taken offline in advance. Once there are no Leaders, take the node offline without waiting for replica migration, and let the system automatically complete the replica count. The default time interval for the system to start completing replicas is 30 minutes, which can be adjusted as needed. The advantage of this method is that it is relatively fast and has minimal impact on the cluster, but some data will temporarily have one less replica until the system completes the replica.
-
Method 3: If the business importance is average and can tolerate momentary jitter or even momentary unavailability of the cluster, you can use the method you mentioned, directly taking the node offline and letting the system automatically schedule the migration of Leaders and Regions. (Not recommended)
Method 1 is relatively safe and reliable, and it is recommended to use this operation. For specific operational details, you can refer to the content posted by @Kongdom above.
Well, I think it’s better to play it safe and not mess around.
By using the pd-ctl config set command, you can increase parameters such as leader-schedule-limit, replica-schedule-limit, and region-schedule-limit to speed up the scheduling of leaders/regions and expedite the offline process. The above command is used to control the generation speed of scheduling commands on the PD side, while the actual execution is limited by the consumption speed on the TiKV side. You can increase the consumption speed through pd-ctl store limit <store_id> <limit>.
|
username: 最强王者
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: