Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: TiFlash下线节点超级慢
[TiDB Usage Environment] Production Environment
[TiDB Version] v5.0.6
[Reproduction Path] Today, two TiFLASH nodes kept restarting for unknown reasons, affecting business operations, so I manually downed the nodes. After that, I used scale-in to shrink one of the TiFLASH nodes. Using the tiup command to check, it was always in a pending off state, and the operator would only initiate scheduling every few dozen minutes.

config show
I don’t quite understand why the offline process is so slow.
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
You might want to look at the scheduling-related parameters, which can speed up the entire decommissioning process.
You can adjust the relevant parameters to speed up the scheduling, but it may have an impact on the production environment.
The parameters are fine, but the main issue is that it takes about 40 minutes each time before scheduling starts. Not sure why.
How many TiFlash nodes are there, and how many TiFlash replicas are set in the cluster? If the number of replicas is greater than the number of nodes, it cannot be scaled down.
10 TiFlash nodes, 2 replicas
So it’s unexpected; usually, you need to add replicas when taking a node offline. I understand that TiFlash is currently experiencing some issues. It’s best to check the error logs to find out why TiFlash keeps restarting.
If that doesn’t work, you’ll need to recreate the TiFlash replicas.
Yes, my concern is that taking TiFlash offline is very slow. It schedules data offline only once every 40 minutes, and then it takes a long time.
Is it initially slow or not moving at all? You can check the TiFlash monitoring to see if the number of regions is decreasing. If you think it’s slow, you can increase the store limit.
There is scheduling in this graph, approximately once every 40 minutes. The store limit is 200.
Take a look at finish, was it successful?