Database version upgrade is very slow

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 数据库版本升级非常缓慢

| username: 像风一样的男子

To improve efficiency, please provide the following information. Clear problem descriptions can be resolved more quickly:
【TiDB Usage Environment】Production
【TiDB Version】4.0.9
【Encountered Problem】
Using the following command for an online upgrade, the speed is very slow. It’s been an hour and the first KV node hasn’t completed the upgrade, stuck at migrating the leader. Is this normal??
tiup cluster upgrade tsp-prod-up-cluster v5.4.2

image
image579×634 2.96 KB

2
| username: TiDBer_jYQINSnf | Original post link

Check if the speed of leader demotion in the monitoring is normal.
If it’s not normal, adjust configurations like storelimit.
There are many similar issues in the forum, search for them.

3
| username: 像风一样的男子 | Original post link

It was quite fast at the beginning, but when it reached the last 10k leaders, the speed dropped and became linear.

4
| username: alfred | Original post link

Are the PD and KV nodes heavily loaded?

5
| username: 像风一样的男子 | Original post link

After adjusting the parameters, the progress is now completely stuck.

6
| username: 像风一样的男子 | Original post link

CPU and memory are both around 50%.

7
| username: TiDBer_jYQINSnf | Original post link

Try using evict_leader to force eviction.
Or log it to see if this TiKV is reporting errors.
Alternatively, use pd-ctl to check which regions are left and whether the replicas of these regions are normal.