Store Leader Jitter During Backup

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 备份期间 store leader 抖动

| username: jeff

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0.13
[Encountered Problem: Phenomenon and Impact]
Recently expanded by 2 TiKV nodes and reduced by 1. After that, leader jitter occurred during the backup period. Various locks were observed in the logs. The system uses pessimistic locking.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

BR logs:

TiKV logs:

Alert information:

| username: Billmay表妹 | Original post link

First expand and then shrink, try to maintain 3 KV nodes.
Shrinking to a single KV node is very risky, it is recommended to quickly expand to 3 nodes.

| username: jeff | Original post link

It is now 4 TiKV.

| username: h5n1 | Original post link

Check if there are any anomalies in the network monitoring of nodeexporter and blackexport.

| username: jeff | Original post link

I found the problem. The IOPS of the two newly expanded nodes are much lower than the old TiKV nodes. I’ll adjust them to be the same and check the backup results tomorrow.