Store Leader Jitter During Backup

Chinese Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 备份期间 store leader 抖动

| username: jeff

[TiDB Usage Environment] Production Environment
[TiDB Version] 4.0.13
[Encountered Problem: Phenomenon and Impact]
Recently expanded by 2 TiKV nodes and reduced by 1. After that, leader jitter occurred during the backup period. Various locks were observed in the logs. The system uses pessimistic locking.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

image
image1008×486 69.3 KB

BR logs:

image
image1380×280 188 KB

TiKV logs:

image
image1289×367 251 KB

Alert information:

image
image824×720 109 KB

2
| username: Billmay表妹 | Original post link

First expand and then shrink, try to maintain 3 KV nodes.
Shrinking to a single KV node is very risky, it is recommended to quickly expand to 3 nodes.

3
| username: jeff | Original post link

It is now 4 TiKV.

4
| username: h5n1 | Original post link

Check if there are any anomalies in the network monitoring of nodeexporter and blackexport.

5
| username: jeff | Original post link

I found the problem. The IOPS of the two newly expanded nodes are much lower than the old TiKV nodes. I’ll adjust them to be the same and check the backup results tomorrow.