TiKV scaling down for several days and still offline, system logs appear abnormal

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv缩容多天还在下线中,系统日志看着不正常

| username: 点点-求助来了

【TiDB Usage Environment】Production Environment
【TiDB Version】5.0.1
【Reproduction Path】TiKV scaling down
【Encountered Problem: Problem Phenomenon and Impact】
【Resource Configuration】4 nodes, 64 cores, 128 GB memory
【Attachment: Screenshot/Log/Monitoring】

99
991336×1000 235 KB

2
| username: wakaka | Original post link

You need to check tikv.log for detailed information.

3
| username: TiDBer_jYQINSnf | Original post link

Currently, I see a sentence saying that TiKV is down. For the specific reason, check the tikv.log.
Then use PD to take a look.
Lastly, just to mention, after scaling down, TiKV still needs to meet the requirement of 3 replicas. If it doesn’t, scaling down won’t work.

4
| username: 点点-求助来了 | Original post link

tikv log

5
| username: 点点-求助来了 | Original post link

Satisfied

6
| username: TiDBer_jYQINSnf | Original post link

Could you check with pd-ctl to see if this TiKV is not working properly? Why is there a duplicated store address? Was it rebuilt?

7
| username: 点点-求助来了 | Original post link

I saw that it kept failing to go offline, so I rebuilt it once. However, now this node has become online by itself again.