Situations When TiFlash Replicas Are Unavailable

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash副本不可用情况

| username: TiDBer_Jack

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.9
[Problem Encountered: Phenomenon and Impact] The trigger reason is that the disk of TiFlash exceeded 80%, causing the addition of new partitions to fail. Then the development team asked me to temporarily disable the TiFlash replicas. After disabling, I was able to add new partitions. However, when trying to add the replicas back, the replica count could not be increased, with PROGRESS stuck at 0.98 and AVAILABLE always at 0. Later, I tested adding new tables, but the replicas could not be added either. Even though PROGRESS reached 1, AVAILABLE was still 0. I also tried adding new TiFlash nodes and tagging them, but it still didn’t work.

image
image1380×460 108 KB

2
| username: xfworld | Original post link

Is there any feedback information in the logs that can be referenced?

3
| username: zhaokede | Original post link

The information provided is too little. Can you send the logs for adding replicas?

4
| username: 小龙虾爱大龙虾 | Original post link

Go check the logs.

5
| username: TiDBer_Jack | Original post link

The issue has been resolved. After reaching 80% disk usage, the DML lock was not released. Even after restarting TiFlash and TiKV, and killing all tasks, it didn’t work. Finally, restarting the TiDB component solved the problem.

6
| username: 小于同学 | Original post link

What is the solution?

7
| username: Kongdom | Original post link

:thinking: Ensuring sufficient available disk space is very important.

8
| username: 健康的腰间盘 | Original post link

Learned a lot, thanks to the original poster.

9
| username: TiDBer_Jack | Original post link

Restarting TiDB to release the lock :joy:. I had restarted TiKV, TiFlash, and PD, but none of them worked. Finally, restarting each TiDB instance one by one solved the problem.

10
| username: TiDBer_LM | Original post link

Learned, marked, thank you, blogger.