TiKV cannot start, showing SST file not found

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv 无法启动,显示找不到sst文件

| username: TiDBer_yM9per2i

Due to the disk being filled by logs, the error message is as follows: No such file or directory\nCan’t access /672945.sst: IO error: No such file or directory while stat a file for size: /opt/tidb/data/tikv-20160/db/672945.sst: No such file or directory\nCan’t access /672946.sst: IO error: No such file or directory while stat a file for size: /opt/tidb/data/tikv-20160/db/672946.sst: No such file or directory\nCan’t access /672947.sst: IO error: No su
[root@kafka03 ~]# vi /opt/tidb/deploy/tikv-20160/log/tikv.log

| username: xfworld | Original post link

Is there no monitoring? This could lead to data loss…

Check if the other TiKV nodes are functioning properly. First, expand a node to ensure no data is lost, then recover the problematic node.
If the other nodes are also not functioning properly, you’ll have to resort to a lossy recovery…

Refer to:

The version is also quite old, which might be a bit troublesome…

| username: TiDBer_yM9per2i | Original post link

The other two nodes are normal. Only the third node has failed.

| username: WalterWj | Original post link

Check the system logs to see if there is a hardware failure.

| username: TiDBer_yM9per2i | Original post link

The hardware is not damaged; it was caused by the disk being full. PD can start, but TiKV cannot start.

| username: TiDBer_yM9per2i | Original post link

With 3 nodes, two nodes are functioning normally, but the third node’s TiKV cannot start, while PD can start.

| username: WalterWj | Original post link

There is this issue with low versions when the disk is full.
Can scaling in and out fix it?

Upgrade. Version 4.x is too old, and the new version has optimizations for full disk situations. 专栏 - TiDB 新特性解读 (6.0~6.6) | TiDB 社区

| username: DBAER | Original post link

It feels like we need to scale up and scale down.

| username: zhaokede | Original post link

Check if the actual files exist in the file system?

| username: TiDBer_yM9per2i | Original post link

The suggested file is missing.

| username: 像风一样的男子 | Original post link

Why is the hostname kafka03, and there is also a Kafka cluster on kv?

| username: oceanzhang | Original post link

Check if the path still exists and if it is readable and writable.

| username: TiDBer_QYr0vohO | Original post link

Scale up, scale down, and repair.