Abnormal TiKV Startup

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv 启动异常

| username: rw12306

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
TiKV cannot start, it reports an error directly.
[Attachment: Screenshot/Log/Monitoring]

00] [FATAL] [server.rs:428] [“panic_mark_file /home/tidb/tidb-data/tikv-20160/panic_mark_file exists, there must be something wrong with the db. Do not remove the panic_mark_file and force the TiKV node to restart. Please contact TiKV maintainers to investigate the issue. If needed, use scale in and scale out to replace the TiKV node. Scale a TiDB Cluster Using TiUP | PingCAP Docs”]

| username: Billmay表妹 | Original post link

How many TiKV instances have you deployed?

| username: Billmay表妹 | Original post link

Please do not delete the panic_mark_file and forcibly restart the TiKV node. If necessary, please use scale in and scale out to replace the TiKV node.
Did you delete something?

| username: 有猫万事足 | Original post link

Refer to this.

| username: rw12306 | Original post link

3 nodes

| username: rw12306 | Original post link

First, it crashed on its own, and then it couldn’t start anymore. Nothing was deleted.

| username: rw12306 | Original post link

This document didn’t solve the problem either. I just deleted that file and restarted. I saw that the file was empty and had nothing in it.

| username: 有猫万事足 | Original post link

It looks like before creating this file, a critical level log will be output, which records the reason for the forced abnormal exit. Then, an atomic boolean value will be set, and if this boolean value is found to be true, this panic_mark_file will be created.

If you have a certain grasp of why it exited last time, this operation is not impossible. However, this is not the recommended approach. The recommended approach is to scale down this TiKV and then scale up a new one. It is best to figure out the reason for the abnormal exit.

| username: redgame | Original post link

You can try moving it to another location, there are no other prompts.