Summary of TiKV Unexpected Crash and Restart Failure: Panic Mark File

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Tikv 意外崩溃, 重启失败 panic_mark_file 总结篇

| username: 最强王者

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred

[Encountered Issues: Issue Phenomenon and Impact]

  1. TiKV cannot restart
  2. Deleting the panic_mark_file and restarting still fails

Solution: First, expand the TiKV instance into the cluster, then shrink the faulty instance;

[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
[2024/01/25 20:46:37.618 +08:00] [FATAL] [server.rs:405] [“panic_mark_file tidb/tidb-data/tikv-20160/panic_mark_file exists, there must be something wrong with the db. Do not remove the panic_mark_file and force the TiKV node to restart. Please contact TiKV maintainers to investigate the issue. If needed, use scale in and scale out to replace the TiKV node. Scale a TiDB Cluster Using TiUP | PingCAP Docs”]

| username: changpeng75 | Original post link

The log clearly states, “Do not remove the panic_mark_file and force the TiKV node to restart.”

| username: Kongdom | Original post link

It is really commendable and worth learning that TiDB error messages include solutions! :+1:

| username: dba远航 | Original post link

Do not remove the panic_mark_file and force the TiKV node to restart.

| username: zhang_2023 | Original post link

An important method for troubleshooting issues is logs.

| username: 最强王者 | Original post link

Forcing a restart is useless; it will result in an error.

| username: 最强王者 | Original post link

Forced restart is useless. I have encountered this before, and restarting or dealing with files doesn’t help.

| username: 这里介绍不了我 | Original post link

The logs are quite detailed.

| username: WinterLiu | Original post link

Learned something new again.

| username: lemonade010 | Original post link

Learned, mark.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.