TiKV Crash and Failure to Restart

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv宕机起不来

| username: Jolyne

[TiDB Usage Environment] Production Environment / Test / Poc
Production Environment
[TiDB Version]
5.2.1
[Reproduction Path] What operations were performed when the issue occurred
The TiKV node crashed inexplicably and could not be started. After checking, the disk and disk space were not the problem.
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

1671158073901
16711580739011380×226 43.8 KB

1671158100525
1671158100525978×243 33.1 KB

1671158243720
16711582437201144×129 13.2 KB

2
| username: WalterWj | Original post link

Check the system logs to see if there are any io error logs.

3
| username: Jolyne | Original post link

There are no errors related to IO.

4
| username: tidb菜鸟一只 | Original post link

Check the system monitoring at the downtime point to see if the memory is exhausted.

5
| username: WalterWj | Original post link

Please send the content of the toml configuration file for tikv.

6
| username: Jolyne | Original post link

After checking the monitoring, the memory is not fully utilized. According to the monitoring, only over 50GB is used.

7
| username: Jolyne | Original post link

The screenshot is not visible. Please provide the text you need translated.

8
| username: WalterWj | Original post link

Alright, try the scaling method to fix it first.

9
| username: Jolyne | Original post link

:rofl: Is there any other way? Because there was a problem with scaling up and down before. After scaling down and then scaling up, it reported that it already existed. I was told that some files need to be completely deleted.

10
| username: WalterWj | Original post link

It’s probably because you’re using the same IP and port. Try using a new server or changing the port.

11
| username: 我是咖啡哥 | Original post link

It feels like the file is corrupted. Try expanding and then shrinking it.

12
| username: Jolyne | Original post link

There should be no problem with scaling in and out using the same IP and port. I encountered an issue with this on TiFlash before.

13
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.