TiKV Keeps Restarting in Kubernetes

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: k8s下tikv一直重启

| username: TiDBer_N8nfiz7d

[TiDB Usage Environment] Test
[TiDB Version] v4.0.9
[Reproduction Path]
[Encountered Problem: Problem Phenomenon and Impact]

TiKV keeps restarting, logs:

Using --previous to query the logs of the previous container:

| username: h5n1 | Original post link

I have helped you modify the title.

| username: xfworld | Original post link

Obviously, the mounted device was not found.

| username: TiDBer_N8nfiz7d | Original post link

I deployed it in another environment and also encountered this error log, but it was able to start normally.
Additionally, can you tell where the issue is from the FATAL level error below?

| username: xfworld | Original post link

It’s very clear here, none of them can be accessed directly.

| username: redgame | Original post link

Indeed, it cannot be accessed.

| username: ShawnYan | Original post link

data dir = /var/lib/tikv, check the pod’s YAML configuration.

| username: ffeenn | Original post link

Check the storage, what are you using, NFS or something else? Send the deployment method and deployment configuration file. Mainly look at how you created the storage.

| username: TiDBer_N8nfiz7d | Original post link

The device issue has been resolved, but there are still FATAL errors (limited resources, no SSD used).

| username: TiDBer_N8nfiz7d | Original post link

Local storage used by local-pv

| username: TiDBer_N8nfiz7d | Original post link

Additionally, does this issue not affect anything? Because I have the same alert in another environment, but it does not affect the cluster creation.

| username: ffeenn | Original post link

Run kubectl describe pod xx to check the TiKV error information. Version 4.0.9 is too outdated, isn’t it?

| username: TiDBer_N8nfiz7d | Original post link

The image you provided is not visible. Please provide the text you need translated.

| username: ffeenn | Original post link

PV was not successfully bound. Check why it wasn’t bound successfully. Verify if the PVC was created; if not, check the localpv configuration.

Operation example:

kubectl get po -n kube-system -l app=local-volume-provisioner && \
kubectl get pv | grep -e ssd-storage -e shared-ssd-storage -e monitoring-storage -e backup-storage

Reference documentation:
Kubernetes Persistent Storage Configuration | PingCAP Documentation Center

| username: TiDBer_N8nfiz7d | Original post link

But looking at the tikv logs, it has already detected that it is not an SSD disk, so it should have been mounted successfully, right?

| username: ffeenn | Original post link

From the logs, it seems so. How is the pod status?

| username: TiDBer_N8nfiz7d | Original post link

The pod keeps restarting


| username: ffeenn | Original post link

View pod details

| username: TiDBer_N8nfiz7d | Original post link

The pod can start up, but it keeps restarting due to a FATAL error inside. Here is the latest log:


| username: yiduoyunQ | Original post link

Check if the OS kernel version is consistent panic on 'Failed to parse patch version' · Issue #11697 · tikv/tikv · GitHub