TiDB on K8s TiKV: Found Raft data set when it should not exist

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb on k8s tikv : Found raft data set when it should not exist

| username: TiDBer_hHNMQ8mN

【TiDB Usage Environment】 Testing
【TiDB Version】
【Reproduction Path】 Using NFS as storage class, TiKV deployment encountered issues
【Encountered Problem: Symptoms and Impact】
TiKV failed to start
CrashLoopBackOff

【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】

[root@tidb06 ~]# kubectl logs basic-tikv-0 -n tidb-cluster
starting tikv-server ...
/tikv-server --pd=http://basic-pd:2379 --advertise-addr=basic-tikv-0.basic-tikv-peer.tidb-cluster.svc:20160 --addr=0.0.0.0:20160 --status-addr=0.0.0.0:20180 --advertise-status-addr=basic-tikv-0.basic-tikv-peer.tidb-cluster.svc:20180 --data-dir=/var/lib/tikv --capacity=0 --config=/etc/tikv/tikv.toml

[2022/12/29 02:07:11.977 +00:00] [INFO] [lib.rs:79] ["Welcome to TiKV"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Release Version:   6.1.0"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Edition:           Community"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Git Commit Hash:   080d086832ae5ce2495352dccaf8df5d40f30687"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Git Commit Branch: heads/refs/tags/v6.1.0"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Enable Features:   jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure"]
[2022/12/29 02:07:11.978 +00:00] [INFO] [lib.rs:84] ["Profile:           dist_release"]
[2022/12/29 02:07:11.979 +00:00] [INFO] [mod.rs:74] ["cgroup quota: memory=Some(9223372036854771712), cpu=None, cores={7, 31, 11, 9, 15, 22, 27, 14, 21, 4, 20, 12, 30, 6, 26, 5, 8, 28, 1, 13, 29, 10, 2, 23, 17, 24, 18, 0, 19, 3, 16, 25}"]
[2022/12/29 02:07:11.995 +00:00] [INFO] [mod.rs:81] ["memory limit in bytes: 34367975424, cpu cores quota: 32"]
[2022/12/29 02:07:11.995 +00:00] [INFO] [config.rs:705] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/12/29 02:07:11.995 +00:00] [WARN] [server.rs:1533] ["check: kernel"] [err="kernel parameters net.core.somaxconn got 128, expect 32768"]
[2022/12/29 02:07:11.996 +00:00] [WARN] [server.rs:1533] ["check: kernel"] [err="check_kernel_params failed No such file or directory (os error 2)"]
[2022/12/29 02:07:12.003 +00:00] [INFO] [util.rs:575] ["connecting to PD endpoint"] [endpoints=http://basic-pd:2379]
[2022/12/29 02:07:12.010 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/12/29 02:07:12.015 +00:00] [INFO] [util.rs:575] ["connecting to PD endpoint"] [endpoints=http://basic-pd-2.basic-pd-peer.tidb-cluster.svc:2379]
[2022/12/29 02:07:12.027 +00:00] [INFO] [util.rs:575] ["connecting to PD endpoint"] [endpoints=http://basic-pd-0.basic-pd-peer.tidb-cluster.svc:2379]
[2022/12/29 02:07:12.038 +00:00] [INFO] [util.rs:701] ["connected to PD member"] [endpoints=http://basic-pd-0.basic-pd-peer.tidb-cluster.svc:2379]
[2022/12/29 02:07:12.038 +00:00] [INFO] [util.rs:567] ["all PD endpoints are consistent"] [endpoints="[\"http://basic-pd:2379\"]"]
[2022/12/29 02:07:12.041 +00:00] [INFO] [server.rs:369] ["connect to PD cluster"] [cluster_id=7182053587741419436]
[2022/12/29 02:07:12.041 +00:00] [INFO] [config.rs:2041] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/12/29 02:07:12.041 +00:00] [INFO] [config.rs:2064] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/12/29 02:07:12.042 +00:00] [FATAL] [setup.rs:304] ["invalid configuration: Found raft data set when it should not exist."]
2
| username: Billmay表妹 | Original post link

Refer to the example to confirm whether the storage-provisioner is installed correctly https://docs.pingcap.com/zh/tidb-in-kubernetes/dev/configure-storage-class#示例

You can check out these articles:
【SOP Series 39】Deploy TiDB through TiOperator
【SOP Series 40】Monitoring and Alerting for TiDB Cluster Deployed by TiDB Operator
【SOP Series 15】How to Deploy TiDB-Operator on Kubernetes (Part 1)

3
| username: TiDBer_hHNMQ8mN | Original post link

I am using the nfs-client-provisioner for the NFS client here, but it is not in the same namespace. Does this affect anything?

nfs-client-provisioner-76cfd5698-xwm29   1/1     Running   0              26h
4
| username: 数据小黑 | Original post link

What are the statuses of the PVC and PV corresponding to TiKV?

5
| username: xfworld | Original post link

“check_kernel_params failed No such file or directory (os error 2)”

It directly reports that the directory was not found…

I feel it is not supported.

6
| username: TiDBer_jYQINSnf | Original post link

NFS is very clear; after the NFS PV is deleted, the data will not be deleted.
Your issue is clearly due to the reuse mentioned above, but the UUID naming of k8s PVs generally doesn’t repeat. Each PV is mounted in a different directory on the NFS. Is that the case?

Find a host to mount the NFS and check the directory structure.

7
| username: TiDBer_hHNMQ8mN | Original post link

I am using the local storage class without any issues. It is most likely caused by the problem you mentioned.

8
| username: Billmay表妹 | Original post link

Try it out according to everyone’s suggestions and see how it goes~

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.