Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: tidb k8s模式启动失败
Previously, TiDB was installed using the k8s mode, but after uninstalling, it cannot be reinstalled. After starting PD, TiDB did not start, and I am not sure where to check the error information. Please help, thanks.
[root@k8s01 tidb]# kubectl get pods | grep tidb
advanced-tidb-discovery-b5b9d8949-7pw8r 1/1 Running 0 27m
advanced-tidb-pd-0 1/1 Running 0 26m
advanced-tidb-pd-1 1/1 Running 0 26m
Describe the corresponding pod. See if there are any issues.
TiDB didn’t start, and the pods haven’t launched. What should I check?
tidb-discovery has the following error message:
[root@k8s01 ~]# kubectl logs advanced-tidb-discovery-b5b9d8949-7pw8r
I1013 12:07:11.901315 1 version.go:38] Welcome to TiDB Operator.
I1013 12:07:11.901404 1 version.go:39] TiDB Operator Version: version.Info{GitVersion:"v1.3.1", GitCommit:"7b170c3c3e4c9d3de8767fba5246eccdd406371b", GitTreeState:"clean", BuildDate:"2022-02-24T00:57:55Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
E1013 12:07:11.919592 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
I1013 12:07:12.002839 1 main.go:109] starting TiDB Discovery server, listening on 0.0.0.0:10261
I1013 12:07:12.002875 1 main.go:116] starting TiDB Proxy server, listening on 0.0.0.0:10262
E1013 12:07:13.484565 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:07:15.798939 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:07:21.209706 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:07:28.624194 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:07:45.288021 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:08:31.731011 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:09:13.163522 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:09:57.242773 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:10:36.048883 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:11:12.618300 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:11:53.456153 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:12:49.343276 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:13:28.268032 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:14:04.477866 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:14:55.391926 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:15:26.253610 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:16:14.483401 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:16:46.879133 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:17:18.665177 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:17:57.723318 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:18:43.968591 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:19:22.336408 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
E1013 12:20:08.266980 1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
Okay, here are the logs for pd0 and pd1. Please help take a look.
Number of k8s nodes
Check the kubelet logs
Screenshots of pvc and pv
Five nodes, k8s04 is tainted
kubelet logs:
kubelet.log (339.7 KB)
pvc:
pv:
There are no PVC or PV resources bound to TiDB.
Are you referring to this place?
So, should I just increase the size of the required disk in the configuration file?
I don’t quite understand, what exactly needs to be done? Sorry for the trouble.
Do I need to manually create a PVC?
Check the description:
kubectl describe tidbcluster basic -n tidb-cluster
Sent to you, please take a look, thank you.
Take a look at the logs of the pod local-volume-provisioner-xxxx.
A few PVs are from 211 days ago. Clear the PVs and start over.
If the PV is deleted, isn’t the data gone?
Provisioner log:
Additionally, I successfully ran it yesterday and found that commenting out TiFlash allows it to start normally. TiFlash is newly added and wasn’t running before. Could it be that starting TiFlash caused TiKV to fail to start?
Here is my cluster configuration, please help me take a look.
tidb-cluster-local.yaml (36.5 KB)
Take a look at this: kubectl describe tc advanced-tidb.
|
username: TiDBer_NEw0xuKK
|
username: 
|
username: 