I’m deploying a self-managed TiDB cluster on Kubernetes (offline, no NetworkPolicy) using the basic cluster example from PingCAP’s GitHub repo, with TiDB Operator version v1.6.1.
I’m encountering an issue where TiKV cannot connect to the PD endpoint.
kubectl get svc basic-pd exposes port 2379.
From inside a TiKV pod:
curl http://basic-pd:2379/health responds OK.
DNS resolution works via getent hosts basic-pd.
PD cluster seems healthy when queried via etcdctl inside the PD pod.
Question:
Why does TiKV still fail with validate PD endpoints failed even though PD is reachable and advertising the correct URL?
Could leftover PVC data or internal PD metadata still be causing an endpoint mismatch?
What’s the recommended workflow to cleanly reset a TiDB cluster deployment in Kubernetes to avoid these kinds of issues?
Any advice, known bugs for v1.6.1, or debugging steps I might have missed would be greatly appreciated
Certainly! Here’s a comprehensive, actionable troubleshooting guide for TiKV ‘validate PD endpoints failed’ errors in a self-managed TiDB cluster on Kubernetes, particularly taking into account issues around leftover PVC data, internal PD metadata mismatches, and TiDB Operator v1.6.1 nuances.
1. Root Cause Analysis
Background: ‘validate PD endpoints failed’ in TiKV
This error occurs when TiKV attempts to join or reconnect to the PD (Placement Driver) service but encounters endpoint verification failures. The most common root causes for this issue in Kubernetes-managed TiDB clusters are:
a) Stale PVC (Persistent Volume Claim) Data
Persistent Volume Claims are used for data storage across pod restarts and recreations.
If PVCs are not deleted after cluster teardown or before recreating a cluster with the same name, any pods (especially StatefulSet-managed ones like TiKV and PD) may reuse old data.
TiKV (and PD) stores unique cluster and store metadata on its disk (PV/PVC). If a new cluster is created but reuses an old PVC, TiKV may try to join a cluster using stale cluster IDs, resulting in endpoint mismatches and errors such as:
“cluster id mismatch”
or
“validate PD endpoints failed”
This is especially problematic if the cluster is deleted but the storage backend (PV/PVC) is set to “Retain” or not cleaned reference.
b) PD Metadata and Cluster Lifecycle
PD persists the overall cluster topology, including node addresses, store IDs, and cluster UUIDs.
If the PD state (either in the etcd backend or on its own disks) is not properly wiped, a new deployment may clash with leftover topology information, causing TiKV/PD registration validation to fail.
Re-deployments—using existing data but intending to “start fresh”—will fail due to these mismatches if clean-up is incomplete.
2. Step-by-Step Workflow for Clean, Effective Cluster Reset
To avoid cluster ID or endpoint issues, always follow a thorough cleanup and reinitialization process when tearing down or redeploying a TiDB cluster. Here is a safe, best-practice workflow:
a) Scale Down and Delete the TiDB Cluster
Scale down all components (PD, TiKV, TiDB, TiFlash, etc.) to zero prior to deletion for a clean shutdown (optional but recommended):
3. Version-Specific Tips, Debugging, and Prevention for v1.6.1
Key Recommendations for TiDB Operator v1.6.1
There are no breaking bugs explicitly documented in v1.6.1 for endpoint validation, but previous patterns apply.
Newer versions have refined rolling update and config update mechanisms—always ensure PVC/PV cleanup before reusing cluster names.
Endpoint Validation Debugging Steps
Check TiKV Pod logs to confirm the exact error:
kubectl -n <namespace> logs <tikv-pod-name>
Look for “validate PD endpoints failed”, “cluster id mismatch”, or other relevant errors.
Verify PD health and endpoints:
Exec into a TiKV pod or any available pod and try:
curl http://<pd-service>:2379/pd/api/v1/members
Confirm that the PD endpoints are correct and reachable from within the cluster.
Double-check PVC usage:
Use kubectl get pvc,pv -n <namespace> and confirm that there are no leftover volumes from prior clusters.
Inspect cluster metadata:
If you see store/cluster ID mismatches, it’s almost always due to old data on the underlying PVs/PVCs, not a networking or configuration fault.
Confirm unique hostnames and endpoints:
Avoid running multiple clusters with the same name in the same namespace unless you’re using separate PVCs/resources.
Cluster Reinstall/Reset Tips
NEVER just delete pods or only StatefulSets; always also clean up PVCs/PVs before redeploying with the same cluster name.
When using local PVs, ensure the underlying disk is formatted or cleared after PVC deletion, or Kubernetes might re-mount old data.
If using an external storage backend (i.e., NFS, cloud disk), ensure any “Retain” policy volumes are deleted manually if not needed.
Operator/PD-specific nuances
If the cluster fails with DNS-related errors, try setting spec.pd.startUpScriptVersion: "v1" in your TidbCluster spec to use dig rather than nslookup for endpoint discovery, as recommended for newer PD/TiDB versions see.
Always check that the advertised PD URLs in the TidbCluster CR match the DNS records exposed by Services and StatefulSets in Kubernetes.
Never re-use existing PVCs/PVs with a new, unrelated cluster deployment—always delete or replace volumes to avoid “validate PD endpoints failed” or “cluster id mismatch” errors.
Automate cleanup as part of your cluster destroy/initialize pipeline.
Before reusing a cluster name or namespace, always check for lingering PVCs or PVs.
By following the above analysis and workflow, you will ensure that all TiDB/TiKV/PD components initialize cleanly, preventing endpoint validation errors rooted in data residue.
Question: