TiDB PD TLS setup with cert-manager — “tls: bad certificate” and handshake errors

TiDB
, ,
1

Hi guys,
I am deploying TiDB with TLS enabled for PD components in a Kubernetes cluster using cert-manager to generate certificates. However, PD logs show errors such as:

tls: bad certificate

tls: first record does not look like a TLS handshake

transport: authentication handshake failed: client certificate authentication failed

I am struggling to get the TLS certs properly validated between TiDB and PD, and my cluster is stuck with failed connections.
What I have done so far:

Created a Certificate resource with cert-manager to issue certificates for TiDB PD servers and clients.

Used cert-allowed-cn = [“tidb-server”] in PD config to specify allowed client CN.

Enable tls in cluster config

Configured SANs for the certificate to include relevant Kubernetes service DNS names.

Mounted certificates into PD and TiDB pods and configured their TLS options accordingly.

PD server TLS config snippet (in config file or Helm values):

[security]
ca-path = “/var/lib/tls/ca.crt”
cert-path = “/var/lib/tls/tidb-server.crt”
key-path = “/var/lib/tls/tidb-server.key”

Errors in PD logs:

[WARN] [config_logging.go:170] [“rejected connection on client endpoint”] [remote-addr=10.42.6.51:39266] [server-name=global-pd.tidb-cluster] [error=“remote error: tls: bad certificate”]
[WARN] [config_logging.go:170] [“rejected connection on client endpoint”] [remote-addr=10.42.6.37:37590] [server-name=] [error=“tls: first record does not look like a TLS handshake”]
[FATAL] [“run server failed”] [error=“transport: authentication handshake failed: client certificate authentication failed”]

Questions:

Am I correctly configuring SANs for the PD server certificates?

Is using “tidb-server” as commonName and cert-allowed-cn sufficient for client cert verification?

Could the errors mean that some clients connect without TLS? How can I identify and fix this?

What are the common pitfalls when setting up TLS with cert-manager for TiDB PD?

How to verify certs inside the pods properly?

Additional info:

Kubernetes version: 1.30

cert-manager version: 1.17.2

TiDB Operator / TiDB version: 8.5

Cluster issuer type: self-signed

Thanks in advance for your help!

2

Troubleshooting TLS Certificate Issues in TiDB PD Components

Table of Contents

  1. Overview of TiDB PD’s TLS Requirements
  2. Common Causes of TLS Certificate Validation Failures
  3. Best Practices for SAN and CN Configuration
  4. Client Verification Setup and Troubleshooting
  5. Step-by-Step Diagnostic Procedures
  6. Sample cert-manager Configurations
  7. Verification Methods

Overview of TiDB PD’s TLS Requirements

The Placement Driver (PD) component in TiDB requires properly configured TLS certificates to ensure secure communication with other components in the TiDB cluster. When deployed in Kubernetes using TiDB Operator, PD needs certificates that:

  • Cover all possible DNS names and IP addresses that might be used to access the PD service
  • Include both server and client authentication capabilities
  • Are signed by a trusted Certificate Authority (CA)
  • Have the correct Common Name (CN) for component authentication

PD uses TLS for:

  • Communication between PD and TiDB server
  • Communication between PD and TiKV
  • Internal communication within the PD cluster (peer-to-peer)
  • Communication with monitoring tools and clients like PD Control

Common Causes of TLS Certificate Validation Failures

When encountering “tls: bad certificate” errors in TiDB PD components, several issues might be at play:

1. Certificate Expiration

  • Expired certificates will cause immediate TLS handshake failures
  • Certificate expiration often occurs without warning if monitoring is not set up

2. Missing or Incorrect SAN Entries

  • Kubernetes service DNS names not included in the certificate
  • Missing wildcard entries for PD peer discovery
  • Missing IP addresses (especially localhost addresses)

3. Certificate Chain Issues

  • Incomplete certificate chain
  • Incorrect CA certificate referenced in the TiDB cluster configuration
  • CA certificate not included in the trust store

4. Kubernetes Secret Management Problems

  • Secret not created with the expected name
  • Secret missing required keys
3

Hi @hamid May I know if you have configured certificates for both TiKV and TiDB?

4

Hi @Hazel
thank you for replying,
I’ve created one certificates as but in client I have error .

5

Hi @hamid Just to confirm, have you enabled mTLS for all components? If not, we recommend doing so by following this guide: Enable TLS between TiDB Components

1 Like