TiDB Cluster PD Leader Restart Recovery is Very Slow

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB集群pd leader重启 恢复很慢

| username: 超7成网友

Version 3.0 cluster with 3 PDs (1, 2-leader, 3) and 3 TiDBs
Operation: Network interruption on leader-2 server for 1 minute. According to the logs, pd-3 became the leader but the logs reported an error
[WARN] [cluster_info.go:92] [“region is stale”] [error="region is stale
At the same time, tidb-server cannot connect
My idea: After pd-2 network recovers, stop pd-3 and let pd-2 continue to be the leader
Operation: Stopped pd-3, the original leader pd-2 automatically became the leader, and then the logs still showed
[WARN] [cluster_info.go:92] [“region is stale”] [error="region is stale
After 13 minutes, the following logs appeared

【Please help me check, what does it mean that loading regions took 13 minutes, is it loading metadata? Why does the PD leader switch affect the connection to tidb-server】

[2024/05/21 19:39:24.880 +08:00] [INFO] [cluster_info.go:101] [“load regions”] [count=196311] [cost=13m11.157778823s]

[2024/05/21 19:39:24.884 +08:00] [INFO] [coordinator.go:199] [“coordinator starts to collect cluster information”]

[2024/05/21 19:39:24.887 +08:00] [INFO] [coordinator.go:202] [“coordinator has finished cluster information preparation”]

[2024/05/21 19:39:24.887 +08:00] [INFO] [coordinator.go:212] [“coordinator starts to run schedulers”]

[2024/05/21 19:39:24.889 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=balance-region-scheduler]

[2024/05/21 19:39:24.890 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=balance-leader-scheduler]

[2024/05/21 19:39:24.890 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=balance-hot-region-scheduler]

[2024/05/21 19:39:24.890 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=label-scheduler]

[2024/05/21 19:39:24.890 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=evict-leader-scheduler-7]

[2024/05/21 19:39:24.891 +08:00] [INFO] [coordinator.go:228] [“create scheduler”] [scheduler-name=random-merge-scheduler]

[2024/05/21 19:39:24.894 +08:00] [INFO] [server.go:147] [“establish sync region stream”] [requested-server=pd3] [url=http://xxxx:2379]

[2024/05/21 19:39:24.894 +08:00] [INFO] [server.go:165] [“requested server has already in sync with server”] [requested-server=pd3] [server=pd2] [last-index=1632672700]

[2024/05/21 19:39:24.899 +08:00] [INFO] [server.go:147] [“establish sync region stream”] [requested-server=pd1] [url=http://xxxx:2379]

[2024/05/21 19:39:24.899 +08:00] [INFO] [server.go:165] [“requested server has already in sync with server”] [requested-server=pd1] [server=pd2] [last-index=1632672700]

[2024/05/21 19:39:24.900 +08:00] [INFO] [coordinator.go:140] [“coordinator begins to actively drive push operator”]

[2024/05/21 19:39:24.900 +08:00] [INFO] [util.go:93] [“load cluster version”] [cluster-version=3.0.12]

[2024/05/21 19:39:24.900 +08:00] [INFO] [leader.go:282] [“PD cluster leader is ready to serve”] [leader-name=pd2]

[2024/05/21 19:39:24.900 +08:00] [INFO] [coordinator.go:95] [“coordinator starts patrol regions”]

[2024/05/21 19:39:24.951 +08:00] [WARN] [tso.go:138] [“clock offset”] [jet-lag=13m11.251885613s] [prev-physical=2024/05/21 19:26:13.699 +08:00] [now=2024/05/21 19:39:24.951 +08:00]

| username: xfworld | Original post link

Are you a die-hard fan, still using 3.X …

Upgrade already~ The version is way too outdated.

| username: 超7成网友 | Original post link

The new business has already set up a version 6; this is the old cluster, and there are also many businesses on it.

| username: tidb菜鸟一只 | Original post link

Are the clocks of PD and TiDB-server synchronized properly? I suggest resynchronizing the time and trying again.

| username: 超7成网友 | Original post link

It’s normal now. I don’t understand why it took 13 minutes for PD to recover, such a long time. Also, why did it affect the connection to the TiDB server?

| username: Kongdom | Original post link

:yum: PD loads timestamps, and all operations in TiDB will first go to PD to get a timestamp.

| username: DBAER | Original post link

TiDB executes transactions by obtaining TSO from PD. You should provide the logs within these 13 minutes. I suggest upgrading.

| username: TIDB-Learner | Original post link

Just passing by, version 3.0, don’t dare to speak. Afraid of showing my ignorance.

| username: tidb菜鸟一只 | Original post link

All operations of tidb-server need to get a timestamp from PD. If there is a time synchronization issue between tidb-server and PD, it may cause loading problems and prolonged PD recovery.