Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: [PD:client:ErrClientGetLeader]get leader from leader address don’t exist error
|
username: TiDBer_oHpB1az2
OS: Ubuntu 22.04.2 LTS
TiDB version: v7.1.0
Development machine testing
When stopping the cluster with tiup cluster stop and then starting it again with tiup cluster start, I encountered the following error:
Error: failed to start tidb: failed to start: 192.168.124.125 tidb-4000.service, please check the instance's log(/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s
Upon checking the logs, there were numerous errors like the following:
[2023/06/20 09:18:26.531 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
After restarting the cluster again with tiup cluster restart, it succeeded.
What does this error mean, and what is the correct way to handle it?
More complete logs:
[2023/06/20 09:19:00.089 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:01.091 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:02.091 +00:00] [WARN] [store.go:84] ["new store with retry failed"] [error="[PD:client:ErrClientGetMember]get member failed"]
[2023/06/20 09:19:02.091 +00:00] [FATAL] [terror.go:309] ["unexpected error"] [error="[PD:client:ErrClientGetMember]get member failed"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:309\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:342\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:241\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:309\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:342\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:241\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/06/20 09:19:22.784 +00:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/06/20 09:19:22.849 +00:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.0] [Edition=Community] ["Git Commit Hash"=635a4362235e8a3c0043542e629532e3c7bb2756] ["Git Branch"=heads/refs/tags/v7.1.0] ["UTC Build Time"="2023-05-30 10:58:57"] [GoVersion=go1.20.3] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]
[2023/06/20 09:19:22.850 +00:00] [INFO] [printer.go:53] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"192.168.124.125\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"192.168.124.125:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"/tidb-deploy/tidb-4000/log/tidb.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"slow-query-file\":\"/tidb-deploy/tidb-4000/log/tidb_slow_query.log\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":300,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":300,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":false,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":false,\"force-init-stats\":false},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first-bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false}"]
[2023/06/20 09:19:22.850 +00:00] [INFO] [main.go:394] ["disable Prometheus push client"]
[2023/06/20 09:19:22.850 +00:00] [INFO] [store.go:76] ["new store"] [path=tikv://192.168.124.125:2379]
[2023/06/20 09:19:22.850 +00:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[192.168.124.125:2379]"] [keyspace-id=0]
[2023/06/20 09:19:22.850 +00:00] [ERROR] [cpu.go:65] [GetCgroupCPU] [error="error when read cpu quota from cgroup v2 at /sys/fs/cgroup/system.slice/tidb-4000.service/cpu.max: open /sys/fs/cgroup/system.slice/tidb-4000.service/cpu.max: no such file or directory"]
[2023/06/20 09:19:22.850 +00:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/06/20 09:19:22.852 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:23.853 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:24.855 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:25.856 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:26.857 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:27.859 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:28.860 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:29.862 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:30.864 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:31.865 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:32.867 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:33.868 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2023/06/20 09:19:35.897 +00:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://192.168.124.125:2379] [error="[PD:client:Err
|
username: 
|
username: