Tidb-server failed to start, can anyone help take a look?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Tidb-server启动失败,有没人帮助看下哈

| username: TiDBer_vZ5xlHrG

[TiDB Usage Environment] Test
[TiDB Version] v6.4.0
[Reproduction Path] Manual deployment failed
[Encountered Problem: Issue Phenomenon and Impact] Deployment using tiup playground was successful, but manual deployment failed, and tidb-server could not start.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

[2022/12/30 18:52:04.516 +08:00] [INFO] [ddl.go:1105] [“[ddl] DDL job is finished”] [jobID=37]
[2022/12/30 18:52:04.516 +08:00] [INFO] [callback.go:121] [“performing DDL change, must reload”]
[2022/12/30 18:52:04.516 +08:00] [INFO] [split_region.go:85] [“split batch regions request”] [“split key count”=1] [“batch count”=1] [“first batch, region ID”=10] [“first split key”=748000000000000024]
[2022/12/30 18:52:04.521 +08:00] [INFO] [split_region.go:187] [“batch split regions complete”] [“batch region ID”=10] [“first at”=748000000000000024] [“first new region left”=“{Id:44 StartKey:7480000000000000ff2200000000000000f8 EndKey:7480000000000000ff2400000000000000f8 RegionEpoch:{ConfVer:1 Version:22} Peers:[id:45 store_id:1 ] EncryptionMeta: IsInFlashback:false}”] [“new region count”=1]
[2022/12/30 18:52:04.521 +08:00] [INFO] [split_region.go:236] [“split regions complete”] [“region count”=1] [“region IDs”=“[44]”]
[2022/12/30 18:52:13.853 +08:00] [INFO] [global.go:34] [“daemon became owner”] [id=378d35a3-36d1-44ee-99e5-32a098f9e08a] [daemon-id=LogBackup::Advancer]
[2022/12/30 18:53:19.518 +08:00] [INFO] [region_request.go:794] [“mark store’s regions need be refill”] [id=1] [addr=192.168.1.187:20160] [error=“context deadline exceeded”]
[2022/12/30 18:53:19.519 +08:00] [WARN] [session.go:989] [“can not retry txn”] [label=internal] [error=“context deadline exceeded”] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=true] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2022/12/30 18:53:19.519 +08:00] [WARN] [session.go:1005] [“commit failed”] [“finished txn”=“Txn{state=invalid}”] [error=“context deadline exceeded”]
[2022/12/30 18:53:19.520 +08:00] [WARN] [session.go:2191] [“run statement failed”] [schemaVersion=18] [error=“context deadline exceeded”] [session=“{\n "currDBName": "mysql",\n "id": 0,\n "status": 2,\n "strictMode": true,\n "user": null\n}”]
[2022/12/30 18:53:19.520 +08:00] [FATAL] [bootstrap.go:2192] [“mustExecute error”] [error=“context deadline exceeded”] [stack=“github.com/pingcap/tidb/session.mustExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:2192\ngithub.com/pingcap/tidb/session.insertBuiltinBindInfoRow\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1513\ngithub.com/pingcap/tidb/session.initBindInfoTable\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1509\ngithub.com/pingcap/tidb/session.doDDLWorks\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:2044\ngithub.com/pingcap/tidb/session.bootstrap\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:455\ngithub.com/pingcap/tidb/session.runInBootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3002\ngithub.com/pingcap/tidb/session.BootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2875\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:310\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:211\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]
[2022/12/30 18:53:34.796 +08:00] [INFO] [cpuprofile.go:113] [“parallel cpu profiler started”]
[2022/12/30 18:53:34.797 +08:00] [INFO] [printer.go:34] [“Welcome to TiDB.”] [“Release Version”=v6.4.0] [Edition=Community] [“Git Commit Hash”=cf36a9ce2fe1039db3cf3444d51930b887df18a1] [“Git Branch”=heads/refs/tags/v6.4.0] [“UTC Build Time”=“2022-11-13 05:25:30”] [GoVersion=go1.19.2] [“Race Enabled”=false] [“Check Table Before Drop”=false] [“TiKV Min Version”=6.2.0-alpha]
[2022/12/30 18:53:34.797 +08:00] [INFO] [printer.go:48] [“loaded config”] [config=“{"host":"0.0.0.0","advertise-address":"192.168.1.187","port":44000,"cors":"","store":"tikv","path":"192.168.1.187:2379","socket":"/tmp/tidb-44000.sock","lease":"45s","split-table":true,"token-limit":1000,"temp-dir":"/tmp/tidb","tmp-storage-path":"/tmp/1001_tidb/MC4wLjAuMDo0NDAwMC8wLjAuMC4wOjEwMDgw/tmp-storage","tmp-storage-quota":-1,"server-version":"","version-comment":"","tidb-edition":"","tidb-release-version":"","log":{"level":"info","format":"text","disable-timestamp":null,"enable-timestamp":null,"disable-error-stack":null,"enable-error-stack":null,"file":{"filename":"/tidb-deploy/tidb-44000/log/tidb.log","max-size":300,"max-days":0,"max-backups":0},"slow-query-file":"/tidb-deploy/tidb-44000/log/tidb_slow_query.log","expensive-threshold":10000,"query-log-max-len":4096,"enable-slow-log":true,"slow-threshold":300,"record-plan-in-slow-log":1},"instance":{"tidb_general_log":false,"tidb_pprof_sql_cpu":false,"ddl_slow_threshold":300,"tidb_expensive_query_time_threshold":60,"tidb_enable_slow_log":true,"tidb_slow_log_threshold":300,"tidb_record_plan_in_slow_log":1,"tidb_check_mb4_value_in_utf8":true,"tidb_force_priority":"NO_PRIORITY","tidb_memory_usage_alarm_ratio":0.8,"tidb_enable_collect_execution_info":true,"plugin_dir":"/data/deploy/plugin","plugin_load":"","max_connections":0,"tidb_enable_ddl":true,"tidb_rc_read_check_ts":false},"security":{"skip-grant-table":false,"ssl-ca":"","ssl-cert":"","ssl-key":"","cluster-ssl-ca":"","cluster-ssl-cert":"","cluster-ssl-key":"","cluster-verify-cn":null,"session-token-signing-cert":"","session-token-signing-key":"","spilled-file-encryption-method":"plaintext","enable-sem":false,"auto-tls":false,"tls-version":"","rsa-key-size":4096,"secure-bootstrap":false,"auth-token-jwks":"","auth-token-refresh-interval":"1h0m0s"},"status":{"status-host":"0.0.0.0","metrics-addr":"","status-port":10080,"metrics-interval":15,"report-status":true,"record-db-qps":false,"grpc-keepalive-time":10,"grpc-keepalive-timeout":3,"grpc-concurrent-streams":1024,"grpc-initial-window-size":2097152,"grpc-max-send-msg-size":2147483647},"performance":{"max-procs":0,"max-memory":0,"server-memory-quota":0,"stats-lease":"3s","stmt-count-limit":5000,"pseudo-estimate-ratio":0.8,"bind-info-lease":"3s","txn-entry-size-limit":6291456,"txn-total-size-limit":104857600,"tcp-keep-alive":true,"tcp-no-delay":true,"cross-join":true,"distinct-agg-push-down":false,"projection-push-down":false,"max-txn-ttl":3600000,"index-usage-sync-lease":"0s","plan-replayer-gc-lease":"10m","gogc":100,"enforce-mpp":false,"stats-load-concurrency":5,"stats-load-queue-size":1000,"analyze-partition-concurrency-quota":16,"enable-stats-cache-mem-quota":false,"committer-concurrency":128,"run-auto-analyze":true,"force-priority":"NO_PRIORITY","memory-usage-alarm-ratio":0.8,"enable-load-fmsketch":false},"prepared-plan-cache":{"enabled":true,"capacity":100,"memory-guard-ratio":0.1},"opentracing":{"enable":false,"rpc-metrics":false,"sampler":{"type":"const","param":1,"sampling-server-url":"","max-operations":0,"sampling-refresh-interval":0},"reporter":{"queue-size":0,"buffer-flush-interval":0,"log-spans":false,"local-agent-host-port":""}},"proxy-protocol":{"networks":"","header-timeout":5},"pd-client":{"pd-server-timeout":3},"tikv-client":{"grpc-connection-count":4,"grpc-keepalive-time":10,"grpc-keepalive-timeout":3,"grpc-compression-type":"none","commit-timeout":"41s","async-commit":{"keys-limit":256,"total-key-size-limit":4096,"safe-window":2000000000,"allowed-clock-drift":500000000},"max-batch-size":128,"overload-threshold":200,"max-batch-wait-time":0,"batch-wait-size":8,"enable-chunk-rpc":true,"region-cache-ttl":600,"store-limit":0,"store-liveness-timeout":"1s","copr-cache":{"capacity-mb":1000},"ttl-refreshed-txn-size":33554432,"resolve-lock-lite-threshold":16},"binlog":{"enable":false,"ignore-error":false,"write-timeout":"15s","binlog-socket":"","strategy":"range"},"compatible-kill-query":false,"pessimistic-txn":{"max-retry-count":256,"deadlock-history-capacity":10,"deadlock-history-collect-retryable":false,"pessimistic-auto-commit":false,"constraint-check-in-place-pessimistic":true},"max-index-length":3072,"index-limit":64,"table-column-count-limit":1017,"graceful-wait-before-shutdown":0,"alter-primary-key":false,"treat-old-version-utf8-as-utf8mb4":true,"enable-table-lock":false,"delay-clean-table-lock":0,"split-region-max-num":1000,"top-sql":{"receiver-address":""},"repair-mode":false,"repair-table-list":,"isolation-read":{"engines":["tikv","tiflash","tidb"]},"new_collations_enabled_on_first_bootstrap":true,"experimental":{"allow-expression-index":false},"skip-register-to-dashboard":false,"enable-telemetry":true,"labels":{},"enable-global-index":false,"deprecate-integer-display-length":false,"enable-enum-length-limit":true,"stores-refresh-interval":60,"enable-tcp4-only":false,"enable-forwarding":false,"max-ballast-object-size":0,"ballast-object-size":0,"transaction-summary":{"transaction-summary-capacity":500,"transaction-id-digest-min-duration":2147483647},"enable-global-kill":true,"enable-batch-dml":false,"mem-quota-query":1073741824,"oom-action":"cancel","oom-use-tmp-storage":true,"check-mb4-value-in-utf8":true,"enable-collect-execution-info":true,"plugin":{"dir":"/data/deploy/plugin","load":""},"max-server-connections":0,"run-ddl":true,"tidb-max-reuse-chunk":64,"tidb-max-reuse-column":256}”]
[2022/12/30 18:53:34.797 +08:00] [INFO] [global.go:34] [“disable Prometheus push client”]
[2022/12/30 18:53:34.797 +08:00] [INFO] [store.go:75] [“new store”] [path=tikv://192.168.1.187:2379]
[2022/12/30 18:53:34.797 +08:00] [INFO] [global.go:34] [“[pd] create pd client with endpoints”] [pd-address=“[192.168.1.187:2379]”]
[2022/12/30 18:53:34.797 +08:00] [INFO] [systime_mon.go:26] [“start system time monitor”]
[2022/12/30 18:53:34.799 +08:00] [INFO] [global.go:34] [“[pd] switch leader”] [new-leader=http://192.168.1.187:2379] [old-leader=]
[2022/12/30 18:53:34.799 +08:00] [INFO] [global.go:34] [“[pd] init cluster id”] [cluster-id=7182892640331459658]
[2022/12/30 18:53:34.799 +08:00] [INFO] [global.go:34] [“[pd] tso dispatcher created”] [dc-location=global]
[2022/12/30 18:53:34.800 +08:00] [INFO] [store.go:81] [“new store with retry success”]
[2022/12/30 18:53:34.805 +08:00] [INFO] [tidb.go:77] [“new domain”] [store=tikv-7182892640331459658] [“ddl lease”=45s] [“stats lease”=3s] [“index usage sync lease”=0s]
[2022/12/30 18:53:34.811 +08:00] [INFO] [domain.go:2098] [acquireServerID] [serverID=486506] [“lease id”=10df8562a8e44e95]
[2022/12/30 18:53:34.818 +08:00] [WARN] [info.go:245] [“init TiFlashReplicaManager”] [“pd addrs”=“[192.168.1.187:2379]”]
[2022/12/30 18:53:34.828 +08:00] [INFO] [domain.go:214] [“full load InfoSchema success”] [currentSchemaVersion=0] [neededSchemaVersion=18] [“start time”=6.701988ms]
[2022/12/30 18:53:34.829 +08:00] [INFO] [domain.go:481] [“full load and reset schema validator”]
[2022/12/30 18:53:34.829 +08:00] [INFO] [ddl.go:701] [“[ddl] start DDL”] [ID=40f78b0a-56f9-4562-bcf3-cec25210a7e4] [runWorker=true]
[2022/12/30 18:53:34.829 +08:00] [INFO] [ddl.go:647] [“[ddl] start delRangeManager OK”] [“is a emulator”=false]
[2022/12/30 18:53:34.829 +08:00] [INFO] [manager.go:151] [“start campaign owner”] [ownerInfo=“[ddl] /tidb/ddl/fg/owner”]
[2022/12/30 18:53:34.829 +08:00] [INFO] [ddl_worker.go:171] [“[ddl] start DDL worker”] [worker=“worker 2, tp add index”]
[2022/12/30 18:53:34.829 +08:00] [INFO] [ddl_worker.go:171] [“[ddl] start DDL worker”] [worker=“worker 1, tp general”]
[2022/12/30 18:53:34.830 +08:00] [INFO] [env.go:101] [“[ddl-ingest] the lightning sorted dir”] [“data path:”=/tmp/tidb/tmp_ddl-44000]
[2022/12/30 18:53:34.830 +

| username: Billmay表妹 | Original post link

Please describe the problem in detail:

[TiDB Version]
[Reproduction Path] What operations were performed when the problem occurred
[Encountered Problem: Problem Phenomenon and Impact]
[Resource Configuration]

Please copy and paste the error message.

| username: TiDBer_vZ5xlHrG | Original post link

Pasted, FATAL information

| username: WalterWj | Original post link

Judging by the error, the cluster initialization did not succeed. Do you have enough space in your home directory?

| username: xfworld | Original post link

What configuration is manual deployment?

Playground is the local experience version…

| username: TiDBer_vZ5xlHrG | Original post link

That’s right, I installed it on the Kylin desktop version, and the total disk space is quite large. However, it defaulted to partitioning into several sections, and the partition for TiDB was almost full. I expanded the capacity and changed the directory, and it can now start successfully. Thank you very much!

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.