Failed to Start TiDB on Port 4000

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 4000端口启动失败

| username: TiDBer_JN14fWU1

【TiDB Environment】Ubuntu20 virtual machine test environment
【TiDB Version】6.1.0
【Problem Encountered】Failed to start TiDB on port 4000
Error: failed to start tidb: failed to start: 192.168.186.131 tidb-4000.service, please check the instance’s log (/data/tidb-deploy/tidb-4000/log) for more details.: timed out waiting for port 4000 to be started after 2m0s
【Reproduction Path】PD and TiKV can start successfully, but TiDB fails to start. The error description and logs are as follows.

Configuration:

pd_servers:
 - host: 192.168.186.131

tidb_servers:
 - host: 192.168.186.131

tikv_servers:
 - host: 192.168.186.131
   port: 20160
   status_port: 20180
   config:
     server.labels: { host: "logic-host-1" }

 - host: 192.168.186.132
   port: 20161
   status_port: 20181
   config:
     server.labels: { host: "logic-host-2" }

 - host: 192.168.186.133
   port: 20162
   status_port: 20182
   config:
     server.labels: { host: "logic-host-3" }

tiflash_servers:
 - host: 192.168.186.131

monitoring_servers:
 - host: 192.168.186.131

grafana_servers:
 - host: 192.168.186.131

Error-related logs are as follows:
[2022/10/03 13:40:34.352 +00:00] [INFO] [session.go:3274] [“CRUCIAL OPERATION”] [conn=0] [schemaVersion=19] [cur_db=mysql] [sql=“CREATE TABLE IF NOT EXISTS mysql.bind_info (
\t\toriginal_sql TEXT NOT NULL,
\t\tbind_sql TEXT NOT NULL,
\t\tdefault_db TEXT NOT NULL,
\t\tstatus TEXT NOT NULL,
\t\tcreate_time TIMESTAMP(3) NOT NULL,
\t\tupdate_time TIMESTAMP(3) NOT NULL,
\t\tcharset TEXT NOT NULL,
\t\tcollation TEXT NOT NULL,
\t\tsource VARCHAR(10) NOT NULL DEFAULT ‘unknown’,
\t\tINDEX sql_index(original_sql(700),default_db(68)) COMMENT "accelerate the speed when add global binding query",
\t\tINDEX time_index(update_time) COMMENT "accelerate the speed when querying with last update time"
\t) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;”] [user=]
[2022/10/03 13:41:56.896 +00:00] [WARN] [backoff.go:158] [“tikvDiskFull backoffer.maxSleep 80000ms is exceeded, errors:
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:41.891026592Z
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:46.892636767Z
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:51.895488103Z
longest sleep type: tikvDiskFull, time: 82500ms”]
[2022/10/03 13:41:56.896 +00:00] [WARN] [backoff.go:158] [“regionMiss backoffer.maxSleep 80000ms is exceeded, errors:
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:41.891026592Z
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:46.892636767Z
tikv disk full: store_id:1 reason:"propose failed: tikv disk full, cmd diskFullOpt=NotAllowedOnFull, leader diskUsage=AlmostFull" ctx: region ID: 3, meta: id:3 start_key:"t\200\000\000\000\000\000\000%" region_epoch:<conf_ver:5 version:18 > peers:<id:4 store_id:1 > peers:<id:51 store_id:2 > peers:<id:77 store_id:7 > , peer: id:4 store_id:1 , addr: 192.168.186.131:20160, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2022-10-03T13:41:51.895488103Z
longest sleep type: tikvDiskFull, time: 82500ms”]
[2022/10/03 13:41:56.896 +00:00] [WARN] [session.go:881] [“can not retry txn”] [label=internal] [error=“tikv disk full”] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=true] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2022/10/03 13:41:56.896 +00:00] [WARN] [session.go:897] [“commit failed”] [“finished txn”=“Txn{state=invalid}”] [error=“tikv disk full”]
[2022/10/03 13:41:56.896 +00:00] [WARN] [session.go:1941] [“run statement failed”] [schemaVersion=19] [error=“tikv disk full”] [session=“{
"currDBName": "mysql",
"id": 0,
"status": 2,
"strictMode": true,
"user": null
}”]
[2022/10/03 13:41:56.896 +00:00] [FATAL] [bootstrap.go:2088] [“mustExecute error”] [error=“tikv disk full”] [stack=“github.com/pingcap/tidb/session.mustExecute
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:2088
github.com/pingcap/tidb/session.insertBuiltinBindInfoRow
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1461
github.com/pingcap/tidb/session.initBindInfoTable
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1457
github.com/pingcap/tidb/session.doDDLWorks
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1941
github.com/pingcap/tidb/session.bootstrap
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:445
github.com/pingcap/tidb/session.runInBootstrapSession
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2941
github.com/pingcap/tidb/session.BootstrapSession
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2829
main.createStoreAndDomain
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:296
main.main
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:202
runtime.main
\t/usr/local/go/src/runtime/proc.go:250”]

2
| username: srstack | Original post link

It looks like the TiKV disk capacity check is insufficient?

3
| username: TiDBer_JN14fWU1 | Original post link

I have set my virtual machine to 20G. I see that it is currently using just over 6G. Could it still run out of capacity?

4
| username: 啦啦啦啦啦 | Original post link

The remaining space is less than the reserved space, right? Try increasing it a bit more.

5
| username: Raymond | Original post link

You can add some space to expand.

6
| username: 特雷西-迈克-格雷迪 | Original post link

Check if the mounted directory of the data directory in TiKV has enough space.

7
| username: zhouzeru | Original post link
8
| username: TiDBer_CEVsub | Original post link

Insufficient space alert

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.