Failed to Start TiDB

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 启动TIDB失败

| username: TiDBer_8PQqVTE6

【TiDB Usage Environment】Production / Testing
【TiDB Version】v6.1.0
【Encountered Problem】TiDB failed to start
topology.yaml:

pd_servers:
  - host: 172.17.0.154

tidb_servers:
  - host: 172.17.0.154

tikv_servers:
  - host: 172.17.0.155
  - host: 172.17.0.156
  - host: 172.17.0.157

tiflash_servers:
  - host: 172.17.0.158
    data_dir: /tidb-data/tiflash-9000
    deploy_dir: /tidb-deploy/tiflash-9000

monitoring_servers:
  - host: 172.17.0.159

grafana_servers:
  - host: 172.17.0.159

alertmanager_servers:
  - host: 172.17.0.159

Error: failed to start tidb: failed to start: 172.17.0.154 tidb-4000.service, please check the instance’s log (/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

【Problem Phenomenon and Impact】
log
Execute command finished {“code”: 1, “error”: “failed to start tidb: failed to start: 172.17.0.154 tidb-4000.service, please check the instance’s log (/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s”, “errorVerbose”: “timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:119\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:151\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:405\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:534\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to start: 172.17.0.154 tidb-4000.service, please check the instance’s log (/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb”}

tidb.log error information:
[FATAL] [session.go:3052] [“check bootstrapped failed”] [error=“context deadline exceeded”] [stack=“github.com/pingcap/tidb/session.getStoreBootstrapVersion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3052\ngithub.com/pingcap/tidb/session.BootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2827\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:296\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:202\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]
【Attachment】

| username: TiDBer_8PQqVTE6 | Original post link

The firewall has been turned off.
image

Port 4000
image

| username: xfworld | Original post link

Please check the instance’s log (/tidb-deploy/tidb-4000/log)

Check the logs.

| username: wuxiangdong | Original post link

Does this pass the tiup cluster check?

| username: gary | Original post link

  1. Check the deployment and data directory permissions.
  2. Use the startup script to start TiDB and check the logs for specific errors.
| username: Hi70KG | Original post link

Check the details using tiup cluster check ./topology.yaml --apply --user root -p.

| username: TiDBer_8PQqVTE6 | Original post link

tidb.log error message:
[FATAL] [session.go:3052] [“check bootstrapped failed”] [error=“context deadline exceeded”] [stack=“github.com/pingcap/tidb/session.getStoreBootstrapVersion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3052\ngithub.com/pingcap/tidb/session.BootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2827\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:296\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:202\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]

| username: xfworld | Original post link

Check bootstrapped failed… The inspection didn’t pass…

| username: wuxiangdong | Original post link

Run tiup cluster check --apply to fix it.

| username: wisdom | Original post link

Check the configuration.

| username: TiDBer_8PQqVTE6 | Original post link

Hello, after adding the data disk with the EXT4 file system and setting the mount parameters on the machine, the check still says “mount point / does not have ‘nodelalloc’ option set”.

| username: TiDBer_8PQqVTE6 | Original post link

Hello, after adding the data disk with the EXT4 file system on the machine and setting the mount parameters, the check still says “mount point / does not have ‘nodelalloc’ option set.”

image

image

image

| username: xfworld | Original post link

It seems like it didn’t mount properly. Try writing data to /data1, /data2, and /data3.

| username: TiDBer_8PQqVTE6 | Original post link

Using the mount command to check, it is there.
image
Disk status
image

| username: xfworld | Original post link

You can only check this yourself more often, and after modifying the configuration, use tiup cluster check to perform a configuration check.

| username: TiDBer_8PQqVTE6 | Original post link

Master, so where is the problem with my configuration?

| username: xfworld | Original post link

You should look at the result of the command first, and then check the configuration based on this result.

| username: TiDBer_8PQqVTE6 | Original post link

After checking, it says the mount point / does not have the ‘nodelalloc’ option set.

| username: xfworld | Original post link

Edit the /etc/fstab file to add the nodelalloc and noatime options. The configuration is as follows:


You need to compare it with your own environment and follow the documentation, otherwise, it will still fail the check (PS the above is just an example)

| username: 会飞的土拨鼠 | Original post link

You can check the ports and the paths of the components in the configuration file; the disk mounting is different from the usual usage.