Failed to Start Cluster on Port 4000 After Initialization

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 初始化后启动集群4000端口启动失败

| username: TiDBer_CkS2lbTx

Failed to start the cluster
tiup cluster start tidb_online_10100 --init
Error: failed to start tidb: failed to start: 172.19.2.1 tidb-4000.service, please check the instance’s log(/data/chj/app/tidb/deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s


The following error is prompted, may I ask if there is a problem with my configuration?
[2024/02/06 17:50:28.191 +08:00] [ERROR] [runaway.go:145] [“try to get new runaway watch”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:28.191 +08:00] [WARN] [runaway.go:172] [“get runaway watch record failed”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:29.191 +08:00] [ERROR] [runaway.go:145] [“try to get new runaway watch”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:29.191 +08:00] [WARN] [runaway.go:172] [“get runaway watch record failed”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:30.191 +08:00] [ERROR] [runaway.go:145] [“try to get new runaway watch”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:30.191 +08:00] [WARN] [runaway.go:172] [“get runaway watch record failed”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:31.191 +08:00] [ERROR] [runaway.go:145] [“try to get new runaway watch”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]
[2024/02/06 17:50:31.191 +08:00] [WARN] [runaway.go:172] [“get runaway watch record failed”] [error=“[schema:1146]Table ‘mysql.tidb_runaway_watch’ doesn’t exist”]

| username: tidb菜鸟一只 | Original post link

Was your cluster upgraded? Why is this table missing?

| username: TiDBer_CkS2lbTx | Original post link

New installation, first time installation, no tables, how can this be resolved?

| username: 江湖故人 | Original post link

Try creating it manually.
Try upgrading to TiDB 7.5.0, how to handle errors when upgrading TiDB servers - :ringer_planet: TiDB Technical Issues / Upgrades - TiDB Q&A Community (asktug.com)

| username: TiDBer_CkS2lbTx | Original post link

How to create it? I guess there is a problem with a certain configuration in the configuration file. It starts up after I clear it.

| username: 哈喽沃德 | Original post link

Have you ever deleted this table?

| username: TiDBer_5Vo9nD1u | Original post link

It is estimated to be accidentally deleted.

| username: TiDBer_CkS2lbTx | Original post link

No, this is a newly deployed cluster on a blank machine. There are no old tables or table deletion operations. The instances haven’t even started yet.

| username: 江湖故人 | Original post link

That post also reported that this system table does not exist, just create it yourself.

CREATE TABLE IF NOT EXISTS mysql.tidb_runaway_watch_done (
    id BIGINT(20) NOT NULL AUTO_INCREMENT PRIMARY KEY,
    record_id BIGINT(20) NOT NULL,
    resource_group_name VARCHAR(32) NOT NULL,
    start_time DATETIME(6) NOT NULL,
    end_time DATETIME(6),
    watch BIGINT(10) NOT NULL,
    watch_text TEXT NOT NULL,
    source VARCHAR(512) NOT NULL,
    action BIGINT(10),
    done_time TIMESTAMP(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
| username: 江湖故人 | Original post link

Which configuration file? Can you share your experience?

| username: TiDBer_5Vo9nD1u | Original post link

Is it possible that the port is already in use?

| username: TiDBer_CkS2lbTx | Original post link

No, mine are all new machines, I’ve checked them all.

| username: TiDBer_CkS2lbTx | Original post link

Unable to create a database or table due to configuration file errors. My cluster hasn’t started up, and the tidb-server entry point at port 4000, along with the authentication, SQL syntax parsing, and the front end of the optimizer, haven’t started either. It’s impossible to create a table.

| username: Jellybean | Original post link

How exactly did you clear the directory, and did you restart the cluster afterward?

Can you specifically share your steps?

| username: TiDBer_CkS2lbTx | Original post link

Delete the cluster, in the yaml file, only add the IPs and data paths for tidb-server, tikv, pd, and tiflash, leave everything else out. Redeploy, and once it’s up, add the configurations back.

| username: dba远航 | Original post link

Then check the configuration file, there should be errors causing it.

| username: zhang_2023 | Original post link

Accidentally deleted the table, right?

| username: zhaokede | Original post link

The original poster already mentioned that the newly deployed database has never been started, so it’s impossible to delete the table.
Can any expert explain which configuration issue might cause this phenomenon, so we can avoid this pitfall when modifying configurations in the future?

| username: zhanggame1 | Original post link

I’ve seen this issue on the forum multiple times. Just destroy the cluster and reinstall it.

| username: TiDBer_CkS2lbTx | Original post link

It should be one of the configurations here. There are too many for me to identify one by one. The blank configuration can start, and then you can use tiup cluster edit-config xxx to add and reload, then restart, and it should work.