TiDB Cluster Upgrade to Version 7.1.3: TiDB Instance Fails to Start

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb集群升级到tidb7.1.3版本tidb实例启动不了

| username: wluckdog

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] Upgraded from TiDB version v6.5.0 to TiDB 7.1.3, upgrade completed successfully
[Reproduction Path] What operations were performed that led to the issue
[Encountered Issue: Problem Phenomenon and Impact] TiDB instance fails to start, TiKV and PD start normally

bc6df95af0fe2c4b82dda4e4f078ce6
bc6df95af0fe2c4b82dda4e4f078ce61350×521 19.6 KB

[Resource Configuration]

[Attachments: Screenshots/Logs/Monitoring]

2
| username: Billmay表妹 | Original post link

What operations have you performed? Can you list them roughly?

3
| username: dba远航 | Original post link

TiDB encountered an exception while retrieving metadata.

4
| username: wluckdog | Original post link
  1. Shut down the cluster
  2. Offline upgrade to 7.1.3
  3. Modified TiKV memory parameters to reduce memory usage
  4. Started the cluster, PD and TiKV started successfully, but TiDB node failed to start
  5. Tried to start TiDB role individually, but it failed to start
  6. Tried to start TiDB as a single instance, but it failed to start
  7. Tried to add a TiDB node, added successfully, but failed to start
5
| username: xingzhenxiang | Original post link

Take a look at the TiDB configuration file to see if there is NUMA binding.

6
| username: wluckdog | Original post link

After binding NUMA, I tried to remove the NUMA binding core of a TiDB instance, but it still couldn’t start.

7
| username: wluckdog | Original post link

Normally, these warnings should not affect the startup, and currently, there are no issues. These are resource controls that need to be configured after TiDB starts. We also did not perform any DDL operations before the upgrade, so we did not check admin show ddl job.

8
| username: 江湖故人 | Original post link

Check the official documentation to see if there are any missing details.
Using TiUP to Upgrade TiDB | PingCAP Documentation Center

9
| username: dba-kit | Original post link

It looks like a system table is missing, causing it to fail to start. Were there any errors reported when upgrading the tidb-server? However, when starting, if the tidb-server detects that the schema_version is incorrect, it should theoretically create it automatically. Or you could try using tiup cluster reload -R tidb?

10
| username: dba-kit | Original post link

Execute tiup cluster display to check the current cluster status.
Also, execute tiup cluster audit to see the actual execution order and check if there are any non-standard operations.

11
| username: ffeenn | Original post link

There is an issue with the schema_version cache record. It is recommended to back up the data first, create a new cluster, and restore it.

12
| username: ffeenn | Original post link

It’s called a cache table, that’s what I call it :joy:

13
| username: 随缘天空 | Original post link

It seems that the issue might be caused by a missing table. Try connecting to the database to check.

14
| username: TiDBer_5Vo9nD1u | Original post link

Purchase the service and seek official technical support.