After upgrading TiDB from V6.5.1 to V7.5.0, the TiDB node fails to start

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 从V6.5.1升级到v7.5.0后,tidb节点无法启动

| username: Hacker_j9odkS8a

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]

| username: 像风一样的男子 | Original post link

Is it just TiDB that hasn’t started in the entire cluster? Is PD functioning normally?

| username: 这里介绍不了我 | Original post link

There is a post quite similar to your situation.

| username: Hacker_j9odkS8a | Original post link

PD and TiKV are both functioning normally, but TiDB fails to start and then stops.

| username: 像风一样的男子 | Original post link

Do you have the logs when TiDB starts? Are there any error messages?

| username: 这里介绍不了我 | Original post link

There’s also this issue Schema version update failure may cause DDL job to get stuck · Issue #43755 · pingcap/tidb · GitHub

| username: ffeenn | Original post link

Post the error log.

| username: Hacker_j9odkS8a | Original post link

It’s the consistent screenshot above.

| username: Hacker_j9odkS8a | Original post link

How to solve the binlog issue?

| username: 像风一样的男子 | Original post link

You don’t need binlog, so turn off the binlog in the configuration file.

Modify Cluster Configuration

tiup cluster edit-config ${cluster-name}
Execute the reload command to roll out the configuration and restart the corresponding components:
tiup cluster reload tidb-feng [-N ] [-R ]

| username: 有猫万事足 | Original post link

The most important thing at this stage is not to panic.

From the first log you provided, it’s all info, and I feel that this is not the root cause of the issue.

Then from the second log you provided,

there are no available pump nodes. This means that the pump is not up, not that TiDB is not up.

TiDB Binlog is no longer maintained in subsequent versions. Only TiCDC is maintained. Therefore, if you upgrade to 7.5, you will not be able to use TiDB Binlog.

  • Starting from TiDB v7.5.0, the data synchronization function of the TiDB Binlog component is no longer supported. It is strongly recommended to use TiCDC as an alternative for data synchronization.
| username: Hacker_j9odkS8a | Original post link

After modifying the binlog configuration

| username: 有猫万事足 | Original post link

During the upgrade, a TiDB system table was not successfully created.

If possible, log in to a TiDB node as root and execute the following statement to create this table.

USE mysql;
CREATE TABLE IF NOT EXISTS mysql.tidb_runaway_watch_done (
    id BIGINT(20) NOT NULL AUTO_INCREMENT PRIMARY KEY,
    record_id BIGINT(20) NOT NULL,
    resource_group_name VARCHAR(32) NOT NULL,
    start_time DATETIME(6) NOT NULL,
    end_time DATETIME(6),
    watch BIGINT(10) NOT NULL,
    watch_text TEXT NOT NULL,
    source VARCHAR(512) NOT NULL,
    action BIGINT(10),
    done_time TIMESTAMP(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
| username: tidb菜鸟一只 | Original post link

Check if these two tables exist now, and if not, create them manually.

| username: Hacker_j9odkS8a | Original post link

Is there a way to create the table in another environment if TiDB is not started?

| username: 有猫万事足 | Original post link

This is very troublesome.

Just provide the tidb.log and tidb_stderr.log logs from the tidb log directory. At the very least, it’s hard to say that the part you screenshot is the root cause of the issue.

| username: Hacker_j9odkS8a | Original post link

tidb.log (189.0 KB)

| username: tidb菜鸟一只 | Original post link

The absence of this table shouldn’t cause TiDB to fail to start, right? Check if there are any other error messages? Or try restarting the tidb-server.

| username: Hacker_j9odkS8a | Original post link

There are TiDB logs above.

| username: Hacker_j9odkS8a | Original post link

You cannot create a table using tidb-ctl when TiDB is not running.