How to handle errors when upgrading TiDB to 7.5.0 and TiDB servers?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb升级到7.5.0, tidb servers升级报错了怎么处理

| username: zhanggame1

[TiDB Usage Environment] Test/
[TiDB Version] v.7.5.0
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Problem Phenomenon and Impact]
Using tiup cluster upgrade tidb-test v7.5.0 to upgrade to the tidb server component encountered an error. How should this be handled?


Checked the cluster status, it looks like this

tidb.txt (3.7 MB)

| username: 芮芮是产品 | Original post link

You should check the TiDB logs for the reasons why it failed to start.

| username: Billmay表妹 | Original post link

You translate the error message, where it pointed you to check the logs.

| username: zhanggame1 | Original post link

I have also encountered this problem. It seems to be related to the configuration of the TiKV node. You can try adjusting the raftstore and storage parameters to see if it helps.

| username: zhanggame1 | Original post link

The default value of tidb_enable_clustered_index is INT_ONLY, which means that only tables with integer primary keys will use clustered indexes. If you want to enable clustered indexes for all tables, you need to set this parameter to ON.

| username: 有猫万事足 | Original post link

The table creation SQL might have failed during the upgrade. Try manually creating it with the root user and see if it works.

| username: 芮芮是产品 | Original post link

Show the table.

| username: zhanggame1 | Original post link

After manually creating the table, what operations should be performed next? The current version hasn’t been upgraded yet, but the table was successfully created manually.

| username: 有猫万事足 | Original post link

Don’t worry, because the table creation statement I gave you is taken directly from bootstrap.go.

You see it has an IF NOT EXISTS. So if the upgrade doesn’t go through, restarting TiDB will cause the code in bootstrap.go to attempt the upgrade again, and there won’t be any major issues.

| username: 芮芮是产品 | Original post link

Awesome! Throwing confetti to celebrate!

| username: zhanggame1 | Original post link

After creating the table, the upgrade was successful. Version 8.0.11-TiDB-v7.5.0

tidb@center:~$ tiup cluster upgrade tidb-test v7.5.0
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.14.0/tiup-cluster upgrade tidb-test v7.5.0
Before the upgrade, it is recommended to read the upgrade guide at https://docs.pingcap.com/tidb/stable/upgrade-tidb-using-tiup and finish the preparation steps.
This operation will upgrade tidb v7.1.2 cluster tidb-test to v7.5.0:
will upgrade component           "tiflash" to "v7.5.0",
will upgrade component                "pd" to "v7.5.0",
will upgrade component              "tikv" to "v7.5.0",
will upgrade component              "tidb" to "v7.5.0",
will upgrade component        "prometheus" to "v7.5.0",
will upgrade component           "grafana" to "v7.5.0",
will upgrade component      "alertmanager" to "",
will upgrade component     "node-exporter" to "",
will upgrade component "blackbox-exporter" to "".
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster...
+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.208
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.211
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.210
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.210
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.209
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.212
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.221
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.220
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.211
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.100
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.100
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.207
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.212
+ [Parallel] - UserSSH: user=tidb, host=10.10.10.100
+ [ Serial ] - Download: component=prometheus, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=tidb, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=tikv, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=pd, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=tiflash, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=grafana, version=v7.5.0, os=linux, arch=amd64
+ [ Serial ] - Download: component=alertmanager, version=, os=linux, arch=amd64
download monitored
  - Download node_exporter: (linux/amd64) ... Done
  - Download blackbox_exporter: (linux/amd64) ... Done
+ [ Serial ] - Mkdir: host=10.10.10.212, directories='/tidb-data/pd-2379'
+ [ Serial ] - Mkdir: host=10.10.10.220, directories='/tidb-data/tiflash-9000'
+ [ Serial ] - Mkdir: host=10.10.10.221, directories='/tidb-data/tiflash-9000'
+ [ Serial ] - Mkdir: host=10.10.10.210, directories='/tidb-data/pd-2379'
+ [ Serial ] - Mkdir: host=10.10.10.211, directories='/tidb-data/pd-2379'
+ [ Serial ] - BackupComponent: component=pd, currentVersion=v7.1.2, remote=10.10.10.211:/tidb-deploy/pd-2379
+ [ Serial ] - CopyComponent: component=pd, version=v7.5.0, remote=10.10.10.211:/tidb-deploy/pd-2379 os=linux, arch=amd64
+ [ Serial ] - BackupComponent: component=pd, currentVersion=v7.1.2, remote=10.10.10.210:/tidb-deploy/pd-2379
+ [ Serial ] - BackupComponent: component=pd, currentVersion=v7.1.2, remote=10.10.10.212:/tidb-deploy/pd-2379
+ [ Serial ] - BackupComponent: component=tiflash, currentVersion=v7.1.2, remote=10.10.10.220:/tidb-deploy/tiflash-9000
+ [ Serial ] - BackupComponent: component=tiflash, currentVersion=v7.1.2, remote=10.10.10.221:/tidb-deploy/tiflash-9000
+ [ Serial ] - CopyComponent: component=pd, version=v7.5.0, remote=10.10.10.210:/tidb-deploy/pd-2379 os=linux, arch=amd64
+ [ Serial ] - CopyComponent: component=pd, version=v7.5.0, remote=10.10.10.212:/tidb-deploy/pd-2379 os=linux, arch=amd64
+ [ Serial ] - CopyComponent: component=tiflash, version=v7.5.0, remote=10.10.10.220:/tidb-deploy/tiflash-9000 os=linux, arch=amd64
+ [ Serial ] - CopyComponent: component=tiflash, version=v7.5.0, remote=10.10.10.221:/tidb-deploy/tiflash-9000 os=linux, arch=amd64
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.211, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/pd-2379.service, deploy_dir=/tidb-deploy/pd-2379, data_dir=[/tidb-data/pd-2379], log_dir=/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.210, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/pd-2379.service, deploy_dir=/tidb-deploy/pd-2379, data_dir=[/tidb-data/pd-2379], log_dir=/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.212, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/pd-2379.service, deploy_dir=/tidb-deploy/pd-2379, data_dir=[/tidb-data/pd-2379], log_dir=/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - Mkdir: host=10.10.10.207, directories='/tidb-data/tikv-20160'
+ [ Serial ] - BackupComponent: component=tikv, currentVersion=v7.1.2, remote=10.10.10.207:/tidb-deploy/tikv-20160
+ [ Serial ] - Mkdir: host=10.10.10.208, directories='/tidb-data/tikv-20160'
+ [ Serial ] - Mkdir: host=10.10.10.209, directories='/tidb-data/tikv-20160'
+ [ Serial ] - CopyComponent: component=tikv, version=v7.5.0, remote=10.10.10.207:/tidb-deploy/tikv-20160 os=linux, arch=amd64
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.221, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tiflash-9000.service, deploy_dir=/tidb-deploy/tiflash-9000, data_dir=[/tidb-data/tiflash-9000], log_dir=/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - BackupComponent: component=tikv, currentVersion=v7.1.2, remote=10.10.10.208:/tidb-deploy/tikv-20160
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.220, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tiflash-9000.service, deploy_dir=/tidb-deploy/tiflash-9000, data_dir=[/tidb-data/tiflash-9000], log_dir=/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - CopyComponent: component=tikv, version=v7.5.0, remote=10.10.10.208:/tidb-deploy/tikv-20160 os=linux, arch=amd64
+ [ Serial ] - BackupComponent: component=tikv, currentVersion=v7.1.2, remote=10.10.10.209:/tidb-deploy/tikv-20160
+ [ Serial ] - CopyComponent: component=tikv, version=v7.5.0, remote=10.10.10.209:/tidb-deploy/tikv-20160 os=linux, arch=amd64
+ [ Serial ] - Mkdir: host=10.10.10.210, directories=''
+ [ Serial ] - BackupComponent: component=tidb, currentVersion=v7.1.2, remote=10.10.10.210:/tidb-deploy/tidb-4000
+ [ Serial ] - Mkdir: host=10.10.10.211, directories=''
+ [ Serial ] - BackupComponent: component=tidb, currentVersion=v7.1.2, remote=10.10.10.211:/tidb-deploy/tidb-4000
+ [ Serial ] - CopyComponent: component=tidb, version=v7.5.0, remote=10.10.10.211:/tidb-deploy/tidb-4000 os=linux, arch=amd64
+ [ Serial ] - CopyComponent: component=tidb, version=v7.5.0, remote=10.10.10.210:/tidb-deploy/tidb-4000 os=linux, arch=amd64
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.207, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tikv-20160.service, deploy_dir=/tidb-deploy/tikv-20160, data_dir=[/tidb-data/tikv-20160], log_dir=/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.208, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tikv-20160.service, deploy_dir=/tidb-deploy/tikv-20160, data_dir=[/tidb-data/tikv-20160], log_dir=/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.211, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tidb-4000.service, deploy_dir=/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.210, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tidb-4000.service, deploy_dir=/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.209, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tikv-20160.service, deploy_dir=/tidb-deploy/tikv-20160, data_dir=[/tidb-data/tikv-20160], log_dir=/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - Mkdir: host=10.10.10.212, directories=''
+ [ Serial ] - BackupComponent: component=tidb, currentVersion=v7.1.2, remote=10.10.10.212:/tidb-deploy/tidb-4000
+ [ Serial ] - Mkdir: host=10.10.10.100, directories='/tidb-data/prometheus-9090'
+ [ Serial ] - CopyComponent: component=tidb, version=v7.5.0, remote=10.10.10.212:/tidb-deploy/tidb-4000 os=linux, arch=amd64
+ [ Serial ] - Mkdir: host=10.10.10.100, directories=''
+ [ Serial ] - BackupComponent: component=grafana, currentVersion=v7.1.2, remote=10.10.10.100:/tidb-deploy/grafana-3000
+ [ Serial ] - CopyComponent: component=grafana, version=v7.5.0, remote=10.10.10.100:/tidb-deploy/grafana-3000 os=linux, arch=amd64
+ [ Serial ] - Mkdir: host=10.10.10.100, directories='/tidb-data/alertmanager-9093'
+ [ Serial ] - BackupComponent: component=prometheus, currentVersion=v7.1.2, remote=10.10.10.100:/tidb-deploy/prometheus-9090
+ [ Serial ] - CopyComponent: component=prometheus, version=v7.5.0, remote=10.10.10.100:/tidb-deploy/prometheus-9090 os=linux, arch=amd64
+ [ Serial ] - BackupComponent: component=alertmanager, currentVersion=v7.1.2, remote=10.10.10.100:/tidb-deploy/alertmanager-9093
+ [ Serial ] - CopyComponent: component=alertmanager, version=, remote=10.10.10.100:/tidb-deploy/alertmanager-9093 os=linux, arch=amd64
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.212, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/tidb-4000.service, deploy_dir=/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.100, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/alertmanager-9093.service, deploy_dir=/tidb-deploy/alertmanager-9093, data_dir=[/tidb-data/alertmanager-9093], log_dir=/tidb-deploy/alertmanager-9093/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache
+ [ Serial ] - InitConfig: cluster=tidb-test, user=tidb, host=10.10.10.100, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/config-cache/grafana-3000.service, deploy_dir=/tidb-deploy/grafana-3000, data_dir=[], log_dir=/tidb-deploy/grafana-3000/log, cache_dir=/home/t
| username: tidb菜鸟一只 | Original post link

Sure, as long as it’s resolved. Just make a note of it to prevent encountering the same issue next time.

| username: dba远航 | Original post link

Question about the CREATE TABLE statement

| username: TI表弟 | Original post link

Which version did you upgrade from?

| username: Jellybean | Original post link

There seems to be an issue during the upgrade process, which feels somewhat abnormal.
Did the original poster perform any additional operations while executing?

| username: 有猫万事足 | Original post link

It is indeed quite strange. However, to analyze why it didn’t succeed before, we would need the logs from the previous upgrade period.

| username: zhanggame1 | Original post link

There’s nothing much to do, I’ve also sent the TiDB logs, you can take a look.

| username: zhanggame1 | Original post link

Upgraded from 7.1.2

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.