Upgrading TiDB from v6.1.2 to 6.5.0: TiDB Node Port 4000 Failure ["unexpected error"] [error="TiKV version must be greater than or equal to 6.2.0-alpha"]

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb从v6.1.2升级到6.5.0 tidb节点提示端口4000失败[“unexpected error”] [error=“TiKV version must greater than or equal to 6.2.0-alpha”]

| username: TI表弟

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version] V6.1.2
[Reproduction Path] tiup cluster upgrade tidb-test v6.5.0
Attempted offline upgrade afterward, indicated success, but the TiDB node could not start anymore.
[Encountered Issue: Problem Phenomenon and Impact] TiKV and PD can upgrade normally, but TiDB upgrade prompts Error: failed to restart: 10.125.76.252 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

2023-02-09T19:10:16.738+0800	DEBUG	TaskFinish	{"task": "StartCluster", "error": "failed to start tidb: failed to start: 10.125.76.252 tidb-4000.service, please check the instance's log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s", "errorVerbose": "timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.76.252 tidb-4000.service, please check the instance's log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb"}

Port 4000 is not occupied

Scaling down also reports an error

[2023/02/09 20:05:33.521 +08:00] [FATAL] [terror.go:300] ["unexpected error"] [error="TiKV version must greater than or equal to 6.2.0-alpha"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:312\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:312\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

TiKV error log

| username: magic | Original post link

Check the TiKV version,
Upgrade documentation:

| username: Billmay表妹 | Original post link

Refer to this solution: “在启动TIDB的时候 4000端口无法起来 - TiDB 的问答社区

| username: TI表弟 | Original post link

Not a port issue, not a firewall issue.

| username: Billmay表妹 | Original post link

Try searching for related cases on asktug and see if you can find the answer yourself.

| username: TI表弟 | Original post link

There are no similar cases.

| username: 人如其名 | Original post link

According to the TiDB error log, it seems that TiKV did not upgrade successfully.

| username: WalterWj | Original post link

It seems so. I recommend checking the TiKV version first.

| username: 裤衩儿飞上天 | Original post link

Take a look at TiKV.

| username: huanglao2002 | Original post link

It is recommended to describe the process:

  1. How is the upgrade evaluation?
  2. What is the result of the pre-upgrade check using tiup cluster check <cluster-name> --cluster?
| username: 考试没答案 | Original post link

Use tiup cluster display tidb-test to check the cluster status. Desensitized.

| username: 考试没答案 | Original post link

The --force parameter forces the action. I see this is a test environment.

| username: TI表弟 | Original post link

Sorry, I can’t assist with that.

| username: TI表弟 | Original post link

The default value of tidb_gc_life_time is 10m, which means that the data can be rolled back within 10 minutes. If you need to roll back data beyond this time, you need to adjust this parameter.

| username: TI表弟 | Original post link

The TiKV version is already 6.5.0, and the TiKV error message is as follows:

| username: 考试没答案 | Original post link

Post the core part of this log and take a look.

| username: 考试没答案 | Original post link

Can scaling out to add a TiDB instance be successful?

| username: TI表弟 | Original post link

The same error with port 4000, even on a machine that has never had TiDB installed and where port 4000 should not be occupied.

| username: TI表弟 | Original post link

2023-02-10T13:41:05.548+0800 DEBUG retry error {error: operation timed out after 2m0s}
2023-02-10T13:41:05.549+0800 DEBUG TaskFinish {task: Start new instances, error: failed to start tidb: failed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s, errorVerbose: timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb}
2023-02-10T13:41:05.549+0800 INFO Execute command finished {code: 1, error: failed to start tidb: failed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s, errorVerbose: timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb}
| username: TI表弟 | Original post link

The version has been upgraded to 6.5.0.