Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tidb6.3 升级报错
|
username: tidb狂热爱好者
【TiDB Usage Environment】Production Environment / Testing / Poc
【TiDB Version】
【Encountered Issues】
The originally deployed TiDB had 1 PD, 1 KV, and 1 TiDB. I first upgraded it to 1 PD, 3 KV, and 1 TiDB, then upgraded TiDB from 6.2 to 6.3, and it resulted in an error.
【Reproduction Path】Operations performed that led to the issue
【Issue Phenomenon and Impact】
2022-10-13T16:53:40.879+0800 ERROR CheckPoint {“instance”: “10.8.35.62:20160”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”, “hash”: “1a4714d7146fa85240a1ff4ef7451df719e0b4f0”, “func”: “github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance”, “hit”: false}
2022-10-13T16:53:40.880+0800 DEBUG setting replication config: leader-schedule-limit=4
2022-10-13T16:53:40.880+0800 ERROR failed counting leader {“host”: “10.8.35.62:20160”, “status addr”: “http://10.8.35.62:20180/metrics”, “error”: “GET request for URL "http://10.8.35.62:20180/metrics\” returned HTTP status 503 Service Unavailable"}
2022-10-13T16:53:40.881+0800 DEBUG setting replication config: region-schedule-limit=2048
2022-10-13T16:53:40.882+0800 DEBUG TaskFinish {“task”: “UpgradeCluster”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”}
2022-10-13T16:53:40.882+0800 INFO Execute command finished {“code”: 1, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”}
【Attachments】
Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.
|
username: 
|
username: 
|
username: