Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tidb upgrade to v6.6.0 遇到tikv Still waitting for 1 store leaders to transfer
[TiDB Usage Environment] Test/PoC
[TiDB Version] v6.6.0
[Reproduction Path] Upgrade TiDB cluster from v6.4.0 to v6.6.0
[Encountered Problem: Issue Phenomenon and Impact]
During the upgrade of TiKV, an exception “Still waiting for 1 store leaders to transfer” was prompted, and eventually, TiKV was restarted.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
Operation Records
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster list
tiup is checking updates for component cluster …
Starting component cluster
: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster list
Name User Version Path PrivateKey
tidb-cluster tidb v6.4.0 /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster upgrade tidb-cluster v6.6.0
tiup is checking updates for component cluster …
Starting component cluster
: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster upgrade tidb-cluster v6.6.0
Before the upgrade, it is recommended to read the upgrade guide at Upgrade TiDB Using TiUP | PingCAP Docs and finish the preparation steps.
This operation will upgrade tidb v6.4.0 cluster tidb-cluster to v6.6.0.
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster…
- [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa.pub
- [Parallel] - UserSSH: user=tidb, host=192.168.2.21
- [Parallel] - UserSSH: user=tidb, host=192.168.2.22
- [Parallel] - UserSSH: user=tidb, host=192.168.2.22
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.21
- [Parallel] - UserSSH: user=tidb, host=192.168.2.22
- [Parallel] - UserSSH: user=tidb, host=192.168.2.21
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.22
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.151
- [Parallel] - UserSSH: user=tidb, host=192.168.2.21
- [ Serial ] - Download: component=prometheus, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=pd, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=tiflash, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=tikv, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=tidb, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=grafana, version=v6.6.0, os=linux, arch=amd64
- [ Serial ] - Download: component=alertmanager, version=, os=linux, arch=amd64
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/pd-2379’
- [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tiflash-9000’
- [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tiflash-9000’
- [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/pd-2379’
- [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/pd-2379’
- [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000
- [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379
- [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379
- [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379
- [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000
- [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
- [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/tikv-20160’
- [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
- [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
- [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tikv-20160’
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tikv-20160’
- [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
- [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000
- [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
- [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
- [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.21, directories=‘’
- [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000
- [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
- [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.22, directories=‘’
- [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/prometheus-9090’
- [ Serial ] - BackupComponent: component=prometheus, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/prometheus-9090
- [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
- [ Serial ] - BackupComponent: component=grafana, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/grafana-3000
- [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/alertmanager-9093’
- [ Serial ] - BackupComponent: component=alertmanager, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093
- [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
- [ Serial ] - CopyComponent: component=alertmanager, version=, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093 os=linux, arch=amd64
- [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/alertmanager-909