TiDB upgrade to v6.6.0 encountered TiKV still waiting for 1 store leaders to transfer

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb upgrade to v6.6.0 遇到tikv Still waitting for 1 store leaders to transfer

| username: lmdb

[TiDB Usage Environment] Test/PoC
[TiDB Version] v6.6.0
[Reproduction Path] Upgrade TiDB cluster from v6.4.0 to v6.6.0
[Encountered Problem: Issue Phenomenon and Impact]
During the upgrade of TiKV, an exception “Still waiting for 1 store leaders to transfer” was prompted, and eventually, TiKV was restarted.
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

Operation Records

[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster list
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster list
Name User Version Path PrivateKey


tidb-cluster tidb v6.4.0 /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster upgrade tidb-cluster v6.6.0
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster upgrade tidb-cluster v6.6.0
Before the upgrade, it is recommended to read the upgrade guide at Upgrade TiDB Using TiUP | PingCAP Docs and finish the preparation steps.
This operation will upgrade tidb v6.4.0 cluster tidb-cluster to v6.6.0.
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster…

  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.21
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.22
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.22
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.21
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.22
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.21
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.22
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.151
  • [Parallel] - UserSSH: user=tidb, host=192.168.2.21
  • [ Serial ] - Download: component=prometheus, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=pd, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tiflash, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tikv, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tidb, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=grafana, version=v6.6.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=alertmanager, version=, os=linux, arch=amd64
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/pd-2379’
  • [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tiflash-9000’
  • [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tiflash-9000’
  • [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/pd-2379’
  • [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/pd-2379’
  • [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379
  • [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000
  • [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/tikv-20160’
  • [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tikv-20160’
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tikv-20160’
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000
  • [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.21, directories=‘’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000
  • [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.22, directories=‘’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/prometheus-9090’
  • [ Serial ] - BackupComponent: component=prometheus, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/prometheus-9090
  • [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
  • [ Serial ] - BackupComponent: component=grafana, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/grafana-3000
  • [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/alertmanager-9093’
  • [ Serial ] - BackupComponent: component=alertmanager, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093
  • [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=, log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
  • [ Serial ] - CopyComponent: component=alertmanager, version=, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/alertmanager-909
| username: lmdb | Original post link

The rolling upgrade method is adopted, and attention is also paid to the upgrade instructions related to TiKV. * The rolling upgrade will upgrade all components one by one. During the TiKV upgrade, all leaders on TiKV will be moved away one by one before stopping the TiKV instance. The default timeout is 5 minutes (300 seconds), after which the instance will be stopped directly.

I feel that the TiKV leader of this TiDB cluster is not being moved away normally.

| username: tidb菜鸟一只 | Original post link

Check the TiKV logs of the corresponding node.

| username: xfworld | Original post link

Why not test with 6.5.X? Why upgrade from 6.4.x to 6.6.0… Want to experience new features?

| username: 海石花47 | Original post link

Currently running version 6.1 online, afraid to upgrade directly, worried about encountering issues…

| username: ffeenn | Original post link

The upgrade is complete. There are no anomalies, everything is normal.

| username: lmdb | Original post link

Yes, doing tests. Restarting TiKV should be a normal situation.

| username: xfworld | Original post link

7.0.0 is coming soon~ :upside_down_face:

| username: lmdb | Original post link

It’s a bit overwhelming to experience, but I support it.

| username: BraveChen | Original post link

Isn’t this normal? Rolling restart.

| username: h5n1 | Original post link

The logs are normal operation logs, waiting for the leader to migrate to other nodes. The slow migration process may be due to the lagging apply of the region replica to be migrated.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.