Issues Related to Upgrading a Cluster with TiFlash Component from v5.1.4 to v5.4.3

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 含有TiFlash组建的集群从v5.1.4升级到v5.4.3相关问题

| username: TiDBer_Lee

[TiDB Usage Environment] Production Environment
[TiDB Version] v5.1.4
[Process] The cluster version is v5.1.4, and the upgrade version is v5.4.3. According to the official documentation, the steps are as follows:

  • tiup cluster stop -R tiflash

  • tiup cluster upgrade --offline

  • tiup cluster reload
    Issue: An error occurred during the second step, indicating that the parameter --offline is for upgrading an already stopped cluster.
    Description: --offline Upgrade a stopped cluster

Later, I removed this parameter and upgraded directly. Although it succeeded, there was an issue where data could not be queried, and relevant data had to be resynchronized.

| username: tidb菜鸟一只 | Original post link

tiup --version
Check the version of tiup. Did you upgrade the version of tiup before upgrading?

| username: TiDBer_Lee | Original post link

tiup version 1.12.1 tiup
Go Version: go1.19.8
Git Ref: v1.12.1
GitHash: f4dae709fa3e904958f3433b617a6bc4bd9049a3

| username: Kongdom | Original post link

The prerequisite for using offline is to shut down the cluster, but according to your description, you only shut down the TiFlash component without shutting down the cluster.

| username: tidb菜鸟一只 | Original post link

Has the version of tiup cluster also been upgraded?

| username: Billmay表妹 | Original post link

Regarding the use of the --offline parameter, according to the TiUP official documentation, this parameter is used to upgrade an already stopped TiDB cluster, not to upgrade after shutting down the cluster. Therefore, when executing the tiup cluster upgrade command, if the cluster has already been shut down, the --offline parameter should be omitted.

Secondly, regarding the issue of data not being queried, this may be due to changes in the data structure after the TiDB upgrade, causing the original query statements to fail. To resolve this issue, you can try the following methods:

  1. Check if the upgraded version of TiDB supports your query statements. You can refer to the TiDB official documentation to understand the features and changes of the new version to better adapt to it.
  2. Check if the configuration file after the TiDB upgrade is correct. After the upgrade, the TiDB configuration file may change and need to be modified according to the new version’s requirements. You can refer to the TiDB official documentation to understand the new version’s configuration file format and parameter meanings.
  3. Check if the statistics after the TiDB upgrade are correct. After the upgrade, the TiDB statistics may become invalid and need to be recollected. You can use the ANALYZE TABLE command to collect statistics for the table to optimize the query plan.
  4. Check if the indexes after the TiDB upgrade are correct. After the upgrade, the TiDB indexes may become invalid and need to be recreated. You can use the CREATE INDEX command to create indexes for the table to optimize the query plan.

If the above methods do not resolve the issue, you can try adding --log-level debug to the TiDB startup parameters and check the TiDB logs for more detailed error information.

| username: TiDBer_Lee | Original post link

No, you can test it out.

| username: TiDBer_Lee | Original post link

Upgrade Documentation:

  • To upgrade TiFlash from a version earlier than 5.3 to version 5.3 or later, a downtime upgrade of TiFlash is required. Follow the steps below to upgrade TiFlash while ensuring other components run normally:
    1. Stop the TiFlash instance: tiup cluster stop <cluster-name> -R tiflash
    2. Upgrade the cluster without restarting (only updating files) using the --offline parameter: tiup cluster upgrade <cluster-name> <version> --offline
    3. Reload the entire cluster: tiup cluster reload <cluster-name>. At this point, TiFlash will also start normally without any additional operations.
| username: Jellybean | Original post link

May I ask if there are any anomalies in the logs of tidb-server and TiFlash?
Logs usually have some indications.

| username: Kongdom | Original post link

Could you please provide the official documentation link for this text?