After Manually Scaling Down TiFlash, Unable to Save and Exit the Cluster Topology Configuration File

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 手动缩容TiFlash后，更新集群拓扑配置文件无法wq保存退出

| username: OnTheRoad

【TiDB Usage Environment】Production\Test Environment\POC
【TiDB Version】v5.4.2
【Encountered Problem】After manually scaling down TiFlash, modifying the cluster topology configuration file cannot be saved and exited.
【Reproduction Path】
Steps to execute:

  1. tiup ctl:v5.4.2 pd -u http://192.168.3.221:2379 store
  2. tiup ctl:v5.4.2 pd -u http://192.168.3.221:2379 store delete 5761
  3. Wait for the store corresponding to the TiFlash node to disappear or the state_name to change to Tombstone, then shut down the TiFlash process.
    Check the TiFlash status as Tombstone using the tiup cluster display command
  4. Delete the deployment directory and data directory of the TiFlash node
  5. After clearing the TiFlash configuration information using tiup cluster edit-config kruidb-cluster, it cannot be saved and exited with wq.

【Problem Phenomenon and Impact】
According to the steps in the Manual Scale Down TiFlash link, manually scale down TiFlash. After waiting for TiFlash to change to Tombstone status, shutting down the TiFlash process, and deleting the data files, updating the cluster configuration cannot be saved with wq.

15
151380×526 124 KB

After entering Y, it re-enters the configuration file editing interface. Entering N results in Nothing Changed.
34
341240×1000 70.2 KB

2
| username: wakaka | Original post link

After confirming that the TiFlash node has no data, you can try using scale-in --force to clean it up. After executing, you can perform edit-config.

3
| username: songxuecheng | Original post link

Try this: tiup cluster prune <cluster-name>

4
| username: OnTheRoad | Original post link

  1. Removed TiFlash information using tiup cluster scale-in kruidb-cluster --node 192.168.3.229:9000 --force.

  2. The official documentation “Manually Scale In TiFlash” is an alternative solution to handle forced node removal or tiup operation failures. The final step involves manually updating the TiUP cluster configuration file using tiup cluster edit-config <cluster-name>, where you manually delete the offline TiFlash node information in edit mode. However, after editing the cluster configuration file, it cannot be saved and exited. Ultimately, if you still need to use tiup cluster scale-in to clean up TiFlash node information, doesn’t this defeat the purpose of manual scaling?

5
| username: forever | Original post link

Have you tried the above?

image
image808×61 4.51 KB

6
| username: OnTheRoad | Original post link

The command tiup cluster prune does not work, indicating that the TiFlash node information has already been removed from PD.

7
| username: h5n1 | Original post link

An immutable field means that some content is not allowed to be modified by edit-config. You can try modifying the meta.yaml file in the .tiup/storage/cluster/clusters/XXXX directory.

8
| username: BraveChen | Original post link

I always feel that some features of tiup are not well implemented.

9
| username: h5n1 | Original post link

Continuous improvement

10
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.