TiUP Modify Cluster IP (Based on Version V6)

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiUP 修改集群 IP(基于版本V6)

| username: big_river

[TiDB Usage Environment] Test
[TiDB Version] V6.1.2
[Reference Document] 【SOP 系列 12】TiUP 修改集群 IP 、Port 和目录 - TiDB 的问答社区

  1. Background
    A TiDB cluster needs to be migrated to another data center due to a data center relocation. The IP addresses of the cluster servers need to be changed accordingly. This document follows the reference document for the operation. Some issues were encountered during the process, but they were resolved with the guidance of PingCAP’s experts. This record aims to help others avoid similar pitfalls in the future. Thank you.

  2. Reference Document, Backup, Always Backup

  3. For cluster environment operations, data must be backed up first. If data is lost during the operation, it may be impossible to recover the cluster. At the very least, back up:

    • The ~/.tiup/storage/cluster/clusters/ directory on the control machine
    • All data directories (data_dir) shown in ~/.tiup/storage/cluster/clusters//meta.yaml
  4. IP adjustments require stopping the cluster. Coordinate downtime windows for online systems and ensure thorough preparation and validation.

  5. Adjusting the PD’s IP and port requires rebuilding the entire PD cluster. If components like TiCDC’s changefeed depend on PD’s persistent data, data may need to be resynchronized. Weigh the pros and cons of the adjustment. The PD rebuild method may vary with the tiup cluster version. If the latest version is unsupported, consider using an older version.

  6. Use mysqldump to back up the databases used within the cluster:
    mysqldump -hxxxx -uxxxx -P4000 --databases xx1 xx2 xx3 > xxx.sql
    Ensure the backup directory has sufficient storage space before executing this step!

  7. Emphasize the importance of backup three times: Always backup before adjustments!

  8. Modify Cluster IP with tiup

  9. Stop the cluster:
    tiup cluster stop clustername
    tiup cluster display clustername

  10. Shut down the machines, relocate them, and change their IP addresses.

  11. Modify the meta.yaml configuration file:
    Note:

    • Backup the configuration file and data directory before making changes.
    • Change all component IP addresses from the source IP to the target IP.
      cp ~/.tiup/storage/cluster/clusters/tidb-test/meta.yaml ~/.tiup/storage/cluster/clusters/tidb-test/meta.yaml.bak
      vi ~/.tiup/storage/cluster/clusters/tidb-test/meta.yaml

    Note: Change the host address and name IP to the new IP:
    host: 172.16.61.111
    ssh port: 22
    name: pd-172.16.61.111-2379

  12. Rebuild the PD Cluster

  1. Obtain Cluster ID:
    Retrieve the Cluster ID from the logs of any PD node (recommended):
    [tidb@localhost ~]$ cat /tidb-deploy/pd-2379/log/*.log | grep “init cluster id”
    [2020/10/10 11:13:22.794 +08:00] [INFO] [server.go:343] [“init cluster id”] [cluster-id=6881824393201874622]
    [2020/10/10 11:15:00.388 +08:00] [INFO] [server.go:343] [“init cluster id”] [cluster-id=6881548221407099619]
    [2020/10/10 11:22:31.005 +08:00] [INFO] [server.go:343] [“init cluster id”] [cluster-id=6881548221407099619]

  2. Obtain Allocated ID:
    Retrieve the maximum allocated ID from the logs of any PD node:
    [tidb@localhost ~]$ cat /tidb-deploy/pd-2379/log/*.log | grep “idAllocator allocates a new id” | awk -F’=’ ‘{print $2}’ | awk -F’]’ ‘{print $1}’ | sort -r | head -n 1
    1000

  3. Remove All Old PD Data Directories:
    mv pd-2379 pd-2379_bak
    Note: Backup and remove the data directories of all PD nodes.

  4. Stop the Cluster:
    After changing the machine IPs, the system may automatically restart the cluster, though it may not function correctly. Some processes might still exist. This happened in our environment.
    tiup cluster stop clustername

  5. Deploy the New PD Cluster:
    tiup cluster reload clustername -R pd --force --skip-restart
    Note: Do not specify the version number; use --skip-restart.
    After successful execution, check if the IP addresses in the run_pd.sh file in the pd node deployment directory scripts are the new IPs.

  6. Start and Stop the PD Cluster:
    tiup cluster start clustername -R pd
    tiup cluster stop clustername -R pd

  7. Use pd-recover to Restore the PD Cluster:
    If the original installation lacks pd-recover, follow these steps (skip if tiup pd-recover is available):
    (1) Download the corresponding toolkit package from the TiDB website and upload it to the control machine.
    Extract it: tar -xzvf tidb-community-toolkit-v6.1.2-linux-amd64.tar.gz
    (2) Assign appropriate user permissions to the directory files. If all are root users, skip this step.
    (3) Use tiup mirror show to find the original mirror directory location:
    cd tidb-community-server-${version}-linux-amd64/
    cp -rp keys ~/.tiup/
    tiup mirror merge …/tidb-community-toolkit-${version}-linux-amd64
    Restore the PD Cluster:
    Use the new PD IP for pd-recover, ensuring pd-recover matches the cluster version:
    tiup pd-recover -endpoints http://172.16.61.111:2379 -cluster-id 6881548221407099619 -alloc-id 10000
    Note: Fill in the IP, cluster-id, and alloc-id based on actual conditions.

  8. Restart the Cluster:
    The original reference article used reload, which may cause errors. Directly restart the cluster:
    tiup cluster stop clustername
    tiup cluster start clustername

  9. Check Cluster Status and Verify Data:
    tiup cluster display clustername

  10. After verifying data accuracy, delete the original backup data and directories to free up storage space.

| username: TiDBer_vfJBUcxl | Original post link

Got it! Please provide the Chinese text you need translated.

| username: Kongdom | Original post link

:+1: :+1:

| username: srstack | Original post link

:+1:

| username: redgame | Original post link

:+1::+1::+1:

| username: 像风一样的男子 | Original post link

:+1: :+1:

| username: cassblanca | Original post link

good TiDB user