Error When Upgrading TiDB from 6.5.2 to 7.1.0

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB 6.5.2 升级到7.1.0报错

| username: Tao

【TiDB Usage Environment】Production Environment
【TiDB Version】7.1.0
【Reproduction Path】Operations performed that led to the issue
tiup cluster upgrade tidb-cluster v7.1.0
【Encountered Issue: Phenomenon and Impact】

Upgrading component tidb
        Restarting instance 10.10.5.28:3306

Error: failed to restart: 10.10.5.28 tidb-3306.service, please check the instance's log(/data1/deploy/log) for more detail.: timed out waiting for port 3306 to be started after 2m0s

Verbose debug logs have been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2023-06-05-22-59-43.log.

【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】
tidb.log

[2023/06/05 22:57:43.848 +08:00] [INFO] [owner_daemon.go:81] ["begin running daemon"] [id=f6cc546c-de99-4c0d-b104-5bab6fc17e22] [daemon-id=LogBackup::Advancer]
[2023/06/05 22:57:43.855 +08:00] [INFO] [session.go:3852] ["CRUCIAL OPERATION"] [conn=0] [schemaVersion=130240] [cur_db=] [sql="ALTER TABLE mysql.stats_meta_history ADD COLUMN IF NOT EXISTS `source` varchar(40) NOT NULL after `version`;"] [user=]
[2023/06/05 22:57:43.856 +08:00] [INFO] [session.go:3852] ["CRUCIAL OPERATION"] [conn=0] [schemaVersion=130240] [cur_db=] [sql="CREATE TABLE IF NOT EXISTS mysql.tidb_ttl_task (\n\t\tjob_id varchar(64) NOT NULL,\n\t\ttable_id bigint(64) NOT NULL,\n\t\tscan_id int NOT NULL,\n\t\tscan_range_start BLOB,\n\t\tscan_range_end BLOB,\n\t\texpire_time timestamp NOT NULL,\n\t\towner_id varchar(64) DEFAULT NULL,\n\t\towner_addr varchar(64) DEFAULT NULL,\n\t\towner_hb_time timestamp DEFAULT NULL,\n\t\tstatus varchar(64) DEFAULT 'waiting',\n\t\tstatus_update_time timestamp NULL DEFAULT NULL,\n\t\tstate text,\n\t\tcreated_time timestamp NOT NULL,\n\t\tprimary key(job_id, scan_id),\n\t\tkey(created_time));"] [user=]
[2023/06/05 22:57:43.856 +08:00] [INFO] [session.go:3852] ["CRUCIAL OPERATION"] [conn=0] [schemaVersion=130240] [cur_db=] [sql="CREATE TABLE IF NOT EXISTS mysql.tidb_ttl_job_history (\n\t\tjob_id varchar(64) PRIMARY KEY,\n\t\ttable_id bigint(64) NOT NULL,\n        parent_table_id bigint(64) NOT NULL,\n    \ttable_schema varchar(64) NOT NULL,\n\t\ttable_name varchar(64) NOT NULL,\n    \tpartition_name varchar(64) DEFAULT NULL,\n\t\tcreate_time timestamp NOT NULL,\n\t\tfinish_time timestamp NOT NULL,\n\t\tttl_expire timestamp NOT NULL,\n        summary_text text,\n\t\texpired_rows bigint(64) DEFAULT NULL,\n    \tdeleted_rows bigint(64) DEFAULT NULL,\n    \terror_delete_rows bigint(64) DEFAULT NULL,\n    \tstatus varchar(64) NOT NULL,\n    \tkey(table_schema, table_name, create_time),\n    \tkey(parent_table_id, create_time),\n    \tkey(create_time)\n\t);"] [user=]
[2023/06/05 22:57:43.904 +08:00] [INFO] [ddl_worker.go:238] ["[ddl] add DDL jobs"] ["batch count"=1] [jobs="ID:123481, Type:create view, State:queueing, SchemaState:none, SchemaID:3, TableID:123480, RowCount:0, ArgLen:3, start time: 2023-06-05 22:57:43.999 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0; "] [table=true]
[2023/06/05 22:57:43.906 +08:00] [INFO] [ddl.go:1056] ["[ddl] start DDL job"] [job="ID:123481, Type:create view, State:queueing, SchemaState:none, SchemaID:3, TableID:123480, RowCount:0, ArgLen:3, start time: 2023-06-05 22:57:43.999 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0"] [query="CREATE OR REPLACE VIEW mysql.tidb_mdl_view as (\n\t\tSELECT job_id,\n\t\t\tdb_name,\n\t\t\ttable_name,\n\t\t\tquery,\n\t\t\tsession_id,\n\t\t\ttxnstart,\n\t\t\ttidb_decode_sql_digests(all_sql_digests, 4096) AS SQL_DIGESTS\n\t\tFROM information_schema.ddl_jobs,\n\t\t\tinformation_schema.cluster_tidb_trx,\n\t\t\tinformation_schema.cluster_processlist\n\t\tWHERE (ddl_jobs.state != 'synced' and ddl_jobs.state != 'cancelled')\n\t\t\tAND Find_in_set(ddl_jobs.table_id, cluster_tidb_trx.related_table_ids)\n\t\t\tAND cluster_tidb_trx.session_id = cluster_processlist.id\n\t);"]
[2023/06/05 22:57:45.843 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/06/05 22:58:06.347 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 22:58:28.847 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 22:58:51.347 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 22:59:13.847 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 22:59:36.348 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 22:59:58.847 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:00:21.347 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:00:43.847 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:01:06.347 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]

| username: zhanggame1 | Original post link

Business is busy, consider finding a time to shut down the database for an upgrade.

| username: Tao | Original post link

I tried upgrading this afternoon, but it didn’t succeed, and the service hasn’t come up since. As a result, there haven’t been any business connections.

| username: RenlySir | Original post link

Was the cluster performing any DDL operations during the upgrade?

| username: Tao | Original post link

No, the database is no longer accessible.

| username: RenlySir | Original post link

Okay, got it.

| username: Tao | Original post link

I just tried an offline upgrade. First, I executed tiup cluster stop tidb-cluster, then executed tiup cluster upgrade tidb-cluster v7.1.0 --offline, and it prompted Upgraded cluster tidb-cluster successfully. However, tiup cluster start tidb-cluster still reported an error:

Starting component tidb
        Starting instance 10.10.5.29:4000
        Starting instance 10.10.5.28:3306

Error: failed to start tidb: failed to start: 10.10.5.29 tidb-4000.service, please check the instance's log(/data1/deploy/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

Verbose debug logs have been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2023-06-05-23-18-43.log.

tidb.log:

[2023/06/05 23:16:57.398 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/06/05 23:17:17.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:17:40.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:18:02.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:18:25.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:18:47.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:19:10.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:19:32.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:19:55.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:20:17.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:20:40.401 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:21:02.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:21:25.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:21:47.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:21:54.582 +08:00] [INFO] [domain.go:2652] ["refreshServerIDTTL succeed"] [serverID=3629854] ["lease id"=76df888c21a2cb56]
[2023/06/05 23:22:10.402 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:22:32.901 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:22:55.403 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
[2023/06/05 23:23:17.902 +08:00] [WARN] [domain.go:215] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=130240]
| username: redgame | Original post link

This log doesn’t seem very useful. There might be an opportunity to completely restart the cluster.

| username: Tao | Original post link

The servers have all been rebooted, but they are still down, and the tidb-server won’t start.

| username: Tao | Original post link

The output of systemctl status tidb-3306.service shows that it is in an active (running) state, but the 3306 port is not started. The command netstat -ntlp | grep 3306 does not return any results, and telnet is not working.

| username: 我是咖啡哥 | Original post link

Can the stop command succeed? Try stopping it and see?
By the way, check the configuration of tidb-3306.service.

| username: Tao | Original post link

The systemctl stop tidb-3306.service command can successfully stop and start the service, but the port does not come up.

tidb-3306.service

[Unit]
Description=tidb service
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
LimitNOFILE=1000000
LimitSTACK=10485760
User=tidb
ExecStart=/bin/bash -c '/data1/deploy/scripts/run_tidb.sh'
Restart=always

RestartSec=15s

[Install]
WantedBy=multi-user.target

cat /data1/deploy/scripts/run_tidb.sh

#!/bin/bash
set -e

# WARNING: This file was auto-generated. Do not edit!
#          All your edit might be overwritten!
DEPLOY_DIR=/data1/deploy

cd "${DEPLOY_DIR}" || exit 1
exec env GODEBUG=madvdontneed=1 bin/tidb-server \
    -P 3306 \
    --status="10080" \
    --host="0.0.0.0" \
    --advertise-address="10.10.5.28" \
    --store="tikv" \
    --initialize-insecure \
    --path="10.10.5.29:2379,10.10.5.28:2379,10.10.5.33:2379" \
    --log-slow-query="/data1/deploy/log/tidb_slow_query.log" \
    --config=conf/tidb.toml \
    --log-file="/data1/deploy/log/tidb.log" 2>> "/data1/deploy/log/tidb_stderr.log"
| username: 我是咖啡哥 | Original post link

Check this output.
Also, send the parameters and logs for review.
/data1/deploy/conf/tidb.toml
/data1/deploy/log/tidb_stderr.log

| username: Tao | Original post link

systemctl status tidb-3306.service

 tidb-3306.service - tidb service
   Loaded: loaded (/etc/systemd/system/tidb-3306.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-06-06 09:09:26 CST; 18min ago
 Main PID: 32564 (tidb-server)
   CGroup: /system.slice/tidb-3306.service
           └─32564 bin/tidb-server -P 3306 --status=10080 --host=0.0.0.0 --advertise-address=10.10.5.28 --store=tikv --initialize-insecure --path=10.10.5.29:2379,10.10.5.28:2379,10.10.5.33:2379 --log-slow-query=/data1/deploy/log/tidb_slow_query.log --config=conf/tidb.toml --log-file=/data1/deploy/log/tidb.log

cat /data1/deploy/conf/tidb.toml

# WARNING: This file is auto-generated. Do not edit! All your modification will be overwritten!
# You can use 'tiup cluster edit-config' and 'tiup cluster reload' to update the configuration
# All configuration items you want to change can be added to:
# server_configs:
#   tidb:
#     aa.b1.c3: value
#     aa.b2.c4: value
compatible-kill-query = false
enable-streaming = false
host = "0.0.0.0"
lease = "45s"
lower-case-table-names = 2
max-index-length = 3072
oom-action = "log"
run-ddl = true
socket = ""
split-table = true
store = "tikv"
token-limit = 1000

[binlog]
ignore-error = false
write-timeout = "15s"

[log]
disable-timestamp = false
expensive-threshold = 10000
format = "text"
level = "info"
query-log-max-len = 2048
slow-threshold = 300
[log.file]
log-rotate = true
max-backups = 0
max-days = 0
max-size = 300

[opentracing]
enable = false
rpc-metrics = false
[opentracing.reporter]
buffer-flush-interval = 0
local-agent-host-port = ""
log-spans = false
queue-size = 0
[opentracing.sampler]
max-operations = 0
param = 1.0
sampling-refresh-interval = 0
sampling-server-url = ""
type = "const"

[performance]
cross-join = true
feedback-probability = 0.0
force-priority = "NO_PRIORITY"
max-procs = 0
pseudo-estimate-ratio = 0.8
query-feedback-limit = 1024
run-auto-analyze = true
stats-lease = "3s"
stmt-count-limit = 5000
tcp-keep-alive = true

[pessimistic-txn]
enable = true
max-retry-count = 256
ttl = "30s"

[prepared-plan-cache]
capacity = 100
enabled = false
memory-guard-ratio = 0.1

[proxy-protocol]
header-timeout = 5
networks = ""

[security]
cluster-ssl-ca = ""
cluster-ssl-cert = ""
cluster-ssl-key = ""
ssl-ca = ""
ssl-cert = ""
ssl-key = ""

[status]
report-status = true

[tikv-client]
commit-timeout = "41s"
grpc-connection-count = 16
grpc-keepalive-time = 10
grpc-keepalive-timeout = 3

[txn-local-latches]
capacity = 2048000
enabled = false

tidb_stderr.log (3.7 MB)

| username: 我是咖啡哥 | Original post link

Can you see this process Main PID: 32564 with ps?

Also, can you check if there is a bak folder (with the version number before the upgrade) in the same directory as bin?
Something like this:

Try backing up the bin, and move the old one back to see if it can start normally?

| username: Tao | Original post link

ps -ef | grep 32564

tidb      32564      1  0 09:09 ?        00:00:14 bin/tidb-server -P 3306 --status=10080 --host=0.0.0.0 --advertise-address=10.10.5.28 --store=tikv --initialize-insecure --path=10.10.5.29:2379,10.10.5.28:2379,10.10.5.33:2379 --log-slow-query=/data1/deploy/log/tidb_slow_query.log --config=conf/tidb.toml --log-file=/data1/deploy/log/tidb.log

Restored bin, and encountered an error when starting again

[2023/06/06 09:38:57.404 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/06/06 09:38:57.404 +08:00] [INFO] [store.go:75] ["new store"] [path=tikv://10.10.5.29:2379,10.10.5.28:2379,10.10.5.33:2379]
[2023/06/06 09:38:57.404 +08:00] [INFO] [client.go:405] ["[pd] create pd client with endpoints"] [pd-address="[10.10.5.29:2379,10.10.5.28:2379,10.10.5.33:2379]"]
[2023/06/06 09:38:57.411 +08:00] [INFO] [base_client.go:360] ["[pd] update member urls"] [old-urls="[http://10.10.5.29:2379,http://10.10.5.28:2379,http://10.10.5.33:2379]"] [new-urls="[http://10.10.5.28:2379,http://10.10.5.29:2379,http://10.10.5.33:2379]"]
[2023/06/06 09:38:57.411 +08:00] [INFO] [base_client.go:378] ["[pd] switch leader"] [new-leader=http://10.10.5.29:2379] [old-leader=]
[2023/06/06 09:38:57.411 +08:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=6708518892049401148]
[2023/06/06 09:38:57.412 +08:00] [INFO] [client.go:698] ["[pd] tso dispatcher created"] [dc-location=global]
[2023/06/06 09:38:57.413 +08:00] [INFO] [store.go:81] ["new store with retry success"]
[2023/06/06 09:38:57.427 +08:00] [INFO] [ddl_api.go:1039] ["Automatically convert BLOB(65535) to MEDIUMBLOB"]
[2023/06/06 09:38:57.429 +08:00] [FATAL] [terror.go:300] ["unexpected error"] [error="[meta:1050]table already exists"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:319\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:218\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:319\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:218\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
| username: 我是咖啡哥 | Original post link

Looking at the log, this should be the reason. You used some deprecated parameters. Please modify it first and remove the unsupported parameters.

| username: 我是咖啡哥 | Original post link

Use tiup to make changes, don’t modify the configuration file directly.

| username: 我是咖啡哥 | Original post link

When upgrading, there is a step to modify the configuration file. I see that your configuration file has added many parameters, probably without carefully checking which ones are incompatible. :joy:

| username: Tao | Original post link

This is my configuration, and it doesn’t contain these configuration items. I don’t know where these configurations came from.

/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/meta.yaml

user: tidb
tidb_version: v7.1.0
last_ops_ver: |-
  v1.0.8 tiup
  Go Version: go1.13
  Git Branch: master
  GitHash: 4276089
topology:
  global:
    user: tidb
    ssh_port: 22
    ssh_type: builtin
    deploy_dir: deploy
    data_dir: data
    os: linux
    arch: amd64
  monitored:
    node_exporter_port: 9100
    blackbox_exporter_port: 9115
    deploy_dir: deploy/monitor-9100
    data_dir: data/monitor-9100
    log_dir: deploy/monitor-9100/log
  server_configs:
    tidb:
      performance.feedback-probability: 0.0
    tikv: {}
    pd: {}
    tidb_dashboard: {}
    tiflash: {}
    tiflash-learner: {}
    pump: {}
    drainer: {}
    cdc: {}
    kvcdc: {}
    grafana: {}
  tidb_servers:
  - host: 10.10.5.28
    ssh_port: 22
    imported: true
    port: 3306
    status_port: 10080
    deploy_dir: /data1/deploy
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  - host: 10.10.5.29
    ssh_port: 22
    imported: true
    port: 4000
    status_port: 10080
    deploy_dir: /data1/deploy
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  tikv_servers:
  - host: 10.10.5.30
    ssh_port: 22
    imported: true
    port: 20160
    status_port: 20180
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  - host: 10.10.5.31
    ssh_port: 22
    imported: true
    port: 20160
    status_port: 20180
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  - host: 10.10.5.32
    ssh_port: 22
    imported: true
    port: 20160
    status_port: 20180
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  tiflash_servers:
  - host: 10.10.5.30
    ssh_port: 22
    tcp_port: 9000
    http_port: 8123
    flash_service_port: 3930
    flash_proxy_port: 20170
    flash_proxy_status_port: 20292
    metrics_port: 8234
    deploy_dir: /data1/tiflash-9000
    data_dir: data
    log_dir: log
    arch: amd64
    os: linux
  - host: 10.10.5.31
    ssh_port: 22
    tcp_port: 9000
    http_port: 8123
    flash_service_port: 3930
    flash_proxy_port: 20170
    flash_proxy_status_port: 20292
    metrics_port: 8234
    deploy_dir: /data1/tiflash-9000
    data_dir: data
    log_dir: log
    arch: amd64
    os: linux
  - host: 10.10.5.32
    ssh_port: 22
    tcp_port: 9000
    http_port: 8123
    flash_service_port: 3930
    flash_proxy_port: 20170
    flash_proxy_status_port: 20292
    metrics_port: 8234
    deploy_dir: /data1/tiflash-9000
    data_dir: data
    log_dir: log
    arch: amd64
    os: linux
  pd_servers:
  - host: 10.10.5.29
    ssh_port: 22
    imported: true
    name: pd_tipd-1
    client_port: 2379
    peer_port: 2380
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data.pd
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  - host: 10.10.5.28
    ssh_port: 22
    imported: true
    name: pd_tidb-1
    client_port: 2379
    peer_port: 2380
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data.pd
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  - host: 10.10.5.33
    ssh_port: 22
    imported: true
    name: pd_tipd-2
    client_port: 2379
    peer_port: 2380
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data.pd
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux
  monitoring_servers:
  - host: 10.10.5.28
    ssh_port: 22
    imported: true
    port: 9090
    ng_port: 12020
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/prometheus2.0.0.data.metrics
    log_dir: /data1/deploy/log
    external_alertmanagers: []
    storage_retention: 30d
    arch: amd64
    os: linux
  grafana_servers:
  - host: 10.10.5.28
    ssh_port: 22
    imported: true
    port: 3000
    deploy_dir: /data1/deploy
    arch: amd64
    os: linux
    username: admin
    password: admin
    anonymous_enable: false
    root_url: ""
    domain: ""
  alertmanager_servers:
  - host: 10.10.5.28
    ssh_port: 22
    imported: true
    web_port: 9093
    cluster_port: 9094
    deploy_dir: /data1/deploy
    data_dir: /data1/deploy/data.alertmanager
    log_dir: /data1/deploy/log
    arch: amd64
    os: linux