Simulating Deployment of Production Environment Cluster on a Single Machine Error: failed to scp

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 在单机上模拟部署生产环境集群报错: failed to scp

| username: TiDBer_oHpB1az2

OS: Fedora Linux 38 (Workstation Edition) x86_64
Kernel: 6.3.7-200.fc38.x86_64
TIDB version: v7.1.0
Referenced document: TiDB 数据库快速上手指南 | PingCAP 文档中心

Deployment configuration:

# # Global variables are applied to all deployments and used as the default value of
# # the deployments if a specific deployment value is missing.
global:
 user: "tidb"
 ssh_port: 22
 deploy_dir: "/data/tidb/deploy"
 data_dir: "/data/tidb/data"

# # Monitored variables are applied to all the machines.
monitored:
 node_exporter_port: 9100
 blackbox_exporter_port: 9115

server_configs:
 tidb:
   log.slow-threshold: 300
 tikv:
   readpool.storage.use-unified-pool: false
   readpool.coprocessor.use-unified-pool: true
 pd:
   replication.enable-placement-rules: true
   replication.location-labels: ["host"]
 tiflash:
   logger.level: "info"

pd_servers:
 - host: 172.21.117.108

tidb_servers:
 - host: 172.21.117.108

tikv_servers:
 - host: 172.21.117.108
   port: 20160
   status_port: 20180
   config:
     server.labels: { host: "logic-host-1" }

 - host: 172.21.117.108
   port: 20161
   status_port: 20181
   config:
     server.labels: { host: "logic-host-2" }

 - host: 172.21.117.108
   port: 20162
   status_port: 20182
   config:
     server.labels: { host: "logic-host-3" }

tiflash_servers:
 - host: 172.21.117.108

monitoring_servers:
 - host: 172.21.117.108

grafana_servers:
 - host: 172.21.117.108

Command and output:

$ tiup cluster deploy tilocal v7.1.0 ./topo.yaml --user root -p
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/user/.tiup/components/cluster/v1.12.3/tiup-cluster deploy tilocal v7.1.0 ./topo.yaml --user root -p
Input SSH password: 

+ Detect CPU Arch Name
  - Detecting node 172.21.117.108 Arch info ... Done

+ Detect CPU OS Name
  - Detecting node 172.21.117.108 OS info ... Done
Please confirm your topology:
Cluster type:    tidb
Cluster name:    tilocal
Cluster version: v7.1.0
Role        Host            Ports                            OS/Arch       Directories
----        ----            -----                            -------       -----------
pd          172.21.117.108  2379/2380                        linux/x86_64  /data/tidb/deploy/pd-2379,/data/tidb/data/pd-2379
tikv        172.21.117.108  20160/20180                      linux/x86_64  /data/tidb/deploy/tikv-20160,/data/tidb/data/tikv-20160
tikv        172.21.117.108  20161/20181                      linux/x86_64  /data/tidb/deploy/tikv-20161,/data/tidb/data/tikv-20161
tikv        172.21.117.108  20162/20182                      linux/x86_64  /data/tidb/deploy/tikv-20162,/data/tidb/data/tikv-20162
tidb        172.21.117.108  4000/10080                       linux/x86_64  /data/tidb/deploy/tidb-4000
tiflash     172.21.117.108  9000/8123/3930/20170/20292/8234  linux/x86_64  /data/tidb/deploy/tiflash-9000,/data/tidb/data/tiflash-9000
prometheus  172.21.117.108  9090/12020                       linux/x86_64  /data/tidb/deploy/prometheus-9090,/data/tidb/data/prometheus-9090
grafana     172.21.117.108  3000                             linux/x86_64  /data/tidb/deploy/grafana-3000
Attention:
    1. If the topology is not what you expected, check your yaml file.
    2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ Generate SSH keys ... Done
+ Download TiDB components
  - Download pd:v7.1.0 (linux/amd64) ... Done
  - Download tikv:v7.1.0 (linux/amd64) ... Done
  - Download tidb:v7.1.0 (linux/amd64) ... Done
  - Download tiflash:v7.1.0 (linux/amd64) ... Done
  - Download prometheus:v7.1.0 (linux/amd64) ... Done
  - Download grafana:v7.1.0 (linux/amd64) ... Done
  - Download node_exporter: (linux/amd64) ... Done
  - Download blackbox_exporter: (linux/amd64) ... Done
+ Initialize target host environments
  - Prepare 172.21.117.108:22 ... Done
+ Deploy TiDB instance
  - Copy pd -> 172.21.117.108 ... Error
  - Copy tikv -> 172.21.117.108 ... Error
  - Copy tikv -> 172.21.117.108 ... Error
  - Copy tikv -> 172.21.117.108 ... Error
  - Copy tidb -> 172.21.117.108 ... Error
  - Copy tiflash -> 172.21.117.108 ... Error
  - Copy prometheus -> 172.21.117.108 ... Error
  - Copy grafana -> 172.21.117.108 ... Error
  - Deploy node_exporter -> 172.21.117.108 ... Error
  - Deploy blackbox_exporter -> 172.21.117.108 ... Error

Error: failed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to 172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz: failed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to tidb@172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz: Process exited with status 1

Verbose debug logs have been written to /home/user/.tiup/logs/tiup-cluster-debug-2023-06-16-16-15-25.log.

Log (last entry):

2023-06-16T16:15:25.300+0800	INFO	Execute command finished	{"code": 1, "error": "failed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to 172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz: failed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to tidb@172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz: Process exited with status 1", "errorVerbose": "Process exited with status 1\nfailed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to tidb@172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz\ngithub.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Transfer\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:207\ngithub.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Transfer\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:114\ngithub.com/pingcap/tiup/pkg/cluster/task.(*InstallPackage).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/install_package.go:45\ngithub.com/pingcap/tiup/pkg/cluster/task.(*CopyComponent).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/copy_component.go:64\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86\ngithub.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:111\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:144\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to 172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz"}

Tried executing the following command:

scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz  tidb@172.21.117.108:/data/tidb/deploy/

But it requires a password.

| username: Hi70KG | Original post link

Did you not configure SSH mutual trust between the nodes?

| username: TiDBer_oHpB1az2 | Original post link

@Hi70KG
I don’t understand what you mean by SSH mutual trust.
Logging in with the root password is not a problem.
If you mean SSH public key login for the tidb user, the tidb user is created by tiup, so why would there be a need for manual intervention to add public key login midway?

| username: 有猫万事足 | Original post link

This part is in the pre-check documentation for installation. The tidb user needs to be created by yourself according to the documentation and configured with sudo.

| username: zhanggame1 | Original post link

Install with root without configuring SSH mutual trust, sudo, adding tidb directory, etc.

| username: 我是咖啡哥 | Original post link

Can you directly execute this command to see if there is an error?

scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to 172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz
| username: zhanggame1 | Original post link

monitored:
node_exporter_port: 9100
blackbox_exporter_port: 9115
What is this used for? Try removing it and see.

| username: TiDBer_oHpB1az2 | Original post link

@zhanggame1
That is included in the official template.

| username: TiDBer_oHpB1az2 | Original post link

scp: remote mkdir “/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz”: Permission denied

After adding sudo:
$ sudo scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz to 172.21.117.108:/data/tidb/deploy/tidb-4000/bin/tidb-v7.1.0-linux-amd64.tar.gz
[sudo] password for wencan:
The authenticity of host ‘172.21.117.108 (172.21.117.108)’ can’t be established.
ED25519 key fingerprint is SHA256:snC38ABnYwNvnIpEvrHvfFc6HjezN76zBxCUPLXXRVA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘172.21.117.108’ (ED25519) to the list of known hosts.
root@172.21.117.108’s password:
tidb-v7.1.0-linux-amd64.tar.gz 100% 73MB 273.3MB/s 00:00
scp: stat local “to”: No such file or directory

| username: 我是咖啡哥 | Original post link

scp /home/user/.tiup/storage/cluster/packages/tidb-v7.1.0-linux-amd64.tar.gz  172.21.117.108:/data/tidb/deploy/tidb-4000/bin/
| username: TiDBer_oHpB1az2 | Original post link

The one posted above is the result of removing “ti”.

| username: 春风十里 | Original post link

  1. I see that the documentation recommends the following operating systems:
  1. For simulating a production environment on a single machine, it is generally installed directly as root, but your output doesn’t seem to be from a root installation?

  2. If it is a single-machine deployment, I believe theoretically SSH mutual trust is not needed. I suspect whether your machine’s IP address is 172.21.117.108?
    You can check your machine’s IP using ip a.
    For example:

Below is my log output, and I didn’t find the scp command

| username: 我是咖啡哥 | Original post link

That’s right, according to the official documentation, Fedora 35 and above should meet the requirements for the OS. I haven’t used Fedora, so I’m not familiar with it.

| username: zhanggame1 | Original post link

Today I installed a standalone cluster on CentOS 7, and it went very smoothly.

| username: redgame | Original post link

I don’t know if manual testing is possible.

| username: zhanggame1 | Original post link

I installed it several times and encountered similar issues. Later, I found out that the root password was entered incorrectly. Try again.

| username: TiDBer_oHpB1az2 | Original post link

@I am Coffee Brother @Spring Breeze Ten Miles
After testing, tiup can indeed achieve one-click deployment on CentOS.
But it doesn’t work on Fedora 38. Looking forward to TiDB improving the deployment and operation experience. Thank you.

| username: 我是咖啡哥 | Original post link

Did you successfully change the operating system?
I usually use CentOS, and it also works on Mac.

| username: TiDBer_oHpB1az2 | Original post link

Which version of CentOS do your friends commonly use?

| username: buptzhoutian | Original post link

When tiup cluster executes scp/ssh, it uses the key pair it generated. If you want to test connectivity, you need to add

-i $TIUP_HOME/storage/cluster/clusters/tilocal/ssh/id_rsa