BR Backup Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份报错

| username: 点点-求助来了

[TiDB Usage Environment] Test
[TiDB Version] 5.4.0
[Reproduction Path] What operations were performed that caused the issue
[Encountered Issue: Problem Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
oup).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [unit-name=“range start:7480000000000000075f69800000000000000100 end:7480000000000000075f698000000000000001fb”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:7480000000000000135f720000000000000000 end:7480000000000000135f72ffffffffffffffff00”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:7480000000000000175f720000000000000000 end:7480000000000000175f72ffffffffffffffff00”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:74800000000000002d5f69800000000000000100 end:74800000000000002d5f698000000000000001fb”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:7480000000000000355f69800000000000000100 end:7480000000000000355f698000000000000001fb”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:74800000000000003d5f69800000000000000100 end:74800000000000003d5f698000000000000001fb”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”] [unit-name=“range start:7480000000000000515f720000000000000000 end:7480000000000000515f72ffffffffffffffff00”] [error=“rpc error: code = Canceled desc = context canceled”] [errorVerbose=“rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*client).GetAllStores\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211118054146-02848d2660ee/client/client.go:1523\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:142\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:179\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:60\ngithub.com/pingcap/tidb/br/pkg/conn.GetAllTiKVStoresWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/conn/conn.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:511\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/backup/client.go:471\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”]
Error: error happen in store 4 at 192.168.30.139:20160: Io(Os { code: 2, kind: NotFound, message: “No such file or directory” }): [BR:KV:ErrKVStorage]tikv storage occur I/O error

| username: tidb菜鸟一只 | Original post link

Path not found?

| username: hey-hoho | Original post link

Confirm that there is a backup directory on this TiKV node and that the permissions are set to 777.

| username: 点点-求助来了 | Original post link

The key issue is that I am backing up on 141, but why is it backing up to 139?

| username: hey-hoho | Original post link

BR backup sends requests to each TiKV, and the data files are exported by TiKV.

| username: caiyfc | Original post link

The image you provided is not accessible. Please provide the text content you need translated.

| username: 点点-求助来了 | Original post link

How to restore? The command tiup br restore full --pd "192.168.30.141:2379" --storage "local:/home/tidb/databak" only restored the table results because the .sst data files are on other nodes!

| username: 点点-求助来了 | Original post link

How to restore? The command tiup br restore full --pd "192.168.30.141:2379" --storage "local:/home/tidb/databak" only restored the table results because the .sst data files are on other nodes!

| username: 裤衩儿飞上天 | Original post link

BR Backup and Restore Use Case Examples | PingCAP Documentation Center

image

| username: caiyfc | Original post link

Normally, when using BR, you need to set up storage on all machines where TiKV is located, either using NFS or directly attaching it. Then all the backup data from TiKV will be aggregated into the same directory. Therefore, when restoring, the complete aggregated data is used, ensuring that the restored data is complete.

| username: 点点-求助来了 | Original post link

This error seems a bit off, unable to restore data.

Starting component br: /root/.tiup/components/br/v5.4.2/br restore full --pd 192.168.30.141:2379 --storage local:/home/tidb/databak
Detail BR log in /tmp/br.log.2023-04-22T18.18.45+0800
Full restore <--------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2023/04/22 18:19:28.966 +08:00] [INFO] [collector.go:67] [“Full restore failed summary”] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=1.211096897s] [restore-ranges=3] [unit-name=file] [error=“Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/55da4800-d305-440b-84db-161cdc8cd93b_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/d8cf0399-0642-4198-9656-bd32ac07d786_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b641d953-c754-47ff-8093-cf5179d40425_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed”] [errorVerbose=“the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/55da4800-d305-440b-84db-161cdc8cd93b_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/d8cf0399-0642-4198-9656-bd32ac07d786_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b641d953-c754-47ff-8093-cf5179d40425_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371”]
Error: Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home

| username: 点点-求助来了 | Original post link

Still can’t recover, here is the error message:

Starting component br: /root/.tiup/components/br/v5.4.2/br restore full --pd 192.168.30.141:2379 --storage local:/home/tidb/databak
Detail BR log in /tmp/br.log.2023-04-22T18.18.45+0800
Full restore <--------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2023/04/22 18:19:28.966 +08:00] [INFO] [collector.go:67] [“Full restore failed summary”] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=1.211096897s] [restore-ranges=3] [unit-name=file] [error=“Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/55da4800-d305-440b-84db-161cdc8cd93b_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/d8cf0399-0642-4198-9656-bd32ac07d786_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b641d953-c754-47ff-8093-cf5179d40425_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed”] [errorVerbose=“the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/55da4800-d305-440b-84db-161cdc8cd93b_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/d8cf0399-0642-4198-9656-bd32ac07d786_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b641d953-c754-47ff-8093-cf5179d40425_104033_5_53_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:507\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1371”]
Error: Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/63be0a61-de35-4654-a2b1-29130cd3478a_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/132cb832-902b-4327-85df-ece5bcf321a8_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/b060d509-5bef-44a3-8b90-5f7329135957_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/3921eb53-fe35-471d-8541-1ff9f8d53e9d_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tidb/databak/4_104021_51_038ac08f6d40baf173f28523ba6f6bc588059d52be870c61313059de6f360297_1682158566517_write.sst into /tidb-data/tikv-20160/import/.temp/f468bdde-cef5-48b2-885c-7b37169644d5_104033_5_53_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///home/tid

| username: 裤衩儿飞上天 | Original post link

Each TiKV node must have a complete backup of the data.

| username: dba-kit | Original post link

BR backups are generally backed up to S3 or NFS. I suggest setting up an S3 using MinIO, and then performing both backup and restore operations by reading and writing from S3, rather than using local files. If you use a local directory, the backup will be directly saved to the local directories of each TiKV. During restoration, you will need to manually transfer the backup files from the TiKV of the old cluster to the temporary directories of the TiKV in the new cluster, which is very inconvenient to operate.

| username: dba-kit | Original post link

There is a dedicated explanation here: