V6.5.0 Primary Cluster: Error Occurs When Using BR to Backup Two Large Tables and Restoring to v6.1.5 Version Cluster Replica

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: v6.5.0主库集群，使用br备份2张大表，再到v6.1.5版本集群从库，进行br恢复时，报错：

| username: vcdog

[TiDB Usage Environment] Production Environment
[TiDB Version] Main Cluster: v6.5.0, Replica Cluster: v6.1.5
[Reproduction Steps]

  1. Operate on the main cluster, use tiup br backup full --filter to back up 2 large tables.
  2. Then, copy the backup files from each KV node to a shared disk.
  3. Next, operate on the replica cluster, copy the backup files from the shared disk to each KV node of the replica cluster.
  4. Copy the backupmeta from the shared disk to the control machine of the replica cluster.
  5. Execute tiup br restore full --filter to perform the restore.

[Encountered Issue: Symptoms and Impact] The error is as follows:

Full Restore <----------------------------------------------------------------------------------------------> 100.00%
[2023/04/14 12:21:46.093 +08:00] [INFO] [collector.go:69] ["Full Restore failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=10.743447468s] [restore-ranges=2090] [unit-name=file] [error="No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

2
| username: vcdog | Original post link

Full Restore <----------------------------------------------------------------------------------------------> 100.00%
[2023/04/14 12:21:46.093 +08:00] [INFO] [collector.go:69] [“Full Restore failed summary”] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=10.743447468s] [restore-ranges=2090] [unit-name=file] [error=“No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed”] [errorVerbose=“the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n The Go Programming Language\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594”]
Error: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed

3
| username: TiDBer_pkQ5q1l0 | Original post link

There seems to be no problem with the recovery steps, but the error indicates that the file cannot be found. I’m not sure if the path for backup and restore is correct.

4
| username: vcdog | Original post link

There are no issues with the backup path and the restore path. This problem is rather peculiar. I first used Dumpling for backup and tidb-lightning for restore. BR is unstable.

5
| username: 裤衩儿飞上天 | Original post link

Does the directory where the backup is located have access permissions during restoration?

6
| username: ljluestc | Original post link

The error message you provided indicates that the BR (Backup and Restore) restore process encountered a failure while downloading SST (SSTable) files. The specific error is “No such file or directory (os error 2).”

To resolve this issue and successfully complete the restore, you can follow these steps:

  1. Verify Backup File Availability: Ensure that the backup files copied to each KV node of the replica are accessible and located in the expected directories. Check for any inconsistencies or missing files.

  2. Check File Permissions and Ownership: Verify that the user running the BR restore process has sufficient permissions to access and read the backup files. Ensure that the ownership and permissions of the backup files are correctly set.

  3. Confirm the backupmeta File: Carefully check the backupmeta file placed on the control machine of the replica. Ensure that the file is valid and contains the metadata required for the restore process.

  4. Verify Network Connectivity: Ensure stable and reliable network connectivity between the control machine and the KV nodes of the replica. Confirm that there are no network-related issues causing the download failures.

  5. Check Storage Capacity: Confirm that the KV nodes of the replica have sufficient storage capacity to accommodate the restored data. Insufficient disk space can lead to download failures and errors.

  6. Review BR Restore Configuration: Review the configuration parameters provided during the BR restore process. Pay attention to the storage paths and file locations specified in the configuration. Ensure they match the actual setup and paths of the backup files.

  7. Consult BR Documentation and Community: Refer to the official TiDB Backup and Restore documentation for more detailed instructions and troubleshooting steps. You can also seek help from the TiDB community or forums, where experienced users and developers can provide guidance and help resolve specific issues.

By carefully checking and addressing the above points, you should be able to identify and resolve the issues causing the download failures during the BR restore process.

7
| username: 考试没答案 | Original post link

What is the file system for backups? Currently, you might only be using NFS, right?

8
| username: 考试没答案 | Original post link

Have you set up NFS? Is it mounted on each node?