Failed to Restore After Using BR to Backup the Entire Database Snapshot Locally

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用BR备份整库快照到本地后，进行恢复操作失败

| username: TiDBer_MuyinmrC

[TiDB Usage Environment] Test
[TiDB Version] v6.3.0
[Reproduction Path] Operations that led to the issue:

  1. Used the official LUA script to initialize 4 tables in the sbtest database and inserted data, approximately ten million rows in total.
  2. Used br to backup sbtest with the following command:
tiup br backup db --pd "192.168.2.124:2379" \
	--db sbtest \
	--backupts '2022-11-21 02:00:00' \ 
	--storage "local:///usr/local/tidb/backup/tidb-test" \
	--ratelimit 128 \
	--log-file /usr/local/tidb/backup/backup.log
  1. Used the drop command to delete the sbtest1 table.
  2. Used the following command to restore sbtest1:
tiup br restore table --pd "192.168.2.124:2379" --db sbtest --table sbtest1 --storage "local:///usr/local/tidb/backup/tidb-test" --log-file restoredb.log

[Encountered Issue: Problem Description and Impact]
Restore failed, reporting that a certain file could not be found. The error log is as follows:

[2022/11/21 01:46:13.948 +08:00] [INFO] [collector.go:69] ["Table Restore failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=684.545µs] [restore-ranges=21] [unit-name=file] [error="Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e
2
| username: TiDBer_MuyinmrC | Original post link

I checked the backup directory and found only the following screenshot file. There are indeed no log error prompt files, which is quite strange as to why it would look for this file.

image
image1380×252 35.6 KB

3
| username: erwadba | Original post link

Backup & Restore FAQ | PingCAP Docs

image
image1085×281 27.6 KB

4
| username: 胡杨树旁 | Original post link

The file you are looking for is an SST file, which should store data. Is it possible that the data was not backed up during the backup process? There are two TiKV nodes; have backup directories been created on both servers?

5
| username: h5n1 | Original post link

When backing up, you can back up to the same directory on each TiKV. During restoration, the backup data from all other TiKVs must be present in that directory. The backup directory on any TiKV must contain all the backup data.

6
| username: TiDBer_MuyinmrC | Original post link

Yes, I have created the backup directories. Previously, when I was backing up, I found that the backup directories on the other two nodes did not have any files, so I didn’t pay attention to it. I just checked and found that they are there now. It might have been an issue with the previous backup. The problem has been resolved. Thank you.

7
| username: TiDBer_MuyinmrC | Original post link

Thank you very much. I forgot about this while doing the backup last night. I had done a backup before, but the backup directories on the other two nodes didn’t have any data, so there must have been an issue with the backup. This time it worked. Thank you very much.

8
| username: TiDBer_MuyinmrC | Original post link

Yes, it’s done. Thank you very much.

9
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.