BR Reports Errors Several Times When Restoring Data

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br恢复数据连续几次报错

| username: TiDBer_Lee

[TiDB Usage Environment] Production Environment
[TiDB Version] v6.5.5
[Execution Command]:

export AWS_ACCESS_KEY_ID=xxxxx
export AWS_SECRET_ACCESS_KEY=xxxxx
tiup br:v6.5.5 restore full \
--pd "10.xx.xx.xx:2379" \
--filter 'npd_xxx.ads_tab1' \
--filter 'npd_xxx.ads_tab2' \
--filter 'npd_xxx.ads_tab3' \
--s3.region cn-northwest-1 \
--storage "s3://db-buket/xxx/prod/filter/20231107143530/" \
--log-file "./rs1108.log"

[Error as follows]:

[2023/11/08 13:08:17.764 +00:00] [INFO] [base_client.go:143] ["[pd] exit member loop due to context canceled"]
[2023/11/08 13:08:17.764 +00:00] [INFO] [client.go:719] ["[pd] exit tso dispatcher"] [dc-location=global]
[2023/11/08 13:08:17.764 +00:00] [INFO] [pd.go:209] ["closed pd http client"]
[2023/11/08 13:08:17.765 +00:00] [INFO] [base_client.go:143] ["[pd] exit member loop due to context canceled"]
[2023/11/08 13:08:17.766 +00:00] [INFO] [collector.go:220] ["units canceled"] [cancel-unit=0]
[2023/11/08 13:08:17.766 +00:00] [INFO] [collector.go:74] ["Full Restore failed summary"] [total-ranges=91964] [ranges-succeed=91964] [ranges-failed=0] [split-region=3m11.786085203s] [restore-ranges=54357]
[2023/11/08 13:08:17.766 +00:00] [INFO] [client.go:783] ["[pd] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
[2023/11/08 13:08:17.766 +00:00] [INFO] [client.go:719] ["[pd] exit tso dispatcher"] [dc-location=global]
[2023/11/08 13:08:17.766 +00:00] [ERROR] [restore.go:59] ["failed to restore"] [error="other error: Coprocessor task terminated due to exceeding the deadline"] [errorVerbose="other error: Coprocessor task terminated due to exceeding the deadline\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1200\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1076\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:945\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:655\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runRestoreCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:59\nmain.newFullRestoreCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/restore.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
[2023/11/08 13:08:17.766 +00:00] [ERROR] [main.go:60] ["br failed"] [error="other error: Coprocessor task terminated due to exceeding the deadline"] [errorVerbose="other error: Coprocessor task terminated due to exceeding the deadline\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1200\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1076\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:945\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:655\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

Has anyone encountered this issue, and how did you resolve it?

| username: xfworld | Original post link

Is the cluster normal?

Coprocessor task terminated due to exceeding the deadline

This error is a bit outrageous…

| username: yulei7633 | Original post link

other error: Coprocessor task terminated due to exceeding. I haven’t encountered this error before. Looking forward to someone checking it out. Will continue to follow.

| username: 有猫万事足 | Original post link

There is no speed limit, right? If the bandwidth is fully occupied, some network services will become inaccessible. Try adding speed limit parameters and try again.

| username: TiDBer_Lee | Original post link

There is no speed limit. Are you referring to the parameter --ratelimit?
What is a suitable limit, 50M?

| username: Fly-bird | Original post link

Can S3 storage be written to normally?

| username: 有猫万事足 | Original post link

It depends on your bandwidth. As long as it doesn’t get fully occupied, it’s fine.

| username: Kongdom | Original post link

Please refer to this link:

| username: TiDBer_Lee | Original post link

No, this parameter is no longer available:
[ERROR] [main.go:60] [“br failed”] [error=“unknown flag: --timeout”] [stack=“main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:60\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]

| username: Kongdom | Original post link

:pleading_face: Then let’s look at other solutions.

| username: 像风一样的男子 | Original post link

It looks like there is a kv anomaly, focus on checking the kv logs.

| username: zhanggame1 | Original post link

Check the TiDB cluster status and the logs of each component. It may not necessarily be a DR issue.

| username: 我是咖啡哥 | Original post link

Is it possible that the storage endpoint parameter wasn’t specified?

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.