BR Backup Failed, Error ["br failed"] [error="other error: Coprocessor task terminated due to exceeding the deadline"]

translator_bot · June 20, 2024, 1:18pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份失败，报错 [“br failed”] [error=“other error: Coprocessor task terminated due to exceeding the deadline”]

| username: TiDBer_BwNZ5U9X

The description provided earlier was not detailed enough, so I am posting a new thread for help.
This time, I chose to back up only a 300G database and did not select a shared directory but directly backed up to each TIKV’s local storage, creating folders with the same name locally.

The backup command is: tiup br backup db --pd 192.168.40.92:2379 --db sbtest -s /tidb/share/

The cluster environment consists of 3 TIKV nodes.

An error occurred during the final checksum:
[ERROR] [main.go:59] [“br failed”] [error=“other error: Coprocessor task terminated due to exceeding the deadline”] [errorVerbose=“other error: Coprocessor task terminated due to exceeding the deadline\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1151\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:1035\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:912\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/store/copr/coprocessor.go:622\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594”] [stack=“main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]

Here are the detailed logs:

br_logs.txt (581.3 KB)

It seems that warnings started appearing right at the beginning of the backup.

Could any experts guide me on the possible issues? I would be very grateful!

Additional information:
Later, I increased the speed limit, but it still failed, and there were many errors during the backup process.

translator_bot · June 20, 2024, 1:18pm

| username: jiayou64 | Original post link

–storage “s3://${backup_collection_addr}/snapshot-${date}?access-key=${access-key}&secret-access-key=${secret-access-key}” \ Shared storage backup directory
–storage “local:///home/tidb/backupdata” \ Each TiKV node must have the relevant directory (create if it doesn’t exist) and it must have read and write permissions set to 777

translator_bot · June 20, 2024, 1:18pm

| username: tidb菜鸟一只 | Original post link

The -s option is incorrect, right?

translator_bot · June 20, 2024, 1:18pm

| username: TiDBer_BwNZ5U9X | Original post link

No, no, to rule out potential issues with shared storage, I only tested local backups. I specifically created the same path on each TiKV node and granted 777 permissions, but it still failed.