Backup Failure

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份失败

| username: dengqee

[TiDB Usage Environment] Production Environment / Test / Poc
[TiDB Version]
[Reproduction Path] What operations were performed to cause the issue
Using br for s3 backup, error occurs halfway through the backup
br command as follows:

export AWS_ACCESS_KEY_ID=******************** && export AWS_SECRET_ACCESS_KEY=******************** && ./br backup full --log-file ./brlog/backupfull.log-instance-164672146297720230512T010635H --pd http://10.0.0.1:2379 --ratelimit 250 --s3.endpoint http://10.0.0.132:7480 --storage s3://tidb-backup/instance1646721462977-20230512T010635H

[Encountered Issue: Issue Phenomenon and Impact]
Error occurs every time, and the store and region that report the error are different each time
br log:

[2023/05/12 02:58:43.826 +08:00] [ERROR] [main.go:58] ["br failed"] [error="error happen in store 126 at 10.0.0.7:20160: Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-hubeijiangxiazone1-hubeijiangxiazone1-zonegroup</HostId></Error>\" }): [BR:KV:ErrKVStorage]tikv storage occur I/O error"] [errorVerbose="[BR:KV:ErrKVStorage]tikv storage occur I/O error\nerror happen in store 126 at 10.0.0.7:20160: Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-zonegroup</HostId></Error>\" })\ngithub.com/pingcap/tidb/br/pkg/backup.(*pushDown).pushBackup\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/push.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/client.go:499\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/client.go:453\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/data1/jenkins/workspace/tibr/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/data1/debezium/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go1.16/src/runtime/asm_amd64.s:1371"] [stack="main.main\n\t/data1/jenkins/workspace/tibr/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go1.16/src/runtime/proc.go:225"]

tikv log:

[2023/05/12 02:58:43.818 +08:00] [ERROR] [endpoint.rs:828] ["backup region failed"] [err_code=KV:Unknown] [err="Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-zonegroup</HostId></Error>\" })"] [end_key=7480000000000000705F72A0000000666073F5] [start_key=7480000000000000705F72A0000000664D78DC] [region="id: 343115 start_key: 7480000000000000FF705F72A000000066FF4D78DC0000000000FA end_key: 7480000000000000FF705F72A000000066FF6073F50000000000FA region_epoch { conf_ver: 1631 version: 3540 } peers { id: 343117 store_id: 120 } peers { id: 343118 store_id: 126 } peers { id: 562292 store_id: 122 }"]

[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: dba-kit | Original post link

What did you use to set up S3?

| username: dba-kit | Original post link

You can set --s3.provider according to the corresponding value, or try adding &force-path-style=false in the URL. Many providers actually just change the value of this parameter.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.