Error occurs after several retries when backing up data to Amazon S3 using BR, upload stops at 4.48%

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 用BR备份数据到亚马逊云的S3上，重试几次都是上传到4.48%后报错

| username: tidb_bruce

[TiDB Usage Environment] Production\Test Environment\POC
[TiDB Version] Server version 5.2.1, br version 6.1.1
[Encountered Problem]
When using br (version 6.1.1) to back up a database on the server version 5.2.1, it initially succeeds, but fails after uploading around 4.8%. This failure occurs consistently at this point.
[Reproduction Path] Operations performed that led to the issue
[Problem Phenomenon and Impact]
[Logs]
[2022/10/19 19:35:34.399 +08:00] [INFO] [info.go:49] [“Welcome to Backup & Restore (BR)”] [release-version=v6.1.1] [git-hash=5263a0abda61f102122735049fd0dfadc7b7f8b2] [git-branch=heads/refs/tags/v6.1.1] [go-version=go1.18.5] [utc-build-time=“2022-08-25 10:36:33”] [race-enabled=false]
[2022/10/19 19:35:34.399 +08:00] [INFO] [common.go:692] [arguments] [__command=“br backup db”] [db=db_corpgenuis_label_xn] [log-file=backuptable.log] [pd=“[192.168.40.56:2379]”] [ratelimit=20] [s3.region=cn-northwest-1] [storage=s3://tidb-online/2022-10-20/]
[2022/10/19 19:35:34.400 +08:00] [INFO] [conn.go:257] [“new mgr”] [pdAddrs=192.168.40.56:2379]
[2022/10/19 19:35:34.405 +08:00] [INFO] [client.go:392] [“[pd] create pd client with endpoints”] [pd-address=“[192.168.40.56:2379]”]
[2022/10/19 19:35:34.412 +08:00] [INFO] [base_client.go:332] [“[pd] update member urls”] [old-urls=“[http://192.168.40.56:2379]”] [new-urls=“[http://192.168.40.56:2379,http://192.168.40.57:2379,http://192.168.40.58:2379]”]
[2022/10/19 19:35:34.412 +08:00] [INFO] [base_client.go:350] [“[pd] switch leader”] [new-leader=http://192.168.40.58:2379] [old-leader=]
[2022/10/19 19:35:34.412 +08:00] [INFO] [base_client.go:105] [“[pd] init cluster id”] [cluster-id=7009174823259873762]
[2022/10/19 19:35:34.412 +08:00] [INFO] [client.go:687] [“[pd] tso dispatcher created”] [dc-location=global]
[2022/10/19 19:35:34.414 +08:00] [INFO] [conn.go:232] [“checked alive KV stores”] [aliveStores=3] [totalStores=3]
[2022/10/19 19:35:34.414 +08:00] [INFO] [client.go:392] [“[pd] create pd client with endpoints”] [pd-address=“[192.168.40.56:2379]”]
[2022/10/19 19:35:34.418 +08:00] [INFO] [base_client.go:332] [“[pd] update member urls”] [old-urls=“[http://192.168.40.56:2379]”] [new-urls=“[http://192.168.40.56:2379,http://192.168.40.57:2379,http://192.168.40.58:2379]”]
[2022/10/19 19:35:34.419 +08:00] [INFO] [base_client.go:350] [“[pd] switch leader”] [new-leader=http://192.168.40.58:2379] [old-leader=]
[2022/10/19 19:35:34.419 +08:00] [INFO] [base_client.go:105] [“[pd] init cluster id”] [cluster-id=7009174823259873762]
[2022/10/19 19:35:34.419 +08:00] [INFO] [client.go:687] [“[pd] tso dispatcher created”] [dc-location=global]
[2022/10/19 19:35:34.422 +08:00] [INFO] [tidb.go:75] [“new domain”] [store=tikv-7009174823259873762] [“ddl lease”=1s] [“stats lease”=-1ns] [“index usage sync lease”=0s]
[2022/10/19 19:35:34.432 +08:00] [INFO] [domain.go:1731] [acquireServerID] [serverID=2974628] [“lease id”=475980f9e3fd300f]
[2022/10/19 19:35:34.436 +08:00] [WARN] [info.go:241] [“init TiFlashPlacementManager”] [“pd addrs”=“[192.168.40.57:2379,192.168.40.56:2379,192.168.40.58:2379]”]
[2022/10/19 19:35:35.090 +08:00] [INFO] [domain.go:176] [“full load InfoSchema success”] [currentSchemaVersion=0] [neededSchemaVersion=10726] [“start time”=641.85005ms]
[2022/10/19 19:35:35.093 +08:00] [INFO] [domain.go:437] [“full load and reset schema validator”]
[2022/10/19 19:35:35.093 +08:00] [WARN] [domain.go:449] [“loading schema takes a long time”] [“take time”=650.686382ms]
[2022/10/19 19:35:35.093 +08:00] [INFO] [ddl.go:382] [“[ddl] start DDL”] [ID=1c49d0b4-82a8-430d-b6ed-d60b93b8f1f5] [runWorker=false]
[2022/10/19 19:35:35.097 +08:00] [INFO] [backup.go:273] [“get new_collations_enabled_on_first_bootstrap config from system table”] [new_collation_enabled=False]
[2022/10/19 19:35:35.097 +08:00] [INFO] [client.go:95] [“new backup client”]
[2022/10/19 19:35:35.585 +08:00] [INFO] [client.go:139] [“backup encode timestamp”] [BackupTS=436778915747332097]
[2022/10/19 19:35:35.585 +08:00] [INFO] [backup.go:308] [“current backup safePoint job”] [safePoint=“{ID=br-eef49f9a-09fb-4593-a8c8-423994ab6097,TTL=5m0s,BackupTime="2022-10-19 19:35:35.584 +0800 CST",BackupTS=436778915747332097}”]
[2022/10/19 19:35:35.596 +08:00] [INFO] [backup.go:369] [“get placement policies”] [count=0]
[2022/10/19 19:35:35.642 +08:00] [INFO] [client.go:594] [“backup started”] [range-sn=0] [startKey=748000000000000BFD5F720000000000000000] [endKey=748000000000000BFD5F72FFFFFFFFFFFFFFFF00] [rateLimit=20971520] [concurrency=4]
[2022/10/19 19:35:35.644 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=0] [store-id=1] [“retry time”=0]
[2022/10/19 19:35:35.645 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=0] [store-id=4] [“retry time”=0]
[2022/10/19 19:35:35.646 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=0] [store-id=5] [“retry time”=0]
[2022/10/19 19:36:31.324 +08:00] [INFO] [client.go:610] [“finish backup push down”] [range-sn=0] [small-range-count=4]
[2022/10/19 19:36:31.324 +08:00] [INFO] [client.go:627] [“time range backed up”] [range-sn=0] [StartVersion=0] [EndVersion=436778915747332097]
[2022/10/19 19:36:31.324 +08:00] [INFO] [client.go:588] [“backup range finished”] [range-sn=0] [take=55.682307132s]
[2022/10/19 19:36:31.324 +08:00] [INFO] [client.go:594] [“backup started”] [range-sn=1] [startKey=748000000000000BFD5F69800000000000000100] [endKey=748000000000000BFD5F698000000000000001FB] [rateLimit=20971520] [concurrency=4]
[2022/10/19 19:36:31.325 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=1] [store-id=1] [“retry time”=0]
[2022/10/19 19:36:31.325 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=1] [store-id=4] [“retry time”=0]
[2022/10/19 19:36:31.325 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=1] [store-id=5] [“retry time”=0]
[2022/10/19 19:37:30.113 +08:00] [INFO] [client.go:610] [“finish backup push down”] [range-sn=1] [small-range-count=4]
[2022/10/19 19:37:30.113 +08:00] [INFO] [client.go:627] [“time range backed up”] [range-sn=1] [StartVersion=0] [EndVersion=436778915747332097]
[2022/10/19 19:37:30.113 +08:00] [INFO] [client.go:588] [“backup range finished”] [range-sn=1] [take=58.789046391s]
[2022/10/19 19:37:30.113 +08:00] [INFO] [client.go:594] [“backup started”] [range-sn=2] [startKey=748000000000000BFF5F720000000000000000] [endKey=748000000000000BFF5F72FFFFFFFFFFFFFFFF00] [rateLimit=20971520] [concurrency=4]
[2022/10/19 19:37:30.121 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=2] [store-id=4] [“retry time”=0]
[2022/10/19 19:37:30.121 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=2] [store-id=5] [“retry time”=0]
[2022/10/19 19:37:30.121 +08:00] [INFO] [client.go:1041] [“try backup”] [range-sn=2] [store-id=1] [“retry time”=0]
[2022/10/19 19:38:29.739 +08:00] [WARN] [push.go:180] [“backup occur storage error”] [range-sn=2] [error=“Io(Custom { kind: Other, error: "failed to put object Error during dispatch: error trying to connect: dns error: failed to lookup address information: Name or service not known" })”]
[2022/10/19 19:39:28.605 +08:00] [WARN] [push.go:180] [“backup occur storage error”] [range-sn=2] [error=“Io(Custom { kind: Other, error: "failed to put object Error during dispatch: error trying to connect: dns error: failed to lookup address information: Name or service not known" })”]

2
| username: xfworld | Original post link

It looks like a network issue.

3
| username: tidb狂热爱好者 | Original post link

The versions must be consistent. You need to use the BR tool from version 5.2 for the backup.

4
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.