Error When Performing Full Backup to S3

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br全量备份至s3报错

| username: 张小凉1

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path]
Using br 7.1.1 for full backup of tidb 7.1.1 to remote s3 object storage
[Encountered Problem: Problem Phenomenon and Impact]
Full backup is stuck
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]

| username: WalterWj | Original post link

I didn’t see any error messages :thinking:

| username: 张小凉1 | Original post link

This is an error.

| username: 我是咖啡哥 | Original post link

Timeout, network unreachable?
Is it an error that occurred after backing up for a while or did it happen right from the start?

| username: 张小凉1 | Original post link

An error occurred right from the start.

| username: Jellybean | Original post link

This error looks like a network-related issue. Check for network jitter between the cluster nodes and the BR tool, and see if there are any network connectivity problems.

If everything is confirmed to be normal, try again.

| username: TiDBer_小阿飞 | Original post link

Error obtaining S3 backup path, could it be due to an unstable network?

| username: 大飞哥online | Original post link

Is there a limit on the S3 backup path, network, and bandwidth? You can perform a fio stress test, transfer a large package, or ping a large package to check. Also, make sure the path provided in the br backup parameters is correct.

| username: 我是咖啡哥 | Original post link

Take a look at how the S3-related configurations are set up, such as the endpoint and so on. If the configuration is incorrect, it can also cause this issue.

| username: 张小凉1 | Original post link

The network between the node where the BR tool is located and the object storage S3 server is accessible, with no bandwidth limitations, and there are no issues with uploading data. The BR tool can also ping the KV nodes.

| username: ffeenn | Original post link

Upload files to S3 using the command line with curl or other clients. First, rule out server-to-S3 issues.

| username: cassblanca | Original post link

It seems that there is an issue with network access to S3. The internal and external network endpoints are different, so please pay attention to that. Check the IAM authorization and ACL permissions.

| username: Fly-bird | Original post link

The permissions for S3 are not configured correctly, right?

| username: 路在何chu | Original post link

Is there some content on S3 or is there no backup at all?

| username: heiwandou | Original post link

There is an issue with S3.

| username: 张小凉1 | Original post link

A portion of the snapshot backup can be transferred to S3, including the checkpoints folder and backup.lock. The folder for the storage node cannot be created.

Additionally, if log backup is enabled, only backup.lock and backupdata can be transferred to S3, and the v1 folder cannot be created.

| username: 有猫万事足 | Original post link

Let me provide you with an idea.

Is it possible that there is no rate limit set when executing the BR backup?
If there is no rate limit, BR’s use of bandwidth can be quite aggressive. Distributed uploads can easily saturate the bandwidth, leading to network timeouts.

Try adding the --ratelimit 100 parameter to BR, where the number specifies the bandwidth limit. For example, 100 means 100 MiB.

--ratelimit uint The rate limit of the task, MB/s per node

Try it again after setting the rate limit.