Issue of Backup Rate Limiting in BR

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: br备份限速问题

| username: chenhanneu

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots / Logs / Monitoring]
Documentation:
Use the --ratelimit parameter to limit the speed of backup tasks. Note that this parameter limits the speed at which backup files are stored to external storage. When calculating the size of the backup file, please refer to the backup data size (after compressed) in the backup log. After setting --ratelimit, to avoid the failure of rate limiting due to too many tasks, the concurrency parameter of br will be automatically adjusted to 1.
Issue:
The ratelimit restricts the maximum speed, and the concurrency parameter of br will be automatically adjusted to 1. In actual observation, during the backup period, 90% of the traffic is very low, far below the ratelimit value, resulting in a very long backup time.
During the backup window, it is actually permissible to run for a long time under the ratelimit traffic load. After limiting the speed and automatically adjusting the concurrency to 1, the backup cannot be completed within the backup window.
Is there a good way to both limit the speed and shorten the backup time?

| username: zhanggame1 | Original post link

Do not write ratelimit, test different concurrency values yourself.

| username: tidb菜鸟一只 | Original post link

Instead of using the ratelimit parameter, change it to SHOW config WHERE NAME LIKE ‘%backup.num-threads%’

| username: chenhanneu | Original post link

How should these two parameters be understood? Are they in a master-slave relationship?

| username: 有猫万事足 | Original post link

In my own backup practice on Tencent Cloud, this ratelimit is necessary. It can slow down the speed a bit, but if br consumes all the bandwidth, it will cause br to fail to connect to pd, resulting in a backup failure.

I think adjustments to other parameters can be put on hold for now. First, check if removing the ratelimit in the current environment will cause the aforementioned issue. If the above problem occurs, then adjusting other parameters won’t be very meaningful.

| username: Daniel-W | Original post link

Configuring --ratelimit makes it serial, while not configuring it makes it parallel.
You can adjust the backup-related parameters.
TiKV Configuration File Description | PingCAP Documentation Center

| username: 江湖故人 | Original post link

Could this situation be caused by a hotspot node slowing down the overall backup speed?

| username: TiDBer_5cwU0ltE | Original post link

Have you tried limiting the speed of the network card? It’s a one-step solution, but it has a significant impact.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.