How to Limit the Speed of Dumpling to Reduce Impact on Production

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dumpling如何限速，减少对生产上的影响

| username: TiDB_C罗

There is a requirement to export a large table on TiDB using dumpling. During the operation, slow queries and response times increased, which also triggered alarms. Is there any way to limit the export speed of dumpling?

2
| username: zhanggame1 | Original post link

How do you write dumping?

3
| username: 啦啦啦啦啦 | Original post link

Try adjusting these parameters.

image
image973×324 26.3 KB

4
| username: tidb菜鸟一只 | Original post link

-t default is 64, you can change it to 4…

5
| username: TiDB_C罗 | Original post link

Okay, I’ll give it a try.

6
| username: TiDB_C罗 | Original post link

tiup dumpling -u xxx -P 4000 -h xxx -p ‘xxx’ --filetype csv -o /data/ -F 100MiB -t 2 -T xxx.xxx --tidb-mem-quota-query=1073741824 --consistency=auto

7
| username: TiDB_C罗 | Original post link

Changing to -t 1 still results in excessive traffic.

8
| username: zhanggame1 | Original post link

-t write 1
-r write 0
Give it a try

9
| username: WalterWj | Original post link

300 MB is okay, it’s just over 100 megabytes per second.

10
| username: redgame | Original post link

If it still doesn’t work, post some logs for us to check.

11
| username: okenJiang | Original post link

You can choose not to set -t.

12
| username: WalterWj | Original post link

If you still want to limit the speed, there is another way. Set the session parameters in dumpling to adjust the table scan operator and index scan operator to 1.

–params dumpling parameters can be found on the official website: 使用 Dumpling 导出数据 | PingCAP 文档中心

The scan operator is: 系统变量 | PingCAP 文档中心

13
| username: cy6301567 | Original post link

This is reliable.

14
| username: TiDB_C罗 | Original post link

Adding these parameters, tidb_distsql_scan_concurrency=1 and tidb_mem_quota_query=209715200, along with -T, can be considered as implementing rate limiting. Beyond this, there’s not much more that can be done.

15
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.