How to Control the Speed of Data Import with Lightning

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning导入数据如何控制速度

| username: 愚者之眷

How to control the concurrent SQL when importing data with lightning using backend = “tidb”?

2
| username: buddyyuan | Original post link

The parameter for controlling concurrency in logical import is region-concurrency.

3
| username: 近墨者zyl | Original post link

The region-concurrency parameter indeed controls concurrency, but more importantly, it is inherently related to memory. By default, each thread allocates 2GB of memory, so the value of region-concurrency multiplied by 2GB should ideally be controlled below 70% of the available memory on the server. This can effectively prevent OOM (Out of Memory) issues.

4
| username: alfred | Original post link

There should be no direct parameters for SQL concurrency control.

5
| username: 愚者之眷 | Original post link

Sure :smiley:

6
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.