How to Configure Memory Settings for TiDB Lightning Import?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lighting 导入数oom,如何配置lighting 内存配置

| username: xingzhenxiang

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path] OOM during data import with Lightning, how to configure Lightning memory settings
[Encountered Problem: Data import with Lightning was killed]

[Attachment: Screenshot/Log/Monitoring]

OOM occurred during data import with Lightning, how to configure Lightning memory usage, currently I have 64GB of memory

lightingoom.txt (8.2 KB)

| username: caiyfc | Original post link

Take a look at this post, you have the same error:
lightning报错“Region distribution is unbalanced, the ratio of the regions” - :ringer_planet: TiDB 技术问题 / 备份&数据迁移 - TiDB 的问答社区 (asktug.com)

| username: dba远航 | Original post link

Rate limiting, concurrency limiting

| username: 小于同学 | Original post link

Restricted.

| username: 小龙虾爱大龙虾 | Original post link

Does it occupy more than 60GB of memory? Can you share the Lightning parameters? What are the characteristics of the imported table? Does it have many columns or very wide columns?

| username: WalterWj | Original post link

Post your current configuration and let’s take a look.

| username: xingzhenxiang | Original post link

Sure, here is the translation:

“Okay, the relevant configuration file is as follows, please help diagnose it
tidb-lightning.toml (3.0 KB)”

| username: WalterWj | Original post link

# Information about the target cluster. The address of the tidb-server, just fill in one.
host = "10.19.110.47"
port = 3306
user = "root"
# Set the password to connect to TiDB, which can be in plain text or Base64 encoded.
password = ""
# Must be configured. Table structure information is obtained from the "status-port" of TiDB.
status-port = 10080

It looks like 47 3306 is a load balancer. Can the 10080 port of 47 access the 10080 port of the tidb-server?

# The concurrency of data. By default, it is the same as the number of logical CPUs.
# In the case of mixed deployment, you can configure its size to 75% of the number of logical CPUs to limit CPU usage.
# region-concurrency =

You can try to set this configuration smaller :thinking:.

| username: xingzhenxiang | Original post link

My 47 is a TiDB server, but I just changed the default port.
region-concurrency = I’ll check this configuration, thank you.

| username: redgame | Original post link

Configure to the minimum and gradually increase.

| username: DBAER | Original post link

Is there a large field?