Frequent OOM during Lightning local import

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning local导入时经常oom

| username: h5n1

Version v7.1.2, # uname -a
Linux pd-152 4.19.90-23.6.v2101.ky10.aarch64 #1 SMP Wed Mar 17 14:45:17 CST 2021 aarch64 aarch64 aarch64 GNU/Linux
The import concurrency has been reduced from 24 to 12.
Available memory on the lightning host:

Currently, only this last large table remains to be imported. The table structure is as follows:

Dump file size:

Configuration file:

dmesg.txt (149.3 KB)

3 concurrent processes are not working:

| username: Jolyne | Original post link

How about changing this parameter and giving it a try?

| username: wakaka | Original post link

Try changing the Lightning version.

| username: yiduoyunQ | Original post link

How about trying CSV?

| username: DBAER | Original post link

Is it possible that other resources are being occupied, causing lightning to have the highest memory usage and resulting in an OOM? Write a script to monitor the actual memory usage of lightning.

| username: h5n1 | Original post link

It should be a bug in version 7.1.2. I used version 7.5.1 to import and upgraded the cluster in the process.