TiDB 7.5 Lightning Incremental Import Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb 7.5 lightning增量导入报错

| username: TiDBer_OB4kHrS7

【TiDB Usage Environment】Production Environment
【TiDB Version】V7.5.0
【Reproduction Path】After full dump export, perform incremental export. After full import using lightning, perform incremental import and encounter an error.
【Encountered Problem: Phenomenon and Impact】
[root@tidb_history_tikv1 tidb-community-server-v7.5.0-linux-amd64]# tail -f lightning.log
[2024/02/04 15:22:00.019 +08:00] [WARN] [config.go:1563] [“currently only per-task configuration can be applied, global configuration changes can only be made on startup”] [“global config changes”=“[lightning.level,lightning.file]”]
[2024/02/04 15:22:00.019 +08:00] [ERROR] [main.go:103] [“tidb lightning encountered error stack info”] [error=“[Lightning:Config:ErrInvalidConfig]conflict.strategy cannot be used with tikv-importer.parallel-import”] [errorVerbose=“[Lightning:Config:ErrInvalidConfig]conflict.strategy cannot be used with tikv-importer.parallel-import\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:155\ngithub.com/pingcap/tidb/br/pkg/lightning/config.(*Conflict).adjust\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/config/config.go:1341\ngithub.com/pingcap/tidb/br/pkg/lightning/config.(*Config).Adjust\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/config/config.go:1594\ngithub.com/pingcap/tidb/br/pkg/lightning.(*Lightning).RunOnceWithOptions\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/lightning.go:350\nmain.main.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/tidb-lightning/main.go:94\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/tidb-lightning/main.go:95\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650”]

【Resource Configuration】
[root@tidb_history_tikv1 tidb-community-server-v7.5.0-linux-amd64]# cat lightning.toml
[lightning]
level = “info”
file = “lightning.log”
table-concurrency = 8
io-concurrency = 6

[tikv-importer]
backend = “tidb”
sorted-kv-dir = “/data/lightning_data” # Specify SSD directory
incremental-import = true

[mydumper]
data-source-dir = “/data/account_db2”
filter = [‘.’,‘!mysql.‘,’!sys.’,‘!INFORMATION_SCHEMA.‘,’!PERFORMANCE_SCHEMA.’,‘!METRICS_SCHEMA.‘,’!INSPECTION_SCHEMA.’]

[checkpoint]
enable = true

[tidb]
host = “10.230.8.29”
port = 4000
user = “root”
password = “XXXXXX”
status-port = 10080
pd-addr = “10.230.6.137:2379”

| username: db_user | Original post link

Lightning does not support incremental import.

| username: TiDBer_OB4kHrS7 | Original post link

In version 5.3, there was this parameter, but in version 7.5, it’s gone. Without it, incremental updates can’t be done.

| username: 小龙虾爱大龙虾 | Original post link

This thing was never meant for incremental import. Doesn’t incremental import mean that the target table already has data? Using the TiDB mode for import directly adds to it, right? :thinking:

| username: db_user | Original post link

Look at my screenshot, it is clearly written there. Although the parameter name looks like “incremental,” it actually controls concurrency.

| username: TiDBer_OB4kHrS7 | Original post link

Yes, incremental import is when the target table already has data, the backend uses TiDB mode, and the downstream TiDB can normally provide services.

| username: TiDBer_OB4kHrS7 | Original post link

How is the incremental-import parameter configured? Can you provide the relevant link?

| username: db_user | Original post link

Version 7.5 no longer has this parameter, no need to configure it, it has been changed to parallel-import.

| username: 小龙虾爱大龙虾 | Original post link

Take a close look at the image the expert replied to you with.

| username: tidb菜鸟一只 | Original post link

Does Lightning support incremental import now? I thought I was going to learn something new…
The error you’re getting, “conflict.strategy cannot be used with tikv-importer.parallel-import,” means that the conflict.strategy parameter cannot be used with parallel-import=true. Since you’ve set the incremental-import = true parameter, it’s equivalent to setting parallel-import=true. I suggest you remove the corresponding parameter and try importing again.

| username: TiDBer_OB4kHrS7 | Original post link

Setting parallel-import=false allows for incremental import, but you need to delete the SQL statements for creating databases and tables.

| username: tidb菜鸟一只 | Original post link

Well, deletion is the same, the default is false.

| username: kkpeter | Original post link

The error message seems quite clear. If I understand correctly, tikv-importer and parallel-import cannot be used together.

conflict.strategy = strategy conflict

| username: dba远航 | Original post link

It seems that the parameter is not supported.

| username: Soysauce520 | Original post link

Switching to TiDB mode allows importing, but local mode does not work.

| username: ffeenn | Original post link

Lightning hasn’t been used for incremental imports, but BR is used frequently. When will there be incremental support?

| username: TiDBer_OB4kHrS7 | Original post link

Starting from version 5.3, there has been incremental updates.

| username: Kongdom | Original post link

:flushed: Why do you think there has been an incremental update since 5.3? I didn’t see any description in the documentation~

| username: 有猫万事足 | Original post link

Let’s sort things out.

First, you didn’t clarify the original format of your import. Because you configured

[tikv-importer]
backend = “tidb”

and used logical import, your original format must be something like CSV or SQL files, not SST or other physical export formats.

Then, in the case of logical import, it’s essentially converting these files into insert statements for execution. There is no requirement for whether there is data in the original database tables. So both full and incremental imports are possible.

Next, let’s talk about the second issue.

incremental-import

This parameter, as @db_user pointed out in the documentation, has never been an incremental import parameter, and its name has been changed in subsequent versions. The new name parallel-import is more appropriate, meaning parallel import. As mentioned in the screenshot above, it refers to having multiple Lightning instances importing into the same table, which is a method used to quickly import data into the database. If you only have one Lightning instance and are not in the scenario of quickly importing to initialize the database, you can directly comment out this parameter or set it to false.

Back to the original issue, the error you encountered initially was indeed because you set

incremental-import = true

From your subsequent replies, it seems you have already resolved this issue.