Error Encountered with Duplicate Data During Lighting Import Despite Configuring to Ignore Duplicate Errors - Please Assist

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lighting导入遇到重复数据出错 已经做了忽略重复错误数据的配置 帮忙看看

| username: tidb狂热爱好者

【TiDB Usage Environment】Production Environment / Testing / Poc
【TiDB Version】
【Reproduction Path】What operations were performed that caused the issue
【Encountered Issue: Issue Phenomenon and Impact】
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots / Logs / Monitoring】
[lightning]

Logs

level = “error”
file = “1tidb-lightning.log”
max-error = 9223372036854775807
[tikv-importer]

Choose the import mode to use

backend = “tidb”
duplicate-resolution = ‘remove’

Set the temporary storage location for sorted key-value pairs, the target path needs to be an empty directory

sorted-kv-dir = “/tmp/tidba”
[[mydumper.files]]

table schema file

pattern = ‘(?i)^(?:[^/]/)kumex_trade_..order_ops_his_v2_..[0-9].csv’
schema = “old_system_data”
table = “order_ops_his_v2_0”
type = “csv”
#pattern = '(?i)^(?:[^/]/)kumex_trade_0.trade_settlement_his_v2_...?.csv’
#schema = “old_system_data”
table = “trade_settlement_his_v2_0”
#type = “csv”

[conflict]
strategy = “replace”
threshold = 9223372036854775807

max-record-rows = 100

[mydumper]

Source data directory.

data-source-dir = “/dataa/”

Configure wildcard rules, the default rules will filter out all tables under the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, INSPECTION_SCHEMA system databases

If this item is not configured, an “unable to find schema” exception will occur when importing system tables

filter = [‘.’, ‘!mysql.', '!sys.’, ‘!INFORMATION_SCHEMA.', '!PERFORMANCE_SCHEMA.’, ‘!METRICS_SCHEMA.', '!INSPECTION_SCHEMA.’]
[tidb]

Information of the target cluster

host = “10.12.8.204”
port = 4000
user = “root”

2
| username: Daniel-W | Original post link

This is . right?
TiDB Lightning Configuration Parameters | PingCAP Documentation Center

3
| username: vcdog | Original post link

Check if the source data has duplicates.

4
| username: 源de爸 | Original post link

Please provide a detailed screenshot of the error information.

5
| username: DBAER | Original post link

Please provide a screenshot of the error.

6
| username: TiDBer_rvITcue9 | Original post link

Waiting for the screenshot.

7
| username: zhaokede | Original post link

Everyone is waiting for detailed errors to analyze.

8
| username: 有猫万事足 | Original post link

Remove the above configuration directly.

Under the [tikv-importer] tag, configure it like this:

[tikv-importer]

# Choose the import mode to use

backend = “tidb”
#duplicate-resolution = ‘remove’
on-duplicate = "replace"

Try again to see if it works.