Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: lighting导入遇到重复数据出错 已经做了忽略重复错误数据的配置 帮忙看看
【TiDB Usage Environment】Production Environment / Testing / Poc
【TiDB Version】
【Reproduction Path】What operations were performed that caused the issue
【Encountered Issue: Issue Phenomenon and Impact】
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots / Logs / Monitoring】
[lightning]
Logs
level = “error”
file = “1tidb-lightning.log”
max-error = 9223372036854775807
[tikv-importer]
Choose the import mode to use
backend = “tidb”
duplicate-resolution = ‘remove’
Set the temporary storage location for sorted key-value pairs, the target path needs to be an empty directory
sorted-kv-dir = “/tmp/tidba”
[[mydumper.files]]
table schema file
pattern = ‘(?i)^(?:[^/]/)kumex_trade_..order_ops_his_v2_..[0-9].csv’
schema = “old_system_data”
table = “order_ops_his_v2_0”
type = “csv”
#pattern = '(?i)^(?:[^/]/)kumex_trade_0.trade_settlement_his_v2_...?.csv’
#schema = “old_system_data”
table = “trade_settlement_his_v2_0”
#type = “csv”
[conflict]
strategy = “replace”
threshold = 9223372036854775807
max-record-rows = 100
[mydumper]
Source data directory.
data-source-dir = “/dataa/”
Configure wildcard rules, the default rules will filter out all tables under the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, INSPECTION_SCHEMA system databases
If this item is not configured, an “unable to find schema” exception will occur when importing system tables
filter = [‘.’, ‘!mysql.', '!sys.’, ‘!INFORMATION_SCHEMA.', '!PERFORMANCE_SCHEMA.’, ‘!METRICS_SCHEMA.', '!INSPECTION_SCHEMA.’]
[tidb]
Information of the target cluster
host = “10.12.8.204”
port = 4000
user = “root”